Planet NUUG

February 12, 2018

Petter Reinholdtsen

Overvåkning i Kina vs. Norge

Jeg lar meg fascinere av en artikkel i Dagbladet om Kinas håndtering av Xinjiang, spesielt følgende utsnitt:

«I den sørvestlige byen Kashgar nærmere grensa til Sentral-Asia meldes det nå at 120.000 uigurer er internert i såkalte omskoleringsleirer. Samtidig er det innført et omfattende helsesjekk-program med innsamling og lagring av DNA-prøver fra absolutt alle innbyggerne. De mest avanserte overvåkingsmetodene testes ut her. Programmer for å gjenkjenne ansikter og stemmer er på plass i regionen. Der har de lokale myndighetene begynt å installere GPS-systemer i alle kjøretøy og egne sporingsapper i mobiltelefoner.

Politimetodene griper så dypt inn i folks dagligliv at motstanden mot Beijing-regimet øker.»

Beskrivelsen avviker jo desverre ikke så veldig mye fra tilstanden her i Norge.

Dataregistrering	Kina	Norge
Innsamling og lagring av DNA-prøver fra befolkningen	Ja	Delvis, planlagt for alle nyfødte.
Ansiktsgjenkjenning	Ja	Ja
Stemmegjenkjenning	Ja	Nei
Posisjons-sporing av mobiltelefoner	Ja	Ja
Posisjons-sporing av biler	Ja	Ja

I Norge har jo situasjonen rundt Folkehelseinstituttets lagring av DNA-informasjon på vegne av politiet, der de nektet å slette informasjon politiet ikke hadde lov til å ta vare på, gjort det klart at DNA tar vare på ganske lenge. I tillegg finnes det utallige biobanker som lagres til evig tid, og det er planer om å innføre evig lagring av DNA-materiale fra alle spebarn som fødes (med mulighet for å be om sletting).

I Norge er det system på plass for ansiktsgjenkjenning, som en NRK-artikkel fra 2015 forteller er aktiv på Gardermoen, samt brukes til å analysere bilder innsamlet av myndighetene. Brukes det også flere plasser? Det er tett med overvåkningskamera kontrollert av politi og andre myndigheter i for eksempel Oslo sentrum.

Jeg er ikke kjent med at Norge har noe system for identifisering av personer ved hjelp av stemmegjenkjenning.

Posisjons-sporing av mobiltelefoner er ruinemessig tilgjengelig for blant annet politi, NAV og Finanstilsynet, i tråd med krav i telefonselskapenes konsesjon. I tillegg rapporterer smarttelefoner sin posisjon til utviklerne av utallige mobil-apper, der myndigheter og andre kan hente ut informasjon ved behov. Det er intet behov for noen egen app for dette.

Posisjons-sporing av biler er rutinemessig tilgjengelig via et tett nett av målepunkter på veiene (automatiske bomstasjoner, køfribrikke-registrering, automatiske fartsmålere og andre veikamera). Det er i tillegg vedtatt at alle nye biler skal selges med utstyr for GPS-sporing (eCall).

Det er jammen godt vi lever i et liberalt demokrati, og ikke en overvåkningsstat, eller?

February 12, 2018 08:40 AM

February 11, 2018

Petter Reinholdtsen

How hard can æ, ø and å be?

We write 2018, and it is 30 years since Unicode was introduced. Most of us in Norway have come to expect the use of our alphabet to just work with any computer system. But it is apparently beyond reach of the computers printing recites at a restaurant. Recently I visited a Peppes pizza resturant, and noticed a few details on the recite. Notice how 'ø' and 'å' are replaced with strange symbols in 'Servitør', 'Å BETALE', 'Beløp pr. gjest', 'Takk for besøket.' and 'Vi gleder oss til å se deg igjen'.

I would say that this state is passed sad and over in embarrassing.

I removed personal and private information to be nice.

February 11, 2018 04:10 PM

January 04, 2018

Holder de ord

Her er Venstre, Høyre og Fremskrittspartiet enige

Venstre har denne uken startet regjeringsforhandlinger med Høyre og Fremskrittspartiet.

Pressen har skrevet flere spaltemeter om de største kampsakene og om hvilke porteføljer som er de viktigste for de tre partiene. De er uenige om mangt, både innenfor klimafeltet, innvandringsfeltet og forholdet til EU. Men det er også en del saker partiene er enige om. Holder de ord har derfor gått gjennom partiprogrammene til de tre for perioden 2017-21 for å finne områdene der partiene er enige seg imellom. Partiprogrammene finner du her.

Familie og helse

På et overordnet nivå ønsker alle tre partier at offentlige og private tjenestetilbydere skal likestilles. De ønsker også et tettere samarbeid mellom skole, barnevern, politi og eventuelle andre aktører på oppvekstfeltet. Venstre og FrP lover også begge at det skal være økt satsing på tidlig innsats og forebygging i helsevesenet, og at frivillige og ideelle aktører i større grad skal bidra her. I en eventuell regjeringserklæring for disse tre partiene vil vi kunne forvente at fosterfamilier blir lovet styrkede rettigheter, at ettervern for barnevernsbarn igjen blir satt på agendaen, og at helsestasjonene blir lovet styrket.

Ettervernet ble lovet gjennomgått i Sundvolden-erklæringen fra 2013, men ble ikke fulgt opp i Solbergs første regjeringsperiode. Styrking av helsestasjoner og økt satsing på velferdsteknologi, som alle tre partier er opptatt av i inneværende periode, ble derimot gjennomført i forrige regjeringsperiode.

Utenriks og nordområder

All tre partier lover fortsatt norsk militær tilstedeværelse i nordområdene, noe som ikke er politisk kontroversielt, selv om de fleste av opposisjonspartiene har et annet fokus enn det militære når nordområdene omtales.

Alle tre partier lover også en reduksjon av tollbarrierer, slik at utviklingsland skal ha bedre tilgang til det norske markedet, men det er neppe noe som vil bli prioritert høyt. Solberg1-regjeringen lovte også i forrige periode at det skulle jobbes for bedre ordninger som ville gi utviklingsland tilgang til norske markeder, men det ble ikke igangsatt noen konkrete tiltak for å nå dette målet.

Næringsliv og finanser

Alle tre partier har vært, og er fortsatt, svært negative til formuesskatten, men Venstre har endret standpunkt noe siden forrige periode. Venstre vil nå redusere skatten gjennom en gradvis økning av bunnfradraget, men nevner ikke lenger at skatten skal fjernes i sitt partiprogram for inneværende periode. En regjering bestående av disse tre partiene vil imidlertid måtte søke støtte hos Senterpartiet for å få gjennomført større endringer i formuesskatten. De øvrige partiene vil enten øke skatten for større formuer, eller, der de er opptatt av reduksjon, kun ønsker reduksjon for skatt på arbeidende kapital.

Alle tre partier har programfestet at kontanter skal beholdes som pliktig betalingsmiddel. Ingen partier har vedtatt at de ønsker å fjerne dette, så dette er en ikke-sak i norsk politisk sammenheng.

Mer substansielt er løftene fra alle tre partier om at det med dem ved makta skal bli enklere å starte egen bedrift. FrP og Høyre utdyper ikke i sine partiprogram på hvilken måte det skal bli enklere, mens Venstre peker på stimulering til økt risikovilje for kapitaltilførsel.

Skole og utdanning

Alle tre partier vil bygge flere studentboliger. FrP har imidlertid gått vekk fra tallet 2000, som de hadde vedtatt skulle bygges i forrige periode, mens Venstre har økt ambisjonene fra 2000 boliger til 3000. Både Høyre og Venstre ønsker også å gjøre studentboliger til et eget reguleringsformål og plan- og bygningsloven.

Alle tre partier går inn for å sikre den pågående innfasingen av 11 måneders studiestøtte.

Alle tre er også enige om at de ønsker åpne tilgjengelige data om skoleresultater. Dette finnes på skoleporten.no, så disse løftene må tolkes som en garanti for at dette ikke endres.

Det kan bli noe uenighet om hvordan friskoler skal behandles, men alle tre partier er positive til konseptet.

Den mest kostbare og vanskeligst gjennomførbare enigheten vil være bedre språkopplæring i barnehagene, da det allerede er vanskelig å få nok pedagogisk utdannet personell til barnehagene. En styrking av finansieringen til universiteter og høgskoler, som både Venstre og FrP har gått til valg på, vil også bli en utgiftspost dersom de to partiene får med seg Høyre på dette.

I tillegg er de tre partiene enige om at entreprenørskap på et eller annet vis bør inn i skolen, at skoleelever skal ha mer fysisk aktivitet og at skolepersonell skal ha tilbud om etter- og videreutdanning.

Kultur og frivillighet

Kultursektoren kan forvente å bli lovet enklere regler og mindre søknadsbyråkrati, mens frivilligheten kan forvente lovnader om en forbedret momskompensasjonsordning. At Venstre får gjennomslag for sitt ønske om å trappe opp momskompensasjonen til 100 prosent er imidlertid usannsynlig. I forrige regjeringsperiode lovet de blåblå at ordningen skulle forbedres, men nøyde seg med en styrking av potten. For inneværende periode lover Høyre kun at ordningen skal være «god og forutsigbar».

Klima, energi og landbruk

Det er liten tvil om at konsekvensutredning av oljeutvinning av Lofoten, Vesterålen og Senja vil være en av de store stridstemaene under forhandlingene. Det er imidlertid enighet om at det skal satses mer på vannkraft, og Venstre og FrP er enige om at skogen skal brukes aktivt som klimatiltak, med blant annet planting og gjødsling av skog. Høyre nevner ikke dette spesifikt.

Høyre er enig med Venstre om at det bør produseres mer biodrivstoff i Norge, at arbeidet med å rydde opp forurensede masser på havbunnen må fortsette, samt at det skal jobbes for å hindre spredning av fremmede arter. FrP omtaler ikke noen av disse temaene i sitt partiprogram.

Alle tre partier lover å oppheve odelsbestemmelsene i Grunnloven. Dette ble forsøkt også i forrige periode, men regjeringen klarte ikke å skaffe flertall for dette.

Det kan også hende at FrP og Venstre vil få med seg Høyre på å arbeide internasjonalt for et forbud mot dumping av fisk.

Justis og beredskap

Alle tre partier er positive til forsøket med dyrepoliti som ble startet i forrige periode, og vi kan forvente at det blir lovet å utvide denne ordningen i en eventuell regjeringserklæring. Det er også enighet om at soningskøene skal ned, og det er da naturlig å forvente at det vil bli lovet bygging av flere soningsplasser. Alle tre partier vil også at utenlandske domfelte i større grad skal sone straffen sin i hjemlandet.

Både Høyre og Venstre ønsker å tillate dobbelt statsborgerskap. FrP er mot dette, men det er flertall for dobbelt statsborgerskap på Stortinget, så det kan hende dette blir gjennomført i denne perioden.

by Hanna Tranås (hanna@holderdeord.no) atJanuary 04, 2018 11:57 AM

December 29, 2017

Peter Hansteen (That Grumpy BSD Guy)

Twenty-plus years on, SMTP callbacks are still pointless and need to die

A rarely used legacy misfeature of the main Internet email protocol creeps back from irrelevance as a minor annoyance. You should ask your mail and antispam provider about their approach to 'SMTP callbacks'. Be wary of any assertion that is not backed by evidence.

Even if you are an IT professional and run an email system, you could be forgiven for not being immediately aware that there is such a thing as SMTP callbacks, also referred to as callback verification. As you will see from the Wikipedia article, the feature was never widely adopted, and for all too understandable reasons.

If you do run a mail system, you have probably heard about that feature's predecessor, the still-required but rarely used SMTP VRFY and EXPN commands. Those commands offer a way to verify whether an address is valid and to show the component addresses that a mailing list resolves to, respectively.

Back when all things inter-networking were considered experimental and it was generally thought that information should flow freely in and between those experimental networks, it was quite common for mail servers to offer VRFY and EXPN service to all comers.

I'm old enough to remember using VRFY by hand, telnet-ing to port 25 on a mail server and running VRFY $user@$domain.$tld commands to check whether an email address was indeed valid. I've forgotten which domains and persons were involved, but I imagine the reason why was that I wanted to contact somebody who had said something interesting in a post to a USENET news group.

But networkers trying to make contact with each other were not the only ones who discovered the VRFY and EXPN commands. Soon spammers were using those commands to actively harvest actually! valid! deliverable! addresses, and by 1999 the RFC2505 best practices document recommended disabling the features altogether. After all, there would usually be some other way available to find somebody's email address (there was even a FAQ, a longish Frequently Asked Questions document with apparent USENET origins written and maintained on the subject, a copy of which can be found here).

In roughly the same time frame, somebody came up with the idea of SMTP callbacks. The idea was that all domains on the Internet need to publish the address of their mail exchangers via DNS MX (mail exchanger) records. The logical next step is then that when a piece of mail arrives over SMTP, the receiving end should be able to contact the sender domain's known mail exchanger to check that the sender address is indeed valid. If you by now hear the echoes of VRFY and EXPN, you're right. There are indications that some early implementations did in fact use VRFY for that purpose.

But then the world changed, and you could not rely on VRFY being available in the post-RFC2505 world.

In the post-RFC2505 world, the other side would most likely not offer up any useful information in response to VRFY commands, and you would most likely be limited to the short interchange that the Wikipedia entry quotes,

HELO <verifier host name>
MAIL FROM:<>
RCPT TO:<the address to be tested>
QUIT

which a perceptive reader would identify as only verifying in a very limited sense that the domain's mail exchanger was indeed equipped with a functional SMTP service.

It is worth noting, as many have over the years, that the MX records only specify where a domain expects to receive mail, not where valid mail from the domain is supposed to originate. Several mechanisms to help identify valid mail senders for a domain have been devised in the intervening years, but none existed at the time SMTP callbacks were considered even remotely useful.

For reasons that are not entirely clear, some developers kept working on SMTP callback code and several mail server implementations available today (2017) still contain code that looks like it was intended to support information-rich callbacks, if the system was configured to enable the feature at all. The default configurations in general do not enable the SMTP callback feature, and mail admins rarely bother to even learn much about the largely disused and (in my opinion at least) not too well thought out feature.

This all happened back in the 1990s, but quite recently an incident occurred that indicates that in some pockets of the Internet, SMTP callbacks are still in use, and in at least some cases data from the callbacks are used for generating blacklists and block mail delivery. The last part should raise a few eyebrows at least.

Jumping forward from the distant 1990s to the present day, regular readers of this column will be aware that bsdly.net and cooperating domains run SMTP service with OpenBSD spamd(8) doing greylisting service, and that the spamd(8) setup produces a greytrapping-based blacklist which is available for download, dumped to a file (available here and here) once per hour.

Maintaining the mail system and the blacklist also involves keeping an eye on mail-related activities, and invalid addresses in our domains that turn up in the greylist are usually added to the list of spamtrap addresses within not too many hours after they first appear. The process of actually adding spamtrap addresses is a manual one, but based on the output of pathetically simple shell scripts that run as cron jobs.

The list of spamtraps has grown over the years to more than 38 000 entries. Most of the entries have local parts that are pure generated gibberish, some entries are probably degraded versions of earlier spamtrap addresses and some again seem to conform with specific patterns, including but not limited to SMTP or NNTP message IDs.

On August 19th and 20th 2017 I noticed a different, but yet familiar pattern in some of the new entries.

The entry that caught my eye had the MAIL FROM: part as

mx42.antispamcloud.com-1503146097-testing@bsdly.com

The local part pattern was somewhat familiar, and breaks down to

$localhostname-$epochtime-testing

with @targetdomain.$tld (in our case, bsdly.com) appended. I had at this point totally forgotten about SMTP callbacks, but I decided to check the logs for any traces of activity involving that host. The only trace I could find in the logs was at the spamd-serving firewall in front of the bsdly.com domain's secondary mail exchanger:

Aug 19 14:35:27 delilah spamd[26915]: 207.244.64.181: connected (25/24)
Aug 19 14:35:38 delilah spamd[26915]: (GREY) 207.244.64.181: <> -> <mx42.antispamcloud.com-1503146097-testing@bsdly.com>
Aug 19 14:35:38 delilah spamd[15291]: new entry 207.244.64.181 from <> to <mx42.antispamcloud.com-1503146097-testing@bsdly.com>, helo mx18-12.smtp.antispamcloud.com
Aug 19 14:35:38 delilah spamd[26915]: 207.244.64.181: disconnected after 11 seconds.

Essentially a normal first contact: spamd at our end answers slowly, one byte per second, but the greylist entry is created in the expectation that any caller with a valid message to deliver would try again within a reasonable time. The spamd synchronization between the hosts in our group of greylisting hosts would see to that an entry matching this sequence would appear in the greylist on all participating hosts.

But the retry never happened, and even if it had, that particular local-part would anyway have produced an "Unknown user" bounce. But at that point I decided to do a bit of investigation and dug out what seemed to be a reasonable point of contact for the antispamcloud.com domain and sent an email with a question about the activity.

That message bounced, with the following explanation in the bounce message body:

DOMAINS@ANTISPAMCLOUD.COM

host filter10.antispamcloud.com [31.204.155.103]

SMTP error from remote mail server after end of data:

550 The sending IP (213.187.179.198) is listed on https://spamrl.com as a source of dictionary attacks.

As you have probably guessed, 213.187.179.198 is the IPv4 address of the primary mail exchanger for bsdly.net, bsdly.com and a few other domains under my care.

If you go to the URL quoted in the bounce, you will notice that the only point of contact is via an email adress in an unrelated domain.

I did fire off a message to that address from an alternate site, but before the answer to that one arrived, I had managed to contact another of their customers and got confirmation that they were indeed running with an exim setup that used SMTP callbacks.

The spamrl.com web site states clearly that they will not supply any evidence in support of their decision to blacklist. Somebody claiming to represent spamrl.com did respond to my message, but as could be expected from their published policy was not willing to supply any evidence to support the claim stated in the bounce.

In my last message to spamrl.com before starting to write this piece, I advised

I remain unconvinced that the description of that problem is accurate, but investigation at this end can not proceed without at least some supporting evidence such as times of incidents, addresses or even networks affected.

If there is a problem at this end, it will be fixed. But that will not happen as a result of handwaving and insults. Actual evidence to support further investigation is needed.

Until verifiable evidence of some sort materializes, I will assume that your end is misinterpreting normal greylisting behavior or acting on unfounded or low-quality reports from less than competent sources.

The domain bsdly.com was one I registered some years back mainly to fend off somebody who offered to help the owner of the bsdly.net domain acquire the very similar bsdly.com domain at the price of a mere few hundred dollars.

My response was to spend something like ten dollars (or was it twenty?) to register the domain via my regular registrar. I may even have sent back a reply about trying to sell me what I already owned, but I have not bothered to dig that far back into my archives.

The domain does receive mail, but is otherwise not actively used. However, as the list of spamtraps can attest (the full list does not display in a regular browser, since some of the traps are interpreted as html tags, if you want to see it all, fetch the text file instead), others have at times tried to pass off something or other with from addresses in that domain.

But with the knowledge that this outfit's customers are believers in SMTP callbacks as a way to identify spam, here is my hypothesis on what actually happened:

On August 19th 2017, my greylist scanner identified the following new entries referencing the bsdly.com domain:

anecowuutp@bsdly.com
pkgreewaa@bsdly.com
eemioiyv@bsdly.com
keerheior@bsdly.com
mx42.antispamcloud.com-1503146097-testing@bsdly.com
vbehmonmin@bsdly.com
euiosvob@bsdly.com
otjllo@bsdly.com
akuolsymwt@bsdly.com

I'll go out on a limb and guess that mx42.antispamcloud.com was contacted by any of the roughly 5000 hosts blacklisted at bsdly.net at the time, with an attempt to deliver a message with a MAIL FROM: of either anecowuutp@bsdly.com, pkgreewaa@bsdly.com, eemioiyv@bsdly.com or perhaps most likely keerheior@bsdly.com, which appears as a bounce-to address in the same hourly greylist dump where mx42.antispamcloud.com-1503146097-testing@bsdly.com first appears as a To: address.

The first seen time in epoch notation for keerheior@bsdly.com is

1503143365

which translates via date -r to

Sat Aug 19 13:49:25 CEST 2017

while mx42.antispamcloud.com-1503146097-testing@bsdly.com is first seen here at epoch 1503146138, which translates to Sat Aug 19 14:35:38 CEST 2017.

The data indicate that this initial (and only) attempt to contact was aimed at the bsdly.com domain's secondary mail exchanger, and was intercepted by the greylisting spamd that sits in the incoming signal path to there. The other epoch-tagged callbacks follow the same pattern, as can be seen from the data preserved here.

Whatever action or address triggered the callback, the callback appears to have followed the familiar script:

register attempt to deliver mail
look up the domain stated in the MAIL FROM: or perhaps even the HELO or EHLO
contact the domain's mail exchangers with the rump SMTP dialog quoted earlier
with no confirmation or otherwise of anything other than the fact that the domain's mail exchangers do listen on the expected port, proceed to whatever the next step is.

The known facts at this point are:

a mail system that is set up for SMTP callbacks received a request to deliver mail from keerheior@bsdly.com
the primary mail exchanger for bsdly.com has the IPv4 address 213.187.179.198

Both of these are likely true. The second we know for sure, and the first is quite likely. What is missing here is any consideration of where the request to deliver came from.

From the data we have here, we do not have any indication of what host contacted the system that initiated the callback. In a modern configuration, it is reasonable to expect that a receiving system checks for sender validity via any SPF, DKIM or DMARC records available, or for that matter, greylist and wait for the next attempt (in fact, greylisting before performing any other checks - as an OpenBSD spamd(8) setup would do by default - is likely to be the least resource intensive approach).

We have no indication that the system performing the SMTP callout used any such mechanism to find an indication as to whether the communication partner was in fact in any way connected to the domain it was trying to deliver mail for.

My hypothesis is that whatever code is running on the SMTP callback adherents' systems does not check the actual sending IP address, but assumes that any message claiming to be from a domain must in fact involve the primary mail exchanger of that domain and since the code likely predates the SPF, DKIM and DMARC specifications by at least a decade, it will not even try to check those types of information. Given the context it is a little odd but perhaps within reason that in all cases we see here, the callback is attempted not to the domain's primary mail exchanger, but the secondary.

With somebody or perhaps even several somebodies generating nonsense addresses in the bsdly.com domain at an appreciable rate (see the record of new spamtraps starting May 20th, 2017, alternate location here) and trying to deliver using those fake From: addresses to somewhere doing SMTP callback, it's not much of a stretch to assume that the code was naive enough to conclude that the purported sender domain's primary mail exchanger was indeed performing a dictionary attack.

The most useful lesson to take home from this sorry affair is likely to be that you need to kill SMTP callback setups in any system where you may find them. In today's environment, SMTP callbacks do not in fact provide useful information that is not available from other public sources, and naive use of results from those lookups is likely to harm unsuspecting third parties.

So,

If you are involved in selling or operating a system that behaves like the one described here and are in fact generating blacklists based on those very naive assumptions, you need to stop doing so right away.

Your mistaken assumptions help produce bad data which could lead to hard to debug problems for innocent third parties.

Or as we say in the trade, you are part of the problem.
If you are operating a system that does SMTP callbacks but doesn't do much else, you are part of a small problem and likely only inconveniencing yourself and your users.

The fossil record (aka the accumulated collection of spamtrap addresses at bsdly.net) indicates that the callback variant that includes epoch times is rare enough (approximately 100 unique hosts registered over a decade) that callback activity in total volume probably does not rise above the level of random background noise.

There may of course be callback variants that have other characteristics, and if you see a way to identify those from the data we have, I would very much like to hear from you.
If you are a customer of somebody selling antispam products, you have reason to demand an answer to just how, if at all, your antispam supplier utilizes SMTP callbacks. If they think it's a fine and current feature, you have probably been buying snake oil for years.
If you are the developer or maintainer of mail server code that contains the SMTP callbacks feature, please remove the code. Leaving it disabled by default is not sufficient. Removing the code is the only way to make sure the misfeature will never again be a source of confusing problems.

For some hints on what I consider a reasonable and necessary level of transparency in blacklist maintenance, please see my April 2013 piece Maintaining A Publicly Available Blacklist - Mechanisms And Principles.

The data this article is based on still exists and will be available for further study as long as the request comes wit a reasonable justification. I welcome comments in the comment field or via email (do factor in any possible greylist delay, though).

Any corrections or updates that I find necessary based on your responses will be appended to the article.

Update 2017-09-05: Since the article was originally published, we've seen a handful of further SMTP callback incidents. The last few we've handled by sending the following to the addresses that could be gleaned from whois on the domain name and source IP address (with mx.nxdomain.nx and 192.0.2.74 inserted as placeholders here to protect the ignorant):

Hi,

I see from my greylist dumps that the host identifying as

mx.nxdomain.nx, IP address 192.0.2.74

is performing what looks like SMTP callbacks, with the (non-existent of course) address

mx.nxdomain.nx-1504629949-testing@bsdly.com

as the RCPT TO: address.

It is likely that this activity has been triggered by spam campaigns using made up addresses in one of our little-used domains as from: addresses.

A series of recent incidents here following the same pattern are summarized in the article

http://bsdly.blogspot.com/2017/08/twenty-plus-years-on-smtp-callbacks-are.html

Briefly, the callbacks do not work as you expect. Please read the article and then disable that misfeature. Otherwise you will be complicit in generating false positives for your SMTP blacklist.

If you have any questions or concerns, please let me know.

Yours sincerely,

Peter N. M. Hansteen

If you've received a similarly-worded notice recently, you know why and may be closer to having a sanely run mail service.

Update 2017-11-02: It looks like the spamrl.com service still has customers that believe in the same claim made in the bounce messages quoted earlier in this article and perform SMTP callbacks exactly like they did when this article was first written. If you have any verifiable information on that outfit and their activities, I would very much like to hear from you.

by Peter N. M. Hansteen (noreply@blogger.com) atDecember 29, 2017 03:52 PM

December 19, 2017

NUUG news

Kort rapport fra to dager i tingretten om DNS-domene-inndragning

Det har vært stille her på bloggen om DNS-beslag-saken, men det har ikke vært stille i saken. Vi har rett og slett ikke rukket blogge på grunn av høy arbeidsbelastning på jobb og privaten, men her er endelig en liten oppdatering. Vi ville satt umåtelig pris på at du viste din støtte til vår innsats her ved å donere penger til forsvarsfondet. Til nå er det sikkert påløpt mer enn 100 000,- i advokatutgifter, og vi har ikke fått i nærheten av dette i donasjoner. Vi holder ut, men det blir enklere med hjelp.

I dag var NUUG, EFN og IMC i tingretten etter at IMC nektet å godta inndragning av DNS-domenet popcorn-time.no. Dette er neste steg i saken som startet med Økokrims beslag av det samme domenet. Siden sist bloggpost har anken av beslaget til lagmannsretten ikke blitt tatt til følge, og anken til høyesterett blitt avvist. Det satte et foreløbig punktum for beslags-saken, og i mellomtiden har Økokrim avsluttet etterforskningen, konkludert med at de ikke har noen å sikte, og at de vil inndra DNS-domenet. Inndragning er visst det de kaller det når politiet tar ting etter at etterforskning er avsluttet, mens beslag er når de tar noe før etterforskningen er avsluttet. Dermed var vi i tingretten for å protestere på inndragningen, for å få bedre belyst sakens prinsipielle sider.

En av de sakkyndige vitnene var Håkon Wium Lie, som har tegnet og skrevet følgende oppsummering av de to dagene i retten:

[kjappe notater fra Follo Tingrett]

Dag 1. Rettens leder (Jonn Ola Sørensen) har med seg to meddommere, en kvinne og en mann som begge ser ut å være eldre enn ham. Ingen av dem er vel i kjernegruppa av popcorn-time brukere, men hvem vet: vitnet Rune Ljostad har forteller oss at Popcorntime brukes av svært mange i Norge. Om dette skyldes Dagsrevyens reportasje:

https://tv.nrk.no/serie/dagsrevyen/NNFA19012715/27-01-2015#t=26m46s

... eller Aftenpostens artikler:

https://www.aftenposten.no/norge/i/2JXl/Gratisfilm-pa-Popcorn-Time-er-ulovlig-likevel

eller lenker fra Google, eller det nå stengte nettsiden:

https://popcorn-time.no/

er et tema i saken.

Aktor Maria Bache Dahl fra Økokrim og Ola Tellesbø, som reprsenterer den norske registraren, hadde hvert sitt innledningsforedrag fra hver sin side i saken. Deretter presenterte advokater fra NUUG og EFN kort sine standpunkt.

Så var det tid for vitner. Først registrar Morten E. Eriksen som fortalte hvordan han enkelt registrerer navn i .no-domenet uten å vite hvem som står bak eller hva slags innhold som legges på sidene. På denne måten får han ofte høre om navn på filmer eller produkter som slippes neste år.

Advokat Rune Ljostad stilte som vitne for Økokrim. Han er profesjonell piratjeger og orienterte om andre lignende saker der norske og andre myndigheter stenger nettsted eller tar domenenavn. Da han ble spurt om det var en "Play"-knapp på popcorn-time.no ville han ikke svare, og Maria Bache Dahl kom raskt inn og sa at Rune var ikke innkalt for å snakke om popcorn-time.no, men om det hun kaller "tjenesten Popcorntime".

Dette ser ut til å bli et annet spørsmål: hva er popcorn-time? En tjeneste? En protokoll? Et nettsted? En avspiller? En søkemotor?

Dommerens kommentarer tyder på at han sliter med å sortere begrepene. Rune Ljostad har en god teknisk forståelse. Det har Petter Reinholdtsen også, men han fikk ikke slippe til i dag pga. tidsmangel.

Siste vitne i dag er Dani Bacsa som er internett "investigator" for Motion Picture Association. Han bor i Brussels og har flydd inn for anledningen. Han kom langt for å fortelle det mange andre vitner i saken også kunne sagt. Ingen er uenige om de tekniske sidene ved saken. Her er en kort oppsummering:

popcorntime.no ble registrert av en ukjent person/virksomhet

på siden ble det lagt informasjon om, og lenker til popcorn-time programvare

det lå ingen filmer på popcorntime.no

Spørsmålet i saken er om dette kvalifiserer til "medvirkning" til en forbytelse. Dersom svaret er ja vil retten antageligvis opprettholde Økokrims beslagleggelse av popcorn-time.no. I motsatt fall vil nettnavnet formodentlig bli returnert til den/dem som registrerte navnet.

Dag 2 startet med at Petter Reinholdtsen forklarte hvorfor Rune Ljostads tall fra i går er feil. Petter argumenterte at tallet på filmer i det fri er langt høyere enn det Rune Ljostad anslo (som var 1%).

Etterpå snakket Willy Johansen, som representerer dem som selger filmer på fysiske media, om hvordan salget har gått ned. Popcorn-time fikk mye av skylda, men han visste ikke hvor stor innvirkning popcorn-time.no hadde hatt. Han sa videre at det er svært enkelt å laste opp filmer fra mobiltelefonen og at disse da automatisk ville spres til alle ens kontakter.

Morten Vestergaard Stephanson fortalte hvor mye det koster å produsere filmer og behovet for inntekter. Han nevnte også at mye at filmproduksjonen er offentlig finansiert i Norge.

Tom Fredrik Blenning presentert saken fra EFNs side: EFN og samarbeidende organisasjoner jobber mot sensur og for ytringsfrihet. Å beslaglegge popcorn-time.no er et inngrep i ytringsfriheten. Han sa også at automatisk spredning av filmer til alle kontakter på mobilen er oppspinn.

Wilhelm Joys Andersen hadde analysert kildekoden bak popcorn-time.no og konkluderte at siden er optimalisert for søkemotorer -- de aller fleste som så siden må ha kommet fra en søkemotor og dermed har vært interessert i temaet allerede. Formålet med å opprette siden er antageligvis å bli funnet av brukere som søker etter "popcorntime" i en søkemotor. Annonser for VPN-tjenester framtredende på siden.

Håkon Wium Lie var siste vitne. Har forklarte at nettleseren Opera har lagt inn støtte for både bittorrent-protokollen og VPN -- teknologien har legitim bruk og de fleste større norske bedrifter tilbyr VPN for ansatte. Mye fildeling er ulovlig i følge norsk rett, men skadevirkningene er begrenset: uten piratebay ville vi ikke fått Spotify og Netflix. Nettsiden popcorn-time.no var svært perifer i forhold til hovedgjerningen, som er ulovlig opplasting av filmer. Toleransen for å skrive om og lenke til kontroversielle tema må være høy i Norge, og fildeing utgjør ikke noe samfunnsproblem på linje med oppfordring til terror.

Etter lunsj holder aktor Maria Bache Dahl sin prosedyre. Hun mener, ikke helt overraskende, at popcorn-time.no var medvirkende til brudd på på opphavsretten og at nettsiden popcorn-time.no må vurderes anneledes enn en artikkel i Aftenposten.

Hun nedlegger påstand om at registrar må tåle inndragning av domenet popcorn-time.no. Hun argumenterer hvorfor.

Ola Tellesbø starter med å nedlegge påstand om at inndragning av bruksretten til popcorn-time.no ikke tåles, og at Økokrim skal betale saksomkostninger. Han argumenterer hvorfor ikke.

Kirill forsetter og sier at "opplasting av kildefiler", som aktor mener er ulovlig, er et for vagt begrep til at det kan regnes om ulovlig. Nå brukere laster opp til YouTube gjøre de akkurat det samme.

Ola fortsetter og trekker fram DVD-Jon saken. Han leser fra dommen, hvor Jon ble frikjent for medvirkning. Han argumeterer at nettsiden er perifer i forhold til hovedgjerningen, og at de som leste teksten kanskje heller ble forvirret enn oppmuntret. Han mener videre at "medvirkning" hører til narko- og drapassaker, og ikke opphavsrett.

Advokat Kjetil Wick Sætre, som representerter NUUG, gir så en kortere kommentar. Han nevner bl.a. at NUUG har tilbudt seg å overta domenet for å bruke det på en klart lovlig måte. NUUG støtter ikke ulovlig fildeling, men mener teknologien også har legitim bruk som skaper verdier. NUUG mener ulovlig bruk bør håndheves så nært lovbruddet som mulig, dvs. ved opplasting.

Advokat Henny Hallingskog-Hultin, som representerte EFN, sa at kjernen i saken er vekting mellom ytringfrihet og håndhevelse av immaterielle rettigheter. EFN anser at ytringene på nettstedet er for langt fra hovedgjerningen -- I denne saken er det mange ledd mellom opplasting av kildefiler (som er den ulovlige hovedgjerningen) og nettstedet popcorn-time.no. Inndragning at popcorn-time.no har minimal innvirkning for ulovlig nedlasting, og inndragning vil ha en "nedkjølende effekt". Saken er et misbruk av opphavsrettigheter for å sensurere internett.
Etter en kort pause kom det en kort replikk fra aktor.

Kirills hovedpunkt, på vegne av alle, var at Økokrim bruker uforholdsmessig store ressurser på saken. Dette er den 5te (!) rettsrunden i Follo Tingrett. Advokatene jobber, met ett unntak, gratis fordi saken har prisipiell betydning.

Dommen forkynnes 15 januar kl 10. Ankefristen er 2 uker.

Retten ble hevet 16:45 og hoveddommer ønsket alle en god jul.

December 19, 2017 08:30 PM

December 10, 2017

Peter Hansteen (That Grumpy BSD Guy)

Forcing the password gropers through a smaller hole with OpenBSD's PF queues

While preparing material for the upcoming BSDCan PF and networking tutorial, I realized that the pop3 gropers were actually not much fun to watch anymore. So I used the traffic shaping features of my OpenBSD firewall to let the miscreants inflict some pain on themselves. Watching logs became fun again.

Yes, in between a number of other things I am currently in the process of creating material for new and hopefully better PF and networking session.

I've been fishing for suggestions for topics to include in the tutorials on relevant mailing lists, and one suggestion that keeps coming up (even though it's actually covered in the existling slides as well as The Book of PF) is using traffic shaping features to punish undesirable activity, such as

Idea for pf tutorial: throttling of http abusers using pf and altq. /cc @pitrh @stucchimax
— Dan Langille (@DLangille) April 16, 2017

What Dan had in mind here may very well end up in the new slides, but in the meantime I will show you how to punish abusers of essentially any service with the tools at hand in your OpenBSD firewall.

Regular readers will know that I'm responsible for maintaining a set of mail services including a pop3 service, and that our site sees pretty much round-the-clock attempts at logging on to that service with user names that come mainly from the local part of the spamtrap addresses that are part of the system to produce our hourly list of greytrapped IP addresses.

But do not let yourself be distracted by this bizarre collection of items that I've maintained and described in earlier columns. The actual useful parts of this article follow - take this as a walkthrough of how to mitigate a wide range of threats and annoyances.

First, analyze the behavior that you want to defend against. In our case that's fairly obvious: We have a service that's getting a volume of unwanted traffic, and looking at our logs the attempts come fairly quickly with a number of repeated attempts from each source address. This similar enough to both the traditional ssh bruteforce attacks and for that matter to Dan's website scenario that we can reuse some of the same techniques in all of the configurations.

I've written about the rapid-fire ssh bruteforce attacks and their mitigation before (and of course it's in The Book of PF) as well as the slower kind where those techniques actually come up short. The traditional approach to ssh bruteforcers has been to simply block their traffic, and the state-tracking features of PF let you set up overload criteria that add the source addresses to the table that holds the addresses you want to block.

I have rules much like the ones in the example in place where there I have a SSH service running, and those bruteforce tables are never totally empty.

For the system that runs our pop3 service, we also have a PF ruleset in place with queues for traffic shaping. For some odd reason that ruleset is fairly close to the HFSC traffic shaper example in The Book of PF, and it contains a queue that I set up mainly as an experiment to annoy spammers (as in, the ones that are already for one reason or the other blacklisted by our spamd).

The queue is defined like this:

   queue spamd parent rootq bandwidth 1K min 0K max 1K qlimit 300

yes, that's right. A queue with a maximum throughput of 1 kilobit per second. I have been warned that this is small enough that the code may be unable to strictly enforce that limit due to the timer resolution in the HFSC code. But that didn't keep me from trying.

And now that I had another group of hosts that I wanted to just be a little evil to, why not let the password gropers and the spammers share the same small patch of bandwidth?

Now a few small additions to the ruleset are needed for the good to put the evil to the task. We start with a table to hold the addresses we want to mess with. Actually, I'll add two, for reasons that will become clear later:

table <longterm> persist counters
table <popflooders> persist counters

The rules that use those tables are:

block drop log (all) quick from <longterm>

pass in quick log (all) on egress proto tcp from <popflooders> to port pop3 flags S/SA keep state \
(max-src-conn 1, max-src-conn-rate 1/1, overload <longterm> flush global, pflow) set queue spamd

pass in log (all) on egress proto tcp to port pop3 flags S/SA keep state \
(max-src-conn 5, max-src-conn-rate 6/3, overload <popflooders> flush global, pflow)

The last one lets anybody connect to the pop3 service, but any one source address can have only open five simultaneous connections and at a rate of six over three seconds.

Any source that trips up one of these restrictions is overloaded into the popflooders table, the flush global part means any existing connections that source has are terminated, and when they get to try again, they will instead match the quick rule that assigns the new traffic to the 1 kilobyte queue.

The quick rule here has even stricter limits on the number of allowed simultaneous connections, and this time any breach will lead to membership of the longterm table and the block drop treatment.

The for the longterm table I already had in place a four week expiry (see man pfctl for detail on how to do that), and I haven't gotten around to deciding what, if any, expiry I will set up for the popflooders.

The results were immediately visible. Monitoring the queues using pfctl -vvsq shows the tiny queue works as expected:

queue spamd parent rootq bandwidth 1K, max 1K qlimit 300
[ pkts:     196136 bytes:   12157940 dropped pkts: 398350 bytes: 24692564 ]
[ qlength: 300/300 ]
[ measured:     2.0 packets/s, 999.13 b/s ]

and looking at the pop3 daemon's log entries, a typical encounter looks like this:

Apr 19 22:39:33 skapet spop3d[44875]: connect from 111.181.52.216
Apr 19 22:39:33 skapet spop3d[75112]: connect from 111.181.52.216
Apr 19 22:39:34 skapet spop3d[57116]: connect from 111.181.52.216
Apr 19 22:39:34 skapet spop3d[65982]: connect from 111.181.52.216
Apr 19 22:39:34 skapet spop3d[58964]: connect from 111.181.52.216
Apr 19 22:40:34 skapet spop3d[12410]: autologout time elapsed - 111.181.52.216
Apr 19 22:40:34 skapet spop3d[63573]: autologout time elapsed - 111.181.52.216
Apr 19 22:40:34 skapet spop3d[76113]: autologout time elapsed - 111.181.52.216
Apr 19 22:40:34 skapet spop3d[23524]: autologout time elapsed - 111.181.52.216
Apr 19 22:40:34 skapet spop3d[16916]: autologout time elapsed - 111.181.52.216

here the miscreant comes in way too fast and only manages to get five connections going before they're shunted to the tiny queue to fight it out with known spammers for a share of bandwidth.

I've been running with this particular setup since Monday evening around 20:00 CEST, and by late Wednesday evening the number of entries in the popflooders table had reached approximately 300.

I will decide on an expiry policy at some point, I promise. In fact, I welcome your input on what the expiry period should be.

One important takeaway from this, and possibly the most important point of this article, is that it does not take a lot of imagination to retool this setup to watch for and protect against undesirable activity directed at essentially any network service.

You pick the service and the ports it uses, then figure out what are the parameters that determine what is acceptable behavior. Once you have those parameters defined, you can choose to assign to a minimal queue like in this example, block outright, redirect to something unpleasant or even pass with a low probability.

All of those possibilities are part of the normal pf.conf toolset on your OpenBSD system. If you want, you can supplement these mechanisms with a bit of log file parsing that produces output suitable for feeding to pfctl to add to the table of miscreants. The only limits are, as always, the limits of your imagination (and possibly your programming abilities). If you're wondering why I like OpenBSD so much, you can find at least a partial answer in my OpenBSD and you presentation.

FreeBSD users will be pleased to know that something similar is possible on their systems too, only substituting the legacy ALTQ traffic shaping with its somewhat arcane syntax for the modern queues rules in this article.

Will you be attending our PF and networking session in Ottawa, or will you want to attend one elsewhere later? Please let us know at the email address in the tutorial description.

Update 2017-04-23: A truly unexpiring table, and downloadable datasets made available

Soon after publishing this article I realized that what I had written could easily be taken as a promise to keep a collection of POP3 gropers' IP addresses around indefinitely, in a table where the entries never expire.

Table entries do not expire unless you use a pfctl(8) command like the ones mentioned in the book and other resources I referenced earlier in the article, but on the other hand table entries will not survive a reboot either unless you arrange to have table contents stored to somewhere more permanent and restored from there. Fortunately our favorite toolset has a feature that implements at least the restoring part.

Changing the table definition quoted earler to read

table <popflooders> persist counters file "/var/tmp/popflooders"

takes part of the restoring, and the backing up is a matter of setting up a cron(8) job to dump current contents of the table to the file that will be loaded into the table at ruleset load.

Then today I made another tiny change and made the data available for download. The popflooders table is dumped at five past every full hour to pop3gropers.txt, a file desiged to be read by anything that takes a list of IP addresses and ignores lines starting with the # comment character. I am sure you can think of suitable applications.

In addition, the same script does a verbose dump, including table statistiscs for each entry, to pop3gropers_full.txt for readers who are interested in such things as when an entry was created and how much traffic those hosts produced, keeping in mind that those hosts are not actually blocked here, only subjected to a tiny bandwidth.

As it says in the comment at the top of both files, you may use the data as you please for your own purposes, for any re-publishing or integration into other data sets please contact me via the means listed in the bsdly.net whois record.

As usual I will answer any reasonable requests for further data such as log files, but do not expect prompt service and keep in mind that I am usually in the Central European time zone (CEST at the moment).

I suppose we should see this as a tiny, incremental evolution of the "Cybercrime Robot Torture As A Service" (CRTAAS) concept.

Update 2017-04-29: While the world was not looking, I supplemented the IP address dumps with versions including one with geoiplocation data added and a per country summary based on the geoiplocation data.

Spending a few minutes with an IP address dump like the one described here and whois data is a useful excersise for anyone investigating incidents of this type. This .csv file is based on the 2017-04-29T1105 dump (preserved for reference), and reveals that not only is the majority of attempts from one country but also a very limited number of organizations within that country are responsible for the most active networks.

The spammer blacklist (see this post for background) was of course ripe for the same treatment, so now in addition to the familiar blacklist, that too comes with a geoiplocation annotated version and a per country summary.

Note that all of those files except the .csv file with whois data are products of automatic processes. Please contact me (the email address in the files works) if you have any questions or concerns.

Update 2017-05-17: After running with the autofilling tables for approximately a month, and I must confess, extracting bad login attempts that didn't actually trigger the overload at semi-random but roughly daily intervals, I thought I'd check a few things about the catch. I already knew roughly how many hosts total, but how many were contactin us via IPv6? Let's see:

[Wed May 17 19:38:02] peter@skapet:~$ doas pfctl -t popflooders -T show | wc -l
5239
[Wed May 17 19:38:42] peter@skapet:~$ doas pfctl -t popflooders -T show | grep -c \: | wc -l
77

Meaning, that of a total 5239 miscreants trapped, only 77, or just short of 1.5 per cent tried contacting us via IPv6. The cybercriminals, or at least the literal bottom feeders like the pop3 password gropers, are still behind the times in a number of ways.

Update 2017-06-13: BSDCan 2017 is past, and the PF and networking tutorial with OpenBSD session had 19 people signed up for it. We made the slides available on the net here during the presentation and announced them on Twitter and elsewhere just after the session concluded. The revised tutorial was fairly well received, and it is likely that we will be offering roughly equivalent but not identical sessions at future BSD events or other occasions as demand dictates.

Update 2017-07-05: Updated the overload criteria for the longterm table to what I've had running for a while: max-src-conn 1, max-src-conn-rate 1/1.

Update 2017-10-07: I've decided to publish a bit more of the SSH bruteforcer data. The origin is from two gateways in my care, both with these entries in their pf.conf:

table persist counters file "/var/tmp/bruteforce"

block drop log (all) quick from <bruteforce>

supplemented with cron jobs that dump the current data to the file so the data survives a reboot. After years of advocating 24-hour expiry on blacklists, I recently changed my mind so, both hosts now run with 28-day expiry. For further severity on my part, the hosts also exchange updates to their bruteforce tables via cron jobs that dump table contents to file, fetch the partner's data and load into their own local table. In addition, a manual process extracts (at quasi-random but approximately daily intervals) addresses of failures that do not reach the limits and add those to the tables as well.

The data comes in three varieties: a raw address list, (with a #-prepended comment at the start) suitable for importing into such things as a PF table you block traffic from, the address list with the country code for each entry appended, and finally a summary of list entries per country code. All varieties are generated twice per hour.

by Peter N. M. Hansteen (noreply@blogger.com) atDecember 10, 2017 10:22 PM

October 24, 2017

NUUG Foundation

Reisestipend for studenter - 2018

NUUG Foundation utlyser reisestipender for 2018. Søknader kan sendes inn til enhver tid.

October 24, 2017 05:11 PM

October 23, 2017

Espen Braastad

ZFS NAS using CentOS 7 from tmpfs

Following up on the CentOS 7 root filesystem on tmpfs post, here comes a guide on how to run a ZFS enabled CentOS 7 NAS server (with the operating system) from tmpfs.

Hardware

HP ProLiant MicroServer
HP ProLiant MicroServer Remote Access Card
AMD Turion™ II Neo N54L Dual-Core Processor
2x 8 GB 1333 MHZ ECC memory modules
4x 4 TB SATA hard drives (HGST, Western Digital and Seagate)
Built-in Broadcom Limited NetXtreme BCM5723 Gigabit Ethernet adapter

Preparing the build environment

The disk image is built in macOS using Packer and VirtualBox. Virtualbox is installed using the appropriate platform package that is downloaded from their website, and Packer is installed using brew:

$ brew install packer

Building the disk image

Three files are needed in order to build the disk image; a Packer template file, an Anaconda kickstart file and a shell script that is used to configure the disk image after installation. The following files can be used as examples:

template.json (Packer template example file)
ks.cfg (Anaconda kickstart example file)
provision.sh (Provision shell script example file)

Create some directories:

$ mkdir ~work/centos-7-zfs/
$ mkdir ~work/centos-7-zfs/http/
$ mkdir ~work/centos-7-zfs/scripts/

Copy the files to these directories:

$ cp template.json ~work/centos-7-zfs/
$ cp ks.cfg ~work/centos-7-zfs/http/
$ cp provision.sh ~work/centos-7-zfs/scripts/

Modify each of the files to fit your environment.

Start the build process using Packer:

$ cd ~work/centos-7-zfs/
$ packer build template.json

This will download the CentOS 7 ISO file, start an HTTP server to serve the kickstart file and start a virtual machine using Virtualbox:

Packer installer screenshot

The virtual machine will boot into Anaconda and run through the installation process as specified in the kickstart file:

Anaconda installer screenshot

When the installation process is complete, the disk image will be available in the output-virtualbox-iso folder with the vmdk extension.

Packer done screenshot

The disk image is now ready to be put in initramfs.

Putting the disk image in initramfs

This section is quite similar to the previous blog post CentOS 7 root filesystem on tmpfs but with minor differences. For simplicity reasons it is executed on a host running CentOS 7.

Create the build directories:

$ mkdir /work
$ mkdir /work/newroot
$ mkdir /work/result

Export the files from the disk image to one of the directories we created earlier:

$ export LIBGUESTFS_BACKEND=direct
$ guestfish --ro -a packer-virtualbox-iso-1508790384-disk001.vmdk -i copy-out / /work/newroot/

Modify /etc/fstab:

$ cat > /work/newroot/etc/fstab << EOF
tmpfs       /         tmpfs    defaults,noatime 0 0
none        /dev      devtmpfs defaults         0 0
devpts      /dev/pts  devpts   gid=5,mode=620   0 0
tmpfs       /dev/shm  tmpfs    defaults         0 0
proc        /proc     proc     defaults         0 0
sysfs       /sys      sysfs    defaults         0 0
EOF

Disable selinux:

echo "SELINUX=disabled" > /work/newroot/etc/selinux/config

Disable clearing the screen on login failure to make it possible to read any error messages:

mkdir /work/newroot/etc/systemd/system/getty@.service.d
cat > /work/newroot/etc/systemd/system/getty@.service.d/noclear.conf << EOF
[Service]
TTYVTDisallocate=no
EOF

Now jump to the Initramfs and Result sections in the CentOS 7 root filesystem on tmpfs and follow those steps until the end when the result is a vmlinuz and initramfs file.

ZFS configuration

The first time the NAS server boots on the disk image, the ZFS storage pool and volumes will have to be configured. Refer to the ZFS documentation for information on how to do this, and use the following command only as guidelines.

Create the storage pool:

$ sudo zpool create data mirror sda sdb mirror sdc sdd

Create the volumes:

$ sudo zfs create data/documents
$ sudo zfs create data/games
$ sudo zfs create data/movies
$ sudo zfs create data/music
$ sudo zfs create data/pictures
$ sudo zfs create data/upload

Share some volumes using NFS:

zfs set sharenfs=on data/documents
zfs set sharenfs=on data/games
zfs set sharenfs=on data/music
zfs set sharenfs=on data/pictures

Print the storage pool status:

$ sudo zpool status
  pool: data
 state: ONLINE
  scan: scrub repaired 0B in 20h22m with 0 errors on Sun Oct  1 21:04:14 2017
config:

	NAME        STATE     READ WRITE CKSUM
	data        ONLINE       0     0     0
	  mirror-0  ONLINE       0     0     0
	    sdd     ONLINE       0     0     0
	    sdc     ONLINE       0     0     0
	  mirror-1  ONLINE       0     0     0
	    sda     ONLINE       0     0     0
	    sdb     ONLINE       0     0     0

errors: No known data errors

October 23, 2017 11:20 PM

October 20, 2017

NUUG news

Spørreundersøkelse blant medlemmer og interesserte 2017

NUUG vil gjerne vite mer om hva medlemmer og interesserte er opptatt av, og gjennomfører derfor en spørreundersøkelse. Vi håper så mange som mulig finner tid til å svare på den.

October 20, 2017 04:00 AM

October 03, 2017

Ole Aamot GNOME Development Blog

Free Internet Radio program in Catalan

GNOME Internet Radio Locator version 1.0.0 for GNOME 3 is available in Catalan.

You can download the freely available source code for the free GNOME Internet Radio Locator program from https://git.gnome.org/gnome-internet-radio-locator/ and binary packages for Debian GNU/Linux, Fedora Core, Ubuntu Linux and Mac OS X from http://people.gnome.org/~ole/gnome-internet-radio-locator/

On October 1, a referendum was held on whether Catalonia, an autonomous region of the northeast of Spain, should declare itself to be an independent country. The Spanish government has ruled the referendum illegal, and is taking action on a number of fronts to shut it down and to censor communications promoting it.

It is not very likely that the Spanish government is capable to shut down Catalan language translations for Free Software, but so far the Spanish government have tried to shut down .cat domains that announced the Catalonian referendum before October 1.

A referendum is a democratic tool for citizens and should not be shut down by the police.

90% voted for a Free Catalonia on October 1, 2017, despite claims by the Spanish prime minister that “no referendum was held in Catalonia”.

by oleaamot atOctober 03, 2017 11:51 AM

September 28, 2017

Ole Aamot GNOME Development Blog

GNOME Internet Radio Locator 1.0.0 for GNOME 3

GNOME Internet Radio Locator 1.0.0 for GNOME 3 is available with map marker popups for Internet radio stations in 76 cities and with text-based location search for 86 Internet Radio stations. The following 76 cities are supported in version 1.0.0:

Adelaide, Australia
Auckland, New Zealand
Austin, Texas
Ayr, Scotland
Bergen, Norway
Berkeley, California
Bern, Switzerland
Boston, Massachusetts
Bristol, United Kingdom
Brno, Czech Republic
Bronx, New York
Brooklyn, New York
Bruxelles, Belgium
Budapest, Hungary
Buenos Aires, Argentina
Calgary, Canada
Cambridge, United Kingdom
Cape Town, South Africa
Centralia, District of Columbia
Chapel Hill, North Carolina
Chicago, Illinois
Cleveland, Ohio
Coimbra, Portugal
Copenhagen, Denmark
Cornwall, United Kingdom
Dublin, Ireland
Gent, Belgium
Guatemala City, Guatemala
Hammond, Louisiana
Honolulu, Hawaii
Houston, Texas
Kingston, Canada
Kristiansand, Norway
Leeds, United Kingdom
London, United Kingdom
Long Island, New York
Los Angeles, California
Lund, Sweden
Manchester, United Kingdom
Memphis, Tennessee
Minneapolis, Minnesota
México City, México
Narvik, Norway
Nashville, Tennessee
New Orleans, Louisiana
New York City, New York
Newcastle, Australia
Nicosia, Cyprus
Nottingham, United Kingdom
Oslo, Norway
Oswego, New York
Ottawa, Canada
Oxford, United Kingdom
Palo Alto, California
Paris, France
Phoenix, Arizona
Pisa, Italy
Pittsburgh, Pennsylvania
Portland, Oregon
Reykjavik, Iceland
Rochester, Michigan
Salford, United Kingdom
San Marcos, Texas
Santiago, Chile
Seattle, Washington
St. Pölten, Austria
Stockholm, Sweden
Sydney, Canada
São Paulo, Brazil
Toronto, Canada
Trondheim, Norway
Tuscaloosa, Alabama
Washington, District of Columbia
Waterloo, Canada
York, United Kingdom
Zürich, Switzerland

If you locate one of these cities on the map and click on the map above the city, you should be able to create a map marker. If you click on the map marker, you will be able to listen to Internet radio from that city. If you click and create several city markers, you can switch between Internet radio stations from different locations on the map.

GNOME Internet Radio Locator 1.0.0 for GNOME 3

You can download the gnome-internet-radio-locator 1.0.0 development tree from https://git.gnome.org/gnome-internet-radio-locator

Source
https://download.gnome.org/sources/gnome-internet-radio-locator/1.0/gnome-internet-radio-locator-1.0.0.tar.xz

Debian GNU/Linux unstable i386
https://people.gnome.org/~ole/gnome-internet-radio-locator/debian/gnome-internet-radio-locator_1.0.0-1_i386.deb

Fedora 26 x86_64
https://people.gnome.org/~ole/gnome-internet-radio-locator/gnome-internet-radio-locator-1.0.0-1.fc26.x86_64.rpm

Ubuntu 17.04 amd64
https://people.gnome.org/~ole/gnome-internet-radio-locator/gnome-internet-radio-locator_1.0.0-1_amd64.deb

This release is built on GTK+ 3.0, GNOME Maps, libchamplain and gstreamer (gst-player).

Enjoy Free Internet Radio.

by oleaamot atSeptember 28, 2017 04:56 PM

July 31, 2017

Dag-Erling Smørgrav

Yes, all men

Since Susan Fowler blogged about her experience at Uber in February, the debate about sexism in tech has dominated IT and business news. Note that this debate is not new, and Fowler’s story isn’t all that different from many other stories we’ve heard before. It’s just that for once, finally, people were paying attention, and There Were Consequences. Then matters escalated in June with a string of revelations about sexism—not just discrimination, but full on sexual harassment.

Then came the apologies. Let me tell you about the apologies. The average response from a manager or venture capitalist accused of sexism went something like this:

I apologized unreservedly for my treatment of X. I realize now that my innocent jokes may have been misinterpreted. I’m actually a pretty nice guy, and X’s refusal to sleep with me had no impact whatsoever on my decision not to invest in her startup.

Then came the White Knights:

As a VC, I’m appalled to hear about my colleagues’ behavior towards women. I would like to reassure you all that Not All Men are like that. I myself am actually a pretty nice guy and completely innocent in all this.

Guys, it’s time to face the music. We have all been That Guy. We have all made sexist jokes, or laughed when others made them, or stood by silently while our male bosses, coworkers and colleagues ignored or patronized or belittled or humiliated women. We have all benefited from a system that eliminates close to 50% of our competition before the race has even started.

We are all complicit. We are all guilty.

So what do we do? Where do we go next?

First, take a deep breath, do a little soul-searching, and re-read that paragraph until any impulse, however minor, to say to yourself “OK, but not me” is gone. Yes, you too.

Next, if you’ve ever acted inappropriately towards a female coworker or friend or acquaintance, or stood by silently while others did, consider apologizing.

Third, vow to never do so again, and work hard to keep that vow. Respect the women around you as much as you respect the men. If someone around you acts or speaks inappropriately, speak up, even if there are no women present. Be proactive: make sure that women are given equal opportunity to join those career-building projects, and are included in those backstage chats where decisions are made. If you are hiring, seek out female candidates, keeping in mind that women have a tendency to underestimate their abilities and experience just as men have a tendency to overestimate them. If you are teaching, encourage and mentor female students. Reach out to them if they seem discouraged. Don’t wait until they drop out.

Open your eyes. Open your ears. Listen to the women around you. Believe them. Respect them. Be someone they can vent to and someone they can count on for support when push comes to shove.

You will slip up. When you do, apologize and vow to do better.

As a man, you are, and always have been, part of the problem. Accept it, and start being part of the solution.

by Dag-Erling Smørgrav atJuly 31, 2017 09:56 PM

April 19, 2017

Holder de ord

BETALT JOBB: Kategorisering av valgløfter

Interessert i politikk? Er du student som ønsker å tjene noen ekstra slanter ved siden av studiene?

Holder de ord er en partipolitisk uavhengig organisasjon med mål om å gjøre det enklere å følge med på norsk stortingspolitikk. Blant tjenestene vi tilbyr er en fullstendig løftedatabase som per i dag består av alle løftene fra de åtte stortingspartienes partiprogram for periodene 2009-2013 og 2013-2017.

I forbindelse med valgåret 2017 skal databasen oppdateres med løftene til alle partiene som er representert på Stortinget. Vårens siste landsmøte avsluttes 21. mai 2017. Da vil alle åtte parti på Stortinget ha vedtatt nye program for perioden 2017-2021. Totalt vil det erfaringsmessig da ha kommet om lag 7.000 nye løfter. Alle disse løftene skal inn i Holder de ords løftedatabase.

Jobbeskrivelse

Dette er en manuell jobb. Hvert enkelt løfte i partiprogrammene kopieres over i et excel-ark og gis kategorier etter Stortingets kategorisystem. Noe omskrivningsarbeid må påregnes slik at løftene kan stå på egne ben, uten nødvendig kontekst. Dette importeres så til Holder de ords nettdatabase.

I tillegg innebærer jobben å grovsortere aktuelle løfter til bruk i Holder de ords chat bot. Egne retningslinjer for disse vil bli gitt. Denne delen av jobben vil ikke påvirke den totale arbeidsmengden.

Vilkår

Det inngås kontrakt hvor det avtales lønn per partiprogram. I utgangspunktet ser vi gjerne at den som påtar seg oppdraget kategoriserer alle eller flere enn to partiprogram. Ved for sen levering kan dagbøter påløpe.

Søknadsfrist: fortløpende
Oppstart: umiddelbart
Leveringsfrist: 1. juni 2017

Send en kort søknad som epost til Tiina Ruohonen og Hanna Tranås merket “Valgløfter” i emnefeltet.

by Hanna Tranås (hanna@holderdeord.no) atApril 19, 2017 07:57 PM

February 13, 2017

Mimes brønn

En innsynsbrønn full av kunnskap

Mimes brønn er en nettjeneste som hjelper deg med å be om innsyn i offentlig forvaltning i tråd med offentleglova og miljøinformasjonsloven. Tjenesten har et offentlig tilgjengelig arkiv over alle svar som er kommet på innsynsforespørsler, slik at det offentlige kan slippe å svare på de samme innsynshenvendelsene gang på gang. Du finner tjenesten på

https://www.mimesbronn.no/

I følge gammel nordisk mytologi voktes kunnskapens kilde av Mime og ligger under en av røttene til verdenstreet Yggdrasil. Å drikke av vannet i Mimes brønn ga så verdifull kunnskap og visdom at den unge guden Odin var villig til å gi et øye i pant og bli enøyd for å få lov til å drikke av den.

Nettstedet vedlikeholdes av foreningen NUUG og er spesielt godt egnet for politisk interesserte personer, organisasjoner og journalister. Tjenesten er basert på den britiske søstertjenesten WhatDoTheyKnow.com, som allerede har gitt innsyn som har resultert i dokumentarer og utallige presseoppslag. I følge mySociety for noen år siden gikk ca 20 % av innsynshenvendelsene til sentrale myndigheter via WhatDoTheyKnow. Vi i NUUG håper NUUGs tjeneste Mimes brønn kan være like nyttig for innbyggerne i Norge.

I helgen ble tjenesten oppdatert med mye ny funksjonalitet. Den nye utgaven fungerer bedre på små skjermer, og viser nå leveringsstatus for henvendelsene slik at innsender enklere kan sjekke at mottakers epostsystem har bekreftet mottak av innsynshenvendelsen. Tjenesten er satt opp av frivillige i foreningen NUUG på dugnad, og ble lansert sommeren 2015. Siden den gang har 121 brukere sendt inn mer enn 280 henvendelser om alt fra bryllupsutleie av Operaen og forhandlinger om bruk av Norges topp-DNS-domene .bv til journalføring av søknader om bostøtte, og nettstedet er en liten skattekiste av interessant og nyttig informasjon. NUUG har knyttet til seg jurister som kan bistå med å klage på manglende innsyn eller sviktende saksbehandling.

– «NUUGs Mimes brønn var uvurderlig da vi lyktes med å sikre at DNS-toppdomenet .bv fortsatt er på norske hender,» forteller Håkon Wium Lie.

Tjenesten dokumenterer svært sprikende praksis i håndtering av innsynshenvendelser, både når det gjelder responstid og innhold i svarene. De aller fleste håndteres raskt og korrekt, men det er i flere tilfeller gitt innsyn i dokumenter der ansvarlig etat i ettertid ønsker å trekke innsynet tilbake, og det er gitt innsyn der sladdingen har vært utført på en måte som ikke skjuler informasjonen som skal sladdes.

– «Offentlighetsloven er en bærebjelke for vårt demokrati. Den bryr seg ikke med hvem som ber om innsyn, eller hvorfor. Prosjektet Mimes brønn innebærer en materialisering av dette prinsippet, der hvem som helst kan be om innsyn og klage på avslag, og hvor dokumentasjon gjøres offentlig. Dette gjør Mimes Brønn til et av de mest spennende åpenhetsprosjektene jeg har sett i nyere tid.» forteller mannen som fikk åpnet opp eierskapsregisteret til skatteetaten, Vegard Venli.

Vi i foreningen NUUG håper Mimes brønn kan være et nyttig verktøy for å holde vårt demokrati ved like.

by Mimes Brønn atFebruary 13, 2017 02:07 PM

January 28, 2017

NUUG Foundation

Reisestipend for studenter - 2017

NUUG Foundation utlyser reisestipender for 2017. Søknader kan sendes inn til enhver tid.

January 28, 2017 12:57 PM

January 06, 2017

Espen Braastad

CentOS 7 root filesystem on tmpfs

Several years ago I wrote a series of posts on how to run EL6 with its root filesystem on tmpfs. This post is a continuation of that series, and explains step by step how to run CentOS 7 with its root filesystem in memory. It should apply to RHEL, Ubuntu, Debian and other Linux distributions as well. The post is a bit terse to focus on the concept, and several of the steps have potential for improvements.

The following is a screen recording from a host running CentOS 7 in tmpfs:

Sensor

Build environment

A build host is needed to prepare the image to boot from. The build host should run CentOS 7 x86_64, and have the following packages installed:

yum install libvirt libguestfs-tools guestfish

Make sure the libvirt daemon is running:

systemctl start libvirtd

Create some directories that will be used later, however feel free to relocate these to somewhere else:

mkdir -p /work/initramfs/bin
mkdir -p /work/newroot
mkdir -p /work/result

Disk image

For simplicity reasons we’ll fetch our rootfs from a pre-built disk image, but it is possible to build a custom disk image using virt-manager. I expect that most people would like to create their own disk image from scratch, but this is outside the scope of this post.

Use virt-builder to download a pre-built CentOS 7.3 disk image and set the root password:

virt-builder centos-7.3 -o /work/disk.img --root-password password:changeme

Export the files from the disk image to one of the directories we created earlier:

guestfish --ro -a /work/disk.img -i copy-out / /work/newroot/

Clear fstab since it contains mount entries that no longer apply:

echo > /work/newroot/etc/fstab

SELinux will complain about incorrect disk label at boot, so let’s just disable it right away. Production environments should have SELinux enabled.

echo "SELINUX=disabled" > /work/newroot/etc/selinux/config

Disable clearing the screen on login failure to make it possible to read any error messages:

mkdir /work/newroot/etc/systemd/system/getty@.service.d
cat > /work/newroot/etc/systemd/system/getty@.service.d/noclear.conf << EOF
[Service]
TTYVTDisallocate=no
EOF

Initramfs

We’ll create our custom initramfs from scratch. The boot procedure will be, simply put:

Fetch kernel and a custom initramfs.
Execute kernel.
Mount the initramfs as the temporary root filesystem (for the kernel).
Execute /init (in the initramfs).
Create a tmpfs mount point.
Extract our CentOS 7 root filesystem to the tmpfs mount point.
Execute switch_root to boot on the CentOS 7 root filesystem.

The initramfs will be based on BusyBox. Download a pre-built binary or compile it from source, put the binary in the initramfs/bin directory. In this post I’ll just download a pre-built binary:

wget -O /work/initramfs/bin/busybox https://www.busybox.net/downloads/binaries/1.26.1-defconfig-multiarch/busybox-x86_64

Make sure that busybox has the execute bit set:

chmod +x /work/initramfs/bin/busybox

Create the file /work/initramfs/init with the following contents:

#!/bin/busybox sh

# Dump to sh if something fails
error() {
	echo "Jumping into the shell..."
	setsid cttyhack sh
}

# Populate /bin with binaries from busybox
/bin/busybox --install /bin

mkdir -p /proc
mount -t proc proc /proc

mkdir -p /sys
mount -t sysfs sysfs /sys

mkdir -p /sys/dev
mkdir -p /var/run
mkdir -p /dev

mkdir -p /dev/pts
mount -t devpts devpts /dev/pts

# Populate /dev
echo /bin/mdev > /proc/sys/kernel/hotplug
mdev -s

mkdir -p /newroot
mount -t tmpfs -o size=1500m tmpfs /newroot || error

echo "Extracting rootfs... "
xz -d -c -f rootfs.tar.xz | tar -x -f - -C /newroot || error

mount --move /sys /newroot/sys
mount --move /proc /newroot/proc
mount --move /dev /newroot/dev

exec switch_root /newroot /sbin/init || error

Make sure it is executable:

chmod +x /work/initramfs/init

Create the root filesystem archive using tar. The following command also uses xz compression to reduce the final size of the archive (from approximately 1 GB to 270 MB):

cd /work/newroot
tar cJf /work/initramfs/rootfs.tar.xz .

Create initramfs.gz using:

cd /work/initramfs
find . -print0 | cpio --null -ov --format=newc | gzip -9 > /work/result/initramfs.gz

Copy the kernel directly from the root filesystem using:

cp /work/newroot/boot/vmlinuz-*x86_64 /work/result/vmlinuz

Result

The /work/result directory now contains two files with file sizes similar to the following:

ls -lh /work/result/
total 277M
-rw-r--r-- 1 root root 272M Jan  6 23:42 initramfs.gz
-rwxr-xr-x 1 root root 5.2M Jan  6 23:42 vmlinuz

These files can be loaded directly in GRUB from disk, or using iPXE over HTTP using a script similar to:

#!ipxe
kernel http://example.com/vmlinuz
initrd http://example.com/initramfs.gz
boot

January 06, 2017 08:34 PM

November 12, 2016

Anders Einar Hilden

Perl Regexp Oneliners and UTF-8

For my project to find as many .no domains as possible, I needed a regexp for extracting valid domains. This task is made more fun by the inclusion of Norwegian and Sami characters in the set of valid characters.

In addition to [a-z0-9\-], valid dot-no domains can contain the Norwegian æ (ae), ø (o with stroke) and å (a with ring above) (Stargate, anyone?) and a number of Sami characters. ŧ (t with stroke), ç (c with cedilla) and ŋ (simply called “eng”) are some of my favourites.

The following code will print only the first match per line, and uses ŧ directly in the regexp.

echo "fooŧ.no baŧ.no" | perl -ne 'if(/([a-zŧ]{2,63}\.no)/ig) { print $1,"\n"; }'
fooŧ.no

If we replace if with while we will print any match found in the whole line.

echo "fooŧ.no baŧ.no" | perl -ne 'while(/([a-zŧ]{2,63}\.no)/ig) { print $1,"\n"; }'
fooŧ.no
baŧ.no

Because I’m afraid the regexp (specifically the non-ASCII characters) may be mangled by being saved and moved between systems, I want to write the Norwegian and Sami characters using their Unicode code points. Perl has support for this using \x{<number>} (see perl unicode)

echo "fooŧ.no baŧ.no" | perl -CSD -ne 'while(/([a-z\x{167}]{2,63}\.no)/ig) { print $1,"\n"; }'
fooŧ.no
baŧ.no

When using code points, I have to specify -CSD for the matching to work. I am not really sure why this is required. If you can explain, please comment or tell my by other means. As you can read in perlrun, -CSD specifies that STDIN, STDOUT, STDERR and all input and output streams should be treated as being UTF-8.

Another problem is that if this last solution is is fed invalid UTF-8, it will die fatally and stop processing input.

Malformed UTF-8 character (fatal) at -e line 1, <> line X.

To prevent this happening I currently sanitize my dirty input using iconv -f utf-8 -t utf-8 -c. If you have a better solution for this, Perl or otherwise, please tell me!.

A simple regexp would match the valid characters for a length between 2 and 63 followed by .no. However, I wanted only and all “domains under .no” as counted by Norid in their statistics. Norids definition of “domains under .no” are all the domains directly under .no, but also domains under category domains i.e. ohv.oslo.no and ola.priv.no. To get comparable results, I have to collect both *.no and *.<category domain>.no domains when scraping data.

The resulting “oneliner” I use is this…. It once was a oneliner, but with more than 10k characters in the regexp it was hard to manage. The resulting script builds up a regexp that is valid for all Norwegian domains using a list of valid category domains, all valid characters and other rules for .no domains.

November 12, 2016 10:00 PM

September 18, 2016

Dag-Erling Smørgrav

Not up to our usual standards

For a few years now, I’ve been working on and off on a set of libraries which collect cryptography- and security-related code I’ve written for other projects as well as functionality which is not already available under a permissive license, or where existing implementations do not meet my expectations of cleanliness, readability, portability and embeddability.

(Aside: the reasons why this has taken years, when I initially expected to publish the first release in the spring or summer of 2014, are too complex to explain here; I may write about them at a later date. Keywords are health, family and world events.)

Two of the major features of that collection are the OATH Authentication Methods (which includes the algorithm used by Google Authenticator and a number of commercial one-time code fobs) and the Common Platform Enumeration, part of the Security Content Automation Protocol. I implemented the former years ago for my employer, and it has languished in the OpenPAM repository since 2012. The latter, however, has proven particularly elusive and frustrating, to the point where it has existed for two years as merely a header file and a set of mostly empty functions, just to sketch out the API. I decided to have another go at it yesterday, and actually made quite a bit of progress, only to hit the wall again. And this morning, I realized why.

The CPE standard exists as a set of NIST Interagency reports: NISTIR 7695 (naming), NISTIR 7696 (name matching), NISTIR 7697 (dictionary) and NISTIR 7698 (applicability language). The one I’ve been struggling with is 7695—it is the foundation for the other three, so I can’t get started on them until I’m done with 7695.

It should have been a breeze. On the surface, the specification seems quite thorough: basic concepts, representations, conversion between representations (including pseudocode). You know the kind of specification that you can read through once, then sit down at the computer, start from the top, and code your way down to the bottom? RFC 4226 and RFC 6238, which describe OATH event-based and time-based one-time passwords respectively, are like that. NISTIR 7695 looks like it should be. But it isn’t. And I’ve been treating it like it was, with my nose so close to the code that I couldn’t see the big picture and realize that it is actually not very well written at all, and that the best way to implement it is to read it, understand it, and then set it aside before coding.

One sign that NISTIR 7695 is a bad specification is the pseudocode. It is common for specifications to describe algorithms, protocols and / or interfaces in the normative text and provide examples, pseudocode and / or a reference implementation (sometimes of dubious quality, as is the case for RFC 4226 and RFC 6238) as non-normative appendices. NISTIR 7695, however, eschews natural-language descriptions and includes pseudocode and examples in the normative text. By way of example, here is the description of the algorithm used to convert (“bind”, in their terminology) a well-formed name to a formatted string, in its entirety:

6.2.2.1 Summary of algorithm

The procedure iterates over the eleven allowed attributes in a fixed order. Corresponding attribute values are obtained from the input WFN and conversions of logical values are applied. A result string is formed by concatenating the attribute values separated by colons.

This is followed by one page of pseudocode and two pages of examples. But the examples are far from exhaustive; as unit tests, they wouldn’t even cover all of the common path, let alone any of the error handling paths. And the pseudocode looks like it was written by someone who learned Pascal in college thirty years ago and hasn’t programmed since.

The description of the reverse operation, converting a formatted string to a well-formed name, is slightly better in some respects and much worse in others. There is more pseudocode, and the examples include one—one!—instance of invalid input… but the pseudocode includes two functions—about one third of the total—which consist almost entirely of comments describing what the functions should do, rather than actual code.

You think I’m joking? Here is one of them:

function get_comp_fs(fs,i)
  ;; Return the i’th field of the formatted string. If i=0,
  ;; return the string to the left of the first forward slash.
  ;; The colon is the field delimiter unless prefixed by a
  ;; backslash.
  ;; For example, given the formatted string:
  ;; cpe:2.3:a:foo:bar\:mumble:1.0:*:...
  ;; get_comp_fs(fs,0) = "cpe"
  ;; get_comp_fs(fs,1) = "2.3"
  ;; get_comp_fs(fs,2) = "a"
  ;; get_comp_fs(fs,3) = "foo"
  ;; get_comp_fs(fs,4) = "bar\:mumble"
  ;; get_comp_fs(fs,5) = "1.0"
  ;; etc.
end.

This function shouldn’t even exist. It should just be a lookup in an associative array, or a call to an accessor if the pseudocode was object-oriented. So why does it exist? Because the main problem with NISTIR 7695, which I should have identified on my first read-through but stupidly didn’t, is that it assumes that implementations would use well-formed names—a textual representation of a CPE name—as their internal representation. The bind and unbind functions, which should be described in terms of how to format and parse URIs and formatted strings, are instead described in terms of how to convert to and from WFNs. I cannot overstate how wrong this is. A specification should never describe a particular internal representation, except in a non-normative reference implementation, because it prevents conforming implementations from choosing more efficient representations, or representations which are better suited to a particular language and environment, and because it leads to this sort of nonsense.

So, is the CPE naming specification salvageable? Well, it includes complete ABNF grammars for URIs and formatted strings, which is good, and a partial ABNF grammar for well-formed names, which is… less good, but fixable. It also explains the meanings of the different fields; it would be useless otherwise. But apart from that, and the boilerplate at the top and bottom, it should be completely rewritten, including the pseudocode and examples, which should reference fictional, rather than real, vendors and products. Here is how I would structure it (text in italic is carried over from the original):

Introduction
1.1. Purpose and scope
1.2. Document structure
1.3. Document conventions
1.4. Relationship to existing specifications and standards
Definitions and abbreviations
Conformance
CPE data model
4.1 Required attributes
4.2 Optional attributes
4.3 Special attribute values
Textual representations
5.1. Well-formed name
5.2. URI
5.3. Formatted string
API
6.1. Creating and destroying names
6.2. Setting and getting attributes
6.3. Binding and unbinding
Non-normative examples
7.1. Valid and invalid attribute values
7.2. Valid and invalid well-formed names
7.3. Valid and invalid URIs
7.4. Valid and invalid formatted strings
Non-normative pseudo-code
References
Change log

I’m still going to implement CPE naming, but I’m going to implement it the way I think the standard should have been written, not the way it actually was written. Amusingly, the conformance chapter is so vague that I can do this and still claim conformance with the Terrible, Horrible, No Good, Very Bad specification. And it should only take a few hours.

By the way, if anybody from MITRE or NIST reads this and genuinely wants to improve the specification, I’ll be happy to help.

PS: possibly my favorite feature of NISTIR 7695, and additional proof that the authors are not programmers: the specification mandates that WFNs are UTF-8 strings, which are fine for storage and transmission but horrible to work with in memory. But in the next sentence, it notes that only characters with hexadecimal values between x00 and x7F may be used, and subsequent sections further restrict the set of allowable characters. In case you didn’t know, the normalized UTF-8 representation of a sequence of characters with hexadecimal values between x00 and x7F is identical, bit by bit, to the ASCII representation of the same sequence.

by Dag-Erling Smørgrav atSeptember 18, 2016 01:54 PM

August 24, 2016

Nicolai Langfeldt

The most important thing about Apache

The most important thing you should know about Apache, the web-server, is that if you feel stupid after trying to get something working on it - this is perfectly normal - and does not mean you are stupid.

Try NGiNX next time. If it supports the thing you need.

by Nicolai Langfeldt (noreply@blogger.com) atAugust 24, 2016 08:26 PM

July 15, 2016

Mimes brønn

Hvem har drukket fra Mimes brønn?

Mimes brønn har nå vært oppe i rundt et år. Derfor vi tenkte det kunne være interessant å få en kortfattet statistikk om hvordan tjenesten er blitt brukt.

I begynnelsen av juli 2016 hadde Mimes brønn 71 registrerte brukere som hadde sendt ut 120 innsynshenvendelser, hvorav 62 (52%) var vellykkede, 19 (16%) delvis vellykket, 14 (12%) avslått, 10 (8%) fikk svar at organet ikke hadde informasjonen, og 12 henvendelser (10%; 6 fra 2016, 6 fra 2015) fortsatt var ubesvarte. Et fåtall (3) av hendvendelsene kunne ikke kategoriseres. Vi ser derfor at rundt to tredjedeler av henvendelsene var vellykkede, helt eller delvis. Det er bra!

Tiden det tar før organet først sender svar varierer mye, fra samme dag (noen henvendelser sendt til Utlendingsnemnda, Statens vegvesen, Økokrim, Mediatilsynet, Datatilsynet, Brønnøysundregistrene), opp til 6 måneder (Ballangen kommune) eller lenger (Stortinget, Olje- og energidepartementet, Justis- og beredskapsdepartementet, UDI – Utlendingsdirektoratet, og SSB har mottatt innsynshenvendelser som fortsatt er ubesvarte). Gjennomsnittstiden her var et par uker (med unntak av de 12 tilfellene der det ikke har kommet noe svar). Det følger av offentlighetsloven § 29 første ledd at henvendelser om innsyn i forvaltningens dokumenter skal besvares «uten ugrunnet opphold», noe som ifølge Sivilombudsmannen i de fleste tilfeller skal fortolkes som «samme dag eller i alle fall i løpet av 1-3 virkedager». Så her er det rom for forbedring.

Klageretten (offentleglova § 32) ble benyttet i 20 av innsynshenvendelsene. I de fleste (15; 75%) av tilfellene førte klagen til at henvendelsen ble vellykket. Gjennomsnittstiden for å få svar på klagen var en måned (med unntak av 2 tillfeller, klager sendt til Statens vegvesen og Ruter AS, der det ikke har kommet noe svar). Det er vel verdt å klage, og helt gratis! Sivilombudsmannen har uttalt at 2-3 uker ligger over det som er akseptabel saksbehandlingstid for klager.

Flest henvendelser var blitt sendt til Utenriksdepartementet (9), tett etterfulgt av Fredrikstad kommune og Brønnøysundregistrene. I alt ble henvendelser sendt til 60 offentlige myndigheter, hvorav 27 ble tilsendt to eller flere. Det står over 3700 myndigheter i databasen til Mimes brønn. De fleste av dem har dermed til gode å motta en innsynshenvendelse via tjenesten.

Når vi ser på hva slags informasjon folk har bedt om, ser vi et bredt spekter av interesser; alt fra kommunens parkeringsplasser, reiseregninger der statens satser for overnatting er oversteget, korrespondanse om asylmottak og forhandlinger om toppdomenet .bv, til dokumenter om Myanmar.

Myndighetene gjør alle mulige slags ting. Noe av det gjøres dårlig, noe gjør de bra. Jo mer vi finner ut om hvordan myndighetene fungerer, jo større mulighet har vi til å foreslå forbedringer på det som fungerer dårlig… og applaudere det som bra. Er det noe du vil ha innsyn i, så er det bare å klikke på https://www.mimesbronn.no/ og så er du i gang

by Mimes Brønn atJuly 15, 2016 03:56 PM

June 01, 2016

Kevin Brubeck Unhammer

Maskinomsetjing vs NTNU-eksaminator

Twitter-brukaren @IngeborgSteine fekk nyleg ein del merksemd då ho tvitra eit bilete av nynorskutgåva av økonomieksamenen sin ved NTNU:

Dette var min økonomieksamen på "nynorsk". #nynorsk #noregsmållag #kvaialledagar https://t.co/RjCKSU2Fyg—
Ingeborg Steine (@IngeborgSteine) May 30, 2016

Kreative nyvinningar som *kvisleis og alle dialektformene og arkaismane ville vore usannsynlege å få i ei maskinomsett utgåve, så då lurte eg på kor mykje betre/verre det hadde blitt om eksaminatoren rett og slett hadde brukt Apertium i staden? Ingeborg Steine var så hjelpsam at ho la ut bokmålsutgåva, så då får me prøva

Ingen kvisleis og fritt for tær og fyr, men det er heller ikkje perfekt: Visse ord manglar frå ordbøkene og får dermed feil bøying, teller blir tolka som substantiv, ein anna maskin har feil bøying på førsteordet (det mangla ein regel der) og at blir ein stad tolka som adverb (som fører til det forunderlege fragmentet det verta at anteke tilvarande). I tillegg blir språket gjenkjent som tatarisk av nettsida, så det var kanskje litt tung norsk? Men desse feila er ikkje spesielt vanskelege å retta på – utviklingsutgåva av Apertium gir no:

Det er enno eit par småting som kunne vore retta, men det er allereie betre enn dei fleste eksamenane eg fekk utdelt ved UiO …

by k atJune 01, 2016 09:45 AM

April 02, 2016

Thomas Sødring

Choices made on the road to 0.1

You can drive yourself mad wondering if you made the right choice with regards to technology. This really is a difficult question to answer as you have to pick components that have longevity and that are in widespread use. The truth is that you just have to pick something and go with it. I think about popularity of libraries, how active development is etc before I make a choice but it’s not easy to just decide. Any component I use now will follow the project and code for a long time going forward.

This week I was wrestling with AngularJS and ReactJS. Basically it boils down to whether or not I go with Google or Facebook. I picked up some cheap courses on Angular and that kinda made that decision. I’m not really that bothered by the GUI side of things at the moment, but I do need an administrative GUI and would like to have an idea how a proof-of-concept GUI would look. Given that this is a REST service, it will be possible to swap Angular out with whatever you want anyway. It is a very time consuming process trying to figure these things out.

The last month has been spent wondering how I should structure the project. If I get the foundation wrong, it will have a negative effect on the project. Baeldung has an interesting project structure with a clean definition of modules and what should be within the modules. This quickly became the basis of my project structure. I kept coming across jhipster and after days of hassle (installing npm, bower, yo) getting it installed I managed to get an interesting project setup. What I learnt from the jhipster sample app was support for swagger, metrics, spring-security, angulaJS and yaml project configuration. I was initially unable to get the jhipster app to run so I have spent the time studying the code and structure and gradually copied elements over to my project. This has resulted in the nikita code base supporting swagger, the introduction of metrics support, and spring security user configuration, all copied from the jhipster sample application.

This approach has really saved me a lot of time and answered many questions about spring and spring-based applications. I have learnt so much from this approach. A plus with such an approach where I try to study best practices is that I hopefully will end up with a good project structure and robust code. A negative is that I’m learning as I go along. Ideally I’d sit down and figure everything out in advance but I think that’s the primary reason why it has been difficult to move this project forward over the last couple of years, I never had my own concrete project structure to work with and was unsure how to proceed.

I also switched coding from Eclipse to Eclipse STS to IntelliJ Idea. I never seemed to be able to get things working nicely in Eclipse and STS. I always ended up with issues like it wasn’t possible to find source code when debugging or download sources and documentation didn’t work properly. I spent a lot of time on stackexchange but it really felt like a waste of time and I didn’t have an environment I felt comfortable and productive in. Idea has been a dream to work with. It just does things intuitively and the integration with git has allowed me to push code and changes quickly to githhub. I have never been so impressed with an IDE as I have been with Idea. It just seems to make sense.

I was also able to confirm that OData support is still in the draft version of Noark 5 v4 and will more than likely be in the final version. This complicates development of the REST-service significantly but I think I will solve this in the codebase by supporting two apis, one with OData and one without. The reason for this is that OData support requires me to handle all incoming HTTP requests manually. To be honest I am unsure about the usefulness of OData in a running installation, but if the standard specifies it then we simply will have to implement it. There is very little REST OData support in the java ecosystem, but there is something available that we can use.

Currently the code is very much a pre alpha version of v0.1. It is mainly a working project structure with the above mentioned libraries and with the domain model copied in and the fonds object is accessible via a REST controller. Don’t expect the code to work until it hits the v0.1 mark as I am updating it continuously. You can check out the code from the github repository.

The post Choices made on the road to 0.1 appeared first on Arkivets rolle i en tjenesteorientert arkitektur.

by tsodring atApril 02, 2016 05:38 AM

April 01, 2016

Thomas Sødring

Current project structure

One of the main challenges with this project is that I am not in a position to work on it full time. In the last month I have probably spent 80 hours and half of that is coming from my own free time. So whatever time I do have has to be spent wisely. I have few days to thoroughly explore issues and threads of thoughts are split up over several days.

In the last month I have made some interesting progress. I have spent the time working on the project structure and have moved files around quite a lot.

Currently the project is a multi module maven project with the following modules

core-client
core-common
core-conversion
core-extraction
core-webapp

core-client is where most of the domain modelling of Noark 5 can be found. All persistence related objects are here, DTO’s etc.

core-common contains a lot of common functionality related to REST handling etc. This is code that could be reused in other Noark 5 REST related projects.

core-conversion will be a REST-service that can convert documents from a production format to an archive format. I will only implement integration to LibreOffice but it is easily imaginable to implement integration to MS Office. I haven’t started this yet.

core-extraction will be a standalone executable jar that can extract the contents of the core in accordance with the extraction rules. Currently a weak arkivstruktur.xml generator has been implemented and that’s just to show a proof-of-concept.

core-webapp is the actual web application that is a spring-boot application and starts up a REST service.

Another module that needs to be implemented will be core-postjournal that talks to the database and publishes postjournal in various formats. Integration with altinn and digipost etc (core-dispatcher) all are obvious candidates for work, but currently the project needs a clear defined roadmap so these can all come later.

All the modules are encapsulated inside a parent module called nikita-noark5-core.

The post Current project structure appeared first on Arkivets rolle i en tjenesteorientert arkitektur.

by tsodring atApril 01, 2016 01:44 PM

October 18, 2015

Anders Nordby

Fighting spam with SpamAssassin, procmail and greylisting

On my private server we use a number of measures to stop and prevent spam from arriving in the users inboxes: - postgrey (greylisting) to delay arrival (hopefully block lists will be up to date in time to stop unwanted mail, also some senders do not retry) - SpamAssasin to block mails by scoring different aspects of the emails. Newer versions of it has URIBL (domain based, for links in the emails) in addtition to the tradional RBL (IP based) block lists. Which works better. I also created my own URIBL block list which you can use, dbl.fupp.net. - Procmail. For user on my server, I recommend this procmail rule: :0 * ^X-Spam-Status: Yes .crapbox/ It will sort emails that has a score indicating it is spam into mailbox "crapbox". - blocking unwanted and dangerous attachments, particularly for Windows users.

by Anders (noreply@blogger.com) atOctober 18, 2015 01:09 PM

April 23, 2015

Kevin Brubeck Unhammer

Orddelingsomsetjing

I førre innlegg i denne serien gjekk eg kort gjennom ymse metodar for å generera omsetjingskandidatar til tospråklege ordbøker; i dette innlegget skal eg gå litt meir inn på kandidatgenerering ved omsetjing av enkeltdelane av samansette ord. Me har som nemnt allereie ei ordbok mellom bokmål og nordsamisk, som me vil utvida til bokmål–lulesamisk og bokmål–sørsamisk. Og ordboka blei utvikla for å omsetja typisk «departementsspråk», så ho er full av lange, samansette ord. Og på samisk kan me setja saman ord omtrent på same måte som på norsk (i tillegg til ein haug med andre måtar, men det hoppar me glatt over for no). Dette bør me kunna utnytta, sånn at viss me veit kva «klage» er på lulesamisk, og me veit kva «frist» er, så har me iallfall éin fornuftig hypotese for kva «klagefrist» kan vera på lulesamisk

Orddeling er flott når du skal omsetja ordbøker. Særskrivingsfeil er flott når du vil smila litt. — «Ananássasuorma» jali «ananássa riŋŋgu»? Ij le buorre diehtet.

Altså kan me bruka dei få omsetjingane me allereie har mellom bokmål og lulesamisk/sørsamisk til å laga fleire omsetjingar, ved å omsetja deler av ord, og så setja dei saman igjen. Me har òg eit par omsetjingar liggande mellom nordsamisk og lulesamisk/sørsamisk, så me kan bruka same metoden der (og utnytta det at me har ei bokmål–nordsamisk-ordbok til å slutta riŋgen tilbake til bokmål).

Dekning og presisjon

Dessverre (i denne samanhengen) har me òg ofte fleire omsetjingar av kvart ord; i dei eksisterande bokmål–lulesamisk-ordbøkene me ser på (i stor grad basert på ordboka til Anders Kintel) står det at «klage» kan vera mellom anna gujdalvis, gujddim, luodjom eller kritihkka, medan «frist» kan vera ájggemierre, giehtadaláduvvat, mierreduvvam eller ájggemærráj. Viss me tillet kvar venstredel å gå med kvar høgredel, får me 16 moglege kandidatar for dette eine ordet! Sannsynlegvis er ikkje meir enn ein eller to av dei brukande (og kanskje ikkje det ein gong). I snitt får me rundt dobbelt så mange kandidatar som kjeldeord med denne metoden. Så me bør finna metodar for å kutta ned på dårlege kandidatar.

Den komplementære utfordringa er å få god nok dekning. Av og til ser me at me ikkje har ei omsetjing av delane av ordet, sjølv om me har omsetjingar av ord med dei same delene i seg. Den setninga krev nok eit døme Me vil gjerne ha ein kandidat for ordet «øyekatarr» på lulesamisk, altså samansetjinga «øye+katarr». Me har kanskje ei omsetjing for «øye» i materialet vårt, men ingenting for «katarr». Derimot står det at «blærekatarr» er gådtjåráhkkovuolssje. Så for å utvida dekninga, kan me i tillegg dela opp kjeldematerialet vårt i alle par av samansetjingsdelar; viss me veit at desse orda kan analyserast som «blære+katarr» og gådtjåráhkko+vuolssje, så kan det jo synast som at «blære» er gådtjåráhkko og «katarr» er vuolssje (og Giellatekno har heldigvis gode morfologiske analysatorar som fint deler opp slike ord på rette staden). Og dette gir ei god utviding av materialet – faktisk får me kandidatar for nesten dobbelt så mange av dei orda som me ønsker kandidatar for, viss me utvidar kjeldematerialet på denne måten. Men det har ei stor ulempe òg: Me får over dobbelt så mange lule-/sørsamiske kandidatar per bokmålsord (i snitt rundt fire kandidatar per kjeldeord).

Filtrering og rangering

Me vil innskrenka dei moglege kandidatane til dei som mest sannsynleg er gode. Den beste testen er å sjå om kandidaten finst i korpus, og då helst i same parallellstilte setning (dette er oftast ein bra kandidat). Viss ikkje, så kan me òg sjå på om kandidaten og kjeldeordet har liknande frekvensar, eller om kandidaten har frekvens i det heile.

Orddelingsomsetjinga foreslo tsavtshvierhtie for «virkemiddel», og der stod dei i ein parallellsetning òg:
<s xml:lang="sma" id="2060"/>Daesnie FoU akte vihkeles tsavtshvierhtie . <s xml:lang="nob" id="2060"/>Her er FoU er et viktig virkemiddel .
– då er det nok eit godt ordpar.

Uheldigvis har me så lite tekstgrunnlag for lule-/sørsamisk at me fort går tom for kandidatar med frekvens i det heile. For sørsamisk har me t.d. berre kandidatar med korpustreff for rundt 10 % av orda me lagar kandidatar for.

Ein annan test, som fungerer på alle ord, er å sjå om det får analyse av dei morfologiske analysatorane våre; viss ikkje (og viss det i tillegg ikkje har korpustreff) er det oftast feil. Men dette fjernar berre rundt 1/4 av kandidatane; med den oppdelte ordboka vår (kor me òg har med par av delar av ord) har me enno i snitt rundt tre kandidatar per kjeldeord.

(Ein test som eg prøvde, men avslo, var filtrering basert på liknande ordlengd. Det verkar jo logisk at lange ord blir omsett til lange og korte til korte, men det finst mange gode unntak. I tillegg fjernar det alt for få dårlege kandidatar til at det ser ut til å vera verdt det.)

Det parallelle korpusmaterialet vårt er altfor lite, men når me skal generera kandidatar til ordbøker så er det jo ikkje parallelle setningar me prøver å predikera, men parallelle ord og ordbokspar. Og då er jo læringsgrunnlaget vårt eigentleg dei eksisterande ordbøkene våre … Derfor prøvde eg å sjå på kva for samansetjingsdelar som faktisk var brukt i dei tidlegare omsetjingane våre, og kva for par av delar som ofte opptredde i tidlegare omsetjingar, og kva for delar som sjeldan eller aldri gjorde det. Til dømes har den oppdelte ordboka vår for bokmål–lulesamisk desse para:

løyve loahpádus
løyve doajmmaloahpe
arbeidsløyve barggoloahpádus
spesialløyve sierraloahpádus

Her ser me at «løyve» anten kan vera loahpádus eller doajmmaloahpe – skal «taxiløyve» då vera táksiloahpádus eller táksidoajmmaloahpe? På bakgrunn av dette materialet bør me nok satsa på det første – sjølv om doajmmaloahpe står oppført, så er det berre loahpádus som opptrer i samansette ord.

Då kan me prøva å generera kandidatar for alle bokmålsorda i materialet vårt, både dei me eigentleg er ute etter å finna kandidatar for, og dei me allereie har omsetjingar for. Gå så gjennom dei genererte kandidatane for dei orda me allereie har omsetjingar for, og tel opp dei para av orddelar som genererte slike ord. Me har kanskje laga kandidatane barggo+loahpádus og barggo+dajmmaloahpe for «arbeids+løyve»; når me så går gjennom dei eksisterande omsetjingane og finn at «arbeidsløyve» stod i ordboka med omsetjinga barggoloahpádus, så aukar me frekvensen til paret «løyve»–loahpádus med éin, medan «løyve»–dajmmaloahpe blir verande null.

For no har berre filtrert ut dei kandidatane kor paret til anten første- eller andreledd hadde nullfrekvens. I følgje litt manuell evaluering frå ein lingvist er det omtrent berre dårlege ord som blir kasta ut, så det filteret ser ut til å fungera bra. På den andre sida blir berre rundt 10 % av kandidatane fjerna viss me berre hiv ut dei med nullfrekvens, så neste steg blir å bruka frekvensane til å få ei full rangering.

Viss alle ord kunne delast i nøyaktig to delar, så ville det kanskje vore nok å telja opp par av delar og enkeltdelar for å estimera sannsyn, altså f(s,t)/f(s). Men av og til kan ord delast på fleire måtar, til dømes kan me sjå på «sommersiidastyre» som «sommer+siidastyre» eller «sommersiida+styre» (eg har valt å halda meg til todelingar av ord, for å unngå for mange alternative kandidatar). Viss omsetjinga er giessesijddastivrra, med analysane giesse+sijddastivrra eller giessesijdda+stivrra, så har me ikkje utan vidare nokon grunn til å velja den eine over den andre (vel, me har lengd i dette tilfellet, men det gjeld ikkje i alle slike døme, og me kan ha par av analysar som er 2–3 eller 3–2). Då kan me heller ikkje seia kva for par av orddelar (s,t) me skal auka når me ser «sommersiidastyre»–giessesijddastivrra i treningsmaterialet. Men viss me i tillegg ser «styre»–stivvra ein annan stad, så har me plutseleg eit grunnlag til å ta ei avgjerd. Metodar som Expectation Maximization kan kombinera relaterte frekvensar på denne måten for å finna fram til gode estimat, men eg har ikkje komme så langt at eg har fått implementert dette enno.

by k atApril 23, 2015 06:11 PM

April 14, 2015

NUUG events video archive

20111108_lisp

April 14, 2015 11:13 AM

February 12, 2015

Salve J. Nilsen

On Bandwagonbuilders and Bandwagoneers

<Farnsworth>Good news, everyone!</Farnsworth>

The League of Bandwagonbuilders have spoken â€“ Perl 6 is likely to be “production ready” sometime in 2015! This means it’s time for the Bandwagoneers to start preparing.

Bandwagoneer â€“ that’s you and me, although you may call yourself something different. Perl Monger. Perl Enthusiast. Or just someone who has realized that all volunteer-based Open Source communities need people who care about making stuff happen in meatspace.

At Oslo.pm (I’m a board member there), we’re doing exactly that. We’re Bandwagoneers, spending some of our own valuable time showing others where the cool stuff is, and showing them how to get it. Here’s some of what we’re up to:

On Tuesday March 3rd 2015, Damian Conway is back in Oslo to offer us an entertaining look at the Backstory of Perl 6. His talk Fifty Shades of Perl will happen at the Scotsman pub at 18:00. You can sign up for the event at Oslo.pm’s meetup.com page.
On Wednesday March 4th, Damian is back to offer us a more sober Introduction to Perl 6 features most useful for regular people. His talk Everyday Perl 6 is especially suited for anyone wanting to get the “advanced beginners” take on the language. The venue is the House of Communities, and we start at 18:30. Signup at meetup.com.
On Tuesday March 10th 2015, Jonathan Worthington is back in Oslo, and he’ll visit Hackeriet (“The Hackery”, one of Oslo’s oldest hackerspaces) for an informal Q&A session about the Perl 6 Virtual Machine. We’ll start around 18:00, and go out for some beers and/or food afterwards. Signup at Hackeriet’s Meetup.com page.
On May 8th-10th 2015, Oslo.pm is part of the Nordic edition of Open Source Developer’s Conference. This will be our Nordic Perl Workshop for 2015, and we’re hoping to have a strong Perl 5 and Perl 6 visibility there.

Also worth mentioning; a few weeks ago we also had an introduction to Perl 6’s Foreign Function Interface (called NativeCall), courtesy Arne SkjĂŚrholt. It was quite useful, and I hear Arne’s happy to accept invitations from Perl Monger groups to come visit and give the same presentation.

Enough bragging, already!

Being a Bandwagoneer means it’s your task to make stuff happen. There are many ways to do it, and I hope you can find some inspiration in what Oslo.pm is doing. Maybe get in touch with some of the Bandwagonbuilders in #perl6 on irc.freenode.org, and ask if anyone there would like to visit your group? I think that would be cool.

Get cracking!

by sjn atFebruary 12, 2015 11:20 PM

January 06, 2015

thefastestwaytobreakamachine

NSA-proof SSH

ssh-picture One of the biggest takeaways from 31C3 and the most recent Snowden-leaked NSA documents is that a lot of SSH stuff is .. broken.

I’m not surprised, but then again I never am when it comes to this paranoia stuff. However, I do run a ton of SSH in production and know a lot of people that do. Are we all fucked? Well, almost, but not really.

Unfortunately most of what Stribika writes about the “Secure Secure Shell” doesn’t work for ~~old~~ production versions of SSH. The cliff notes for us real-world people, who will realistically be running SSH 5.9p1 for years is hidden in the bettercrypto.org repo.

Edit your /etc/ssh/sshd_config:

Ciphers aes256-ctr,aes192-ctr,aes128-ctr MACs hmac-sha2-512,hmac-sha2-256,hmac-ripemd160 KexAlgorithms diffie-hellman-group-exchange-sha256

sshh
Basically the nice and forward secure aes-*-gcm chacha20-poly1305 ciphers, the curve25519-sha256 Kex algorithm and Encrypt-Then-MAC message authentication modes are not available to those of us stuck in the early 2000s. That’s right, provably NSA-proof stuff not supported. Upgrading at this point makes sense.

Still, we can harden SSH, so go into /etc/ssh/moduli and delete all the moduli that have 5th column < 2048, and disable ECDSA host keys:

cd /etc/ssh
mkdir -p broken
mv moduli ssh_host_dsa_key* ssh_host_ecdsa_key* ssh_host_key* broken
awk '{ if ($5 > 2048){ print } }' broken/moduli > moduli
# create broken links to force SSH not to regenerate broken keys
ln -s ssh_host_ecdsa_key ssh_host_ecdsa_key
ln -s ssh_host_dsa_key ssh_host_dsa_key
ln -s ssh_host_key ssh_host_key

Your clients, which hopefully have more recent versions of SSH, could have the following settings in /etc/ssh/ssh_config or .ssh/config:

Host all-old-servers

    Ciphers aes256-gcm@openssh.com,aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
    MACs hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-ripemd160-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-ripemd160
    KexAlgorithms curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256

Note: Sadly, ~~the -ctr ciphers do not provide forward security and~~ hmac-ripemd160 isn’t the strongest MAC. But if you disable these, there are plenty of places you won’t be able to connect to. Upgrade your servers to get rid of these poor auth methods!

Handily, I have made a little script to do all this and more, which you can find in my Gone distribution.

There, done.

sshh obama

Updated Jan 6th to highlight the problems of not upgrading SSH.
Updated Jan 22nd to note CTR mode isn’t any worse.
Go learn about COMSEC if you didn’t get trolled by the title.

by kacper atJanuary 06, 2015 04:33 PM

December 08, 2014

thefastestwaytobreakamachine

sound sound

Intermission..

Recently I been doing some video editing.. less editing than tweaking my system tho.
If you want your jack output to speak with Kdenlive, a most excellent video editing suite,
and output audio in a nice way without choppyness and popping, which I promise you is not nice,
you’ll want to pipe it through pulseaudio because the alsa to jack stuff doesn’t do well with phonom, at least not on this convoluted setup.

Remember, to get that setup to work, ALSA pipes to jack with the pcm.jack { type jack .. thing, and remove the alsa to pulseaudio stupidity at /usr/share/alsa/alsa.conf.d/50-pulseaudio.conf

So, once that’s in place, it won’t play even though Pulse found your Jack because your clients are defaulting out on some ALSA device… this is when you change /etc/pulse/client.conf and set default-sink = jack_out.

by kacper atDecember 08, 2014 12:18 AM

November 18, 2014

Anders Einar Hilden

Changing the Subnet Mask in Vmware Workstation on Debian Jessie

I’m currently attending SANS SEC504: Hacker Tools, Techniques, Exploits and Incident Handling in London. For some of the labs in the course we need machines on the IPs 10.10.0.1 and 10.10.71.1 with a subnet mask of 255.255.0.0.

Changing the Subnet Mask for the NAT or host-only networks in VMware Workstation seems like such a easy thing to do. According to VMware it should be as easy as opening the Virtual Network Editor and “type a new value in the Subnet mask text box”.

Oh wait … I can’t change it. The field for subnet mask in the Virtual Network Editor is not editable.

VMware Virtual Network Editor: the field for subnet mask is not editable

Let’s keep googling - plenty matches, but everyone keeps insisting it can be changed in the GUI, or mixes the subnet mask with the subnet IP. Some posts blame permission, but since the Virtual Network Editor always runs as root, that’s not the problem. There are no listings for the vmnets in /etc/network/interfaces or /etc/network/interfaces.d/ and changing the subnet mask in NetworkManager does nothing.

After a lot of thinking (and just after I checked /etc/network/interfaces ) I found /etc/vmware/networking - BINGO! This looks like just the file we were looking for.

Before editing the file we should stop any vmware-related services that might use these files.

$ sudo service vm<TAB>
vmamqpd vmware vmware-USBArbitrator vmware-workstation-server

I’m not sure witch of these services use the files we are editing, so we’ll stop them all

$ sudo service vmamqpd
$ sudo service vmware
$ sudo service vmware-USBArbitrator
$ sudo service vmware-workstation-server

For the SANS course I have set up a new host-only network vmnet2. Since we are using static IPs, and will be running malware on these systems, I have disabled DHCP and not connected a host virtual adapter. The shared folder option Map as a network drive in Windows guests still work, don’t ask me how. Below is the configuration for vmnet2 with a subnet mask of 255.255.0.0.

answer VNET_2_DHCP no
answer VNET_2_DHCP_CFG_HASH E9892EF1006EBB5D4996DF1A377B10EB0D542B94
answer VNET_2_HOSTONLY_NETMASK 255.255.0.0
answer VNET_2_HOSTONLY_SUBNET 10.10.0.0
answer VNET_2_VIRTUAL_ADAPTER no

Success! (but continue reading, we update the DHCP configuration below the picture)

VMware Virtual Network Editor: the uneditable field contains the subnet mask we wanted

VMware stores DHCP config and leases in /etc/vmware/vmnet<NUM>/dhcpd/. If we have changed the subnet IP, subnet mask, or turned on or off DHCP, these files need to be updated. The configfile contains autogenerated information surronded by “DO NOT MODIFY SECTION”, so we should probably not edit it manually.

If we open VMware Virtual Network Editor (sudo vmware-netcfg), change a setting (e.g. the subnet IP from 10.10.0.0 to 10.11.0.0), save, and then change it back again, VMware will update the files for us.

November 18, 2014 02:50 PM

January 29, 2014

Nicolai Langfeldt

git log -p splitting

At work we have two related code bases. Recently one of them received a lot of loving, and the other needed the same treatment to work better with new Perl and modules. The one ha gotten several hundred patches, and browsing that many and cherrypicking them got tiresome. It was better for me to split the whole log into separate patches and review and apply one by one and then moving the "done" patches to a different directory.

Here, the small hack to split "git log -p" into one patch pr. file:

#!/bin/bash

awk 'BEGIN { FN=0; }

/^commit / { FN++; }

{ print $0 >> FN }' $1

by Nicolai Langfeldt (noreply@blogger.com) atJanuary 29, 2014 09:40 AM

February 24, 2013

Bjørn Venn

Chromebook; a real cloud computer – but will it work in the clouds?

Lyst på én? Den er ikke i salg i Norge enda, men du kan kjøpe den på Amazon. Les her hvordan jeg kjøpte min på Amazon (bla litt nedover på siden). Med norsk moms, levert til Rimi-butikken 100 meter fra der jeg bor, kom den på 1.850 kroner. Det er den så absolutt verdt:)

by Bjorn Venn atFebruary 24, 2013 07:34 PM

February 22, 2013

Bjørn Venn

Hvem klarer å skaffe meg en slik før påske?

Den nye Chromebook-en til Google, Chromebook Pixel. Foreløbig kun i salg i USA og UK via Google Play og BestBuy.

Verden er urettferdig:)

by Bjorn Venn atFebruary 22, 2013 12:44 PM

January 07, 2013

NUUG events video archive

Utfordringer innen identitetshåndtering og autentisering

Dag-Erling Smørgrav påpeker hvordan Unix har et autentiserings-paradigme som ikke har endret seg på 40 år, mens det har skjedd store utviklinger de siste årene på denne fronten.

January 07, 2013 10:00 PM

May 29, 2012

Salve J. Nilsen

Inviting to the Moving to Moose Hackathon 2012

Oslo Perl Mongers are organizing a hackathon for everyone who would like to dive deep into the details of Moose! We have invited the #p5-mop crowd to work on getting a proper Meta Object Protocol into Perl core, and we’ve invited the #perlrdf crowd to come and convert the Perl RDF toolchain to Moose.

You’re welcome to join us!

Topic: Moving to Moose – in Perl 5 core and on CPAN
Arrival Stavanger, Norway: August 25th before 14:00 CET DST. Check-in before 19:00.
Four full days of hacking: August 26th to 29th 2012, at Preikestolen Mountain Lodge, near Stavanger.
Check-out and departure: August 30th 2012.
Event ticket includes food and lodging (limited availablility).
Event website: http://act.yapc.eu/mtmh2012/.
Registration: http://act.yapc.eu/mtmh2012/register/.

Special rebate for members of the Perl and CPAN communities

If you’re working on a project that is considering on moving to Moose then you’re especially welcome! We have a set of promo codes you can use when signing up for the hackathon. Please get in touch with us (or some of the existing participants) to get your promo code and a significant rebate!

Commercial tickets available

Would you like to support the hackathon, but don’t have access to a sponsorship budget? Does your company plan on using Moose, and sees the value of having excellent contacts in the open source communities around this technology? For you, we have a limited amount of commercial tickets. Please check out the hackathon participation page for details.

Sponsorship opportunities

The hackathon is already well sponsored, but there is room for more! If you want to support the us, please contact the organizers as soon as possible!

Who can come?

In short: Everyone who cares about Moose and object-oriented programming in Perl! We’re trying to make the Perl community better by hacking on the stuff that makes the biggest difference (at least in our eyes ;)). If you agree, you’re very welcome to join us! Check out the event site for details, and get in touch with us on IRC if you’re interested.

And finally, keep in mind Oslo.pm’s 2-point plan:

Do something cool
Tell about it!

See you in Stavanger?

by sjn atMay 29, 2012 02:44 PM

October 31, 2011

Anders Nordby

Taile wtmp-logg i 64-bit Linux med Perl?

Jeg liker å la ting skje hendelsesbasert, og har i den forbindelse lagd et script for å rsynce innhold etter opplasting med FTP. Jeg tailer da wtmp-loggen med Perl, og starter sync når brukeren er eller har blitt logget ut (kort idle timeout). Å taile wtmp i FreeBSD var noe jeg for lenge siden fant et fungerende eksempel på nettet:

$typedef = 'A8 A16 A16 L'; $sizeof = length pack($typedef, () ); while ( read(WTMP, $buffer, $sizeof) == $sizeof ) { ($line, $user, $host, $time) = unpack($typedef, $buffer); # Gjør hva du vil med disse verdiene her }

FreeBSD bruker altså bare verdiene line (ut_line), user (ut_name), host (ut_host) og time (ut_time), jfr. utmp.h. Linux (x64, hvem bryr seg om 32-bit?) derimot, lagrer en hel del mer i wtmp-loggen, og etter en del Googling, prøving/feiling og kikking i bits/utmp.h kom jeg frem til:

$typedef = "s x2 i A32 A4 A32 A256 s2 l i2 i4 A20"; $sizeof = length pack($typedef, () ); while ( read(WTMP, $buffer, $sizeof) == $sizeof ) { ($type, $pid, $line, $id, $user, $host, $term, $exit, $session, $sec, $usec, $addr, $unused) = unpack($typedef, $buffer); # Gjør hva du vil med disse verdiene her }

Som bare funker, flott altså. Da ser jeg i sanntid brukere som logger på og av, og kan ta handlinger basert på dette.

by Anders (noreply@blogger.com) atOctober 31, 2011 07:37 PM

Planet NUUG

February 12, 2018

February 11, 2018

January 04, 2018

Familie og helse

Utenriks og nordområder

Næringsliv og finanser

Skole og utdanning

Kultur og frivillighet

Klima, energi og landbruk

Justis og beredskap

December 29, 2017

December 19, 2017

December 10, 2017

October 24, 2017

October 23, 2017

Hardware

Preparing the build environment

Building the disk image

Putting the disk image in initramfs

ZFS configuration

October 20, 2017

October 03, 2017

September 28, 2017

July 31, 2017

April 19, 2017

Jobbeskrivelse

Vilkår

February 13, 2017

January 28, 2017

January 06, 2017

Build environment

Disk image

Initramfs

Result

November 12, 2016

September 18, 2016

August 24, 2016

July 15, 2016

June 01, 2016

April 02, 2016

April 01, 2016

October 18, 2015

April 23, 2015

Dekning og presisjon

Filtrering og rangering

April 14, 2015

February 12, 2015

<Farnsworth>Good news, everyone!</Farnsworth>

Enough bragging, already!

January 06, 2015

December 08, 2014

November 18, 2014

January 29, 2014

February 24, 2013

February 22, 2013

January 07, 2013

May 29, 2012

Special rebate for members of the Perl and CPAN communities

Commercial tickets available

Sponsorship opportunities

Who can come?

October 31, 2011

Subscriptions