Category Archives: General

General chitchat

Lessons learnt from adding OpenBSD/x86_64 support to pkgsrc

Before even getting into the internals of operating systems to learn about differences among a group of operating systems, It’s fairly evident that something as simple as naming is different between operating systems.

For example, the generations of trusty 32bit x86 PC is commonly named i386 in most operating systems, FreeBSD may also refer to it as just pc, Solaris & derivatives refer to it as i86pc, Mac OS X refers to it as i486 (NeXTSTEP never ran on a 386, it needed a minimum of a 486 and up until Sierra, machine(1) would report i486 despite being on a Core i7 system), this is one of the many architectures which needed to hadled within pkgsrc. To simplify things and reduce lengthy statements, all variants for an arch are translated to a common name whiche is then used for reference in pkgsrc. This means that all the examples above are grouped together under the MACHINE_ARCH i386. In the case of 64bit x86 or commonly referred to as amd64, we group under x86_64 or at least tried to. The exception to this grouping was OpenBSD/amd64, this resulted in the breakage of many packages because any special attention required was generally handled under the context of MACHINE_ARCH=x86_64. In some packages, developers had added a new exception for MACHINE_ARCH=amd64 when OPSYS=OPENBSD but it was not a sustainable strategy because to be affective, the entire tree would need to be handled. I covered the issue at the time in A week of pkgsrc #11 but to summarise, $machine_arch may be set at the start in the bootstrap script but as the process works through the list of tasks, the value of this variable is overriden despite being passed down the chain at the begining of a step. After some experimentation and the help of Jonathan Perkin, the hurdles were removed and thus OpenBSD/x86_64 was born in pkgsrc ūüėČ

The value of this exercise for me was that I learnt the number of places within the internals of pkgsrc I could set something (by the nature of coupling components which share the same conventions (pkgtools, bsd make)) and really the only place I should be seeking to set something is at the start of the process and have that carry through, rather than trying to short circuit the process and repeat myself.

Thanks to John Klos, I was given control of a IBM Power 8+ S822LC running Ubuntu, which started setting up for pkgsrc bulk builds.
First issue I hit was pkgsrc not being able to find, this turned out to be the lack of handling for the multilib paths found on Debian & derivates for PowerPC based systems.
This system is a little endian 64bit PowerPC machine which is a new speciality in itself and so I set out to make my first mistake. Adding a new check for the wrong MACHINE_ARCH, long forgotten about the previous battle with OpenBSD/x86_64 I added a new statment to resolve the relevant paths for ppc64le systems. Bootstrap was happy with that & things moved forward. At this point I was pointed to lang/python27 most likely being borken by Maya Rashish, John had previously reported the issue and we started to poke at things. As we started rummaging through the internals of pkgsrc (pkgsrc/mk) I started to realise we’re heading down the wrong path of marking things up in multiple places again, rather than setting things once & propogating through.

It turned out that I only need to make 3 changes to add support for Linux running on little endian 64bit PowerPC to pkgsrc (2 additions & 1 correction ūüėČ )
First, add a case in the pkgsrc/bootstrap/bootstrap script to set $machine_arch to what we want to group under when the relevant machine type is detected. In this case it was when Linux running on a ppc64le host, set $machine_arch to powerpc64le. As this is a new machine arch, also ensure it’s listed in the correct endianness category of pkgsrc/mk/, in this case add powerpc64le to _LITTLEENDIANCPUS.
Then correct the first change to replace the reference to ppc64le for handling the multilib paths in pkgsrc/mk/platform/

The bulkbuild is still in progress as I write this post but 5708/18148 packages in an the only fall out so far appears to be the ruby interepreters.

pkgsrcCon 2016 videos

The videos from some of the talks that took place at pkgsrcCon in Kraków, Poland during the Summer are now available on the Internet Archive.
This years conference drew speakers from many different project, not necessarily BSD related though Net/Free/OpenBSD were represented. Talks were on a pretty diverse range of topics around software but unfortunately not all talks were recorded. I had the opportunity to tag on Mateusz Kocielski’s slot on security & give a brief talk on the work of the pkgsrc security team (slides), whilst he covered the work of the NetBSD security team (see files security-team & flash-die).

The schedule for the conf is available here

A slow / low-end system capable of running most modern BSDs

I was looking to test a change related to buffering in cat(1) and wondered what was the slowest system I could use which was capable of running the current versions of NetBSD, FreeBSD, OpenBSD. An old PC and the ARM based BeagleBone Black sprang to mind immediately, then a PowerPC Mac? SPARC64?

Apart from a Sun Fire T1000, I do not have any SPARC hardware, sun4v is only supported on NetBSD & OpenBSD at present, FreeBSD/sun4v was only a pre-alpha rough cut from before the days of version 7 and sparc64 support may be going away in FreeBSD moving forward.

Considered the BeagleBone Black but currently NetBSD-HEAD does not boot on it port-arm/51380 and FreeBSD has issues with running DTrace bug/211389. So that was off the list.

A G4 based PowerPC Mac is supported between my choice of BSDs, unfortunately I couldn’t get a working disk burnt from the FreeBSD iso files to try it out on a 12″ PowerBook. bug/211488.

I settled on running i386 builds on a Alix 2c3 I have, it has 256MB RAM and a 500Mhz Geode CPU, currently running FreeBSD/i386 11-BETA3 without issue and has no problems with any of the other BSDs. It’s a little too “modern” and high spec though in my test.

The BSD family of operating systems

At OSHUG #46 I was given the opportunity to present the BSD’s to a group of open source hardware enthusiast & speak about why this family of operating systems would benefit the person running a flavour on their hardware. There was a recording made of the talk but it may be some time before¬†it is¬†made available¬†online, so I thought I’d take the time to write something up to share in the meantime.

My slides.

This line of operating systems started out life as a series of patches to AT&T UNIX which was introduced to the University of Berkeley by Ken Thompson whilst on sabbatical in 1977.
From the 1BSD TAPE file included in the CSRG archive CD set

Berkeley UNIX Software Tape
Jan 16, 1978 TP 800BPI

The first release came with things such as the ex editor, ashell and Pascal compiler as an add-on for UNIX v7, running on a PDP-11. Over the life time of the CSRG they produced releases which included vi, csh, the IPv4 TCP/IP network stack, the virtual memory subsystem (the kernel being named vmunix, parodied by Linux as vmlinuz) and UFS.
The distribution tapes were only available to AT&T licensees; over time the code base of the distribution grew increasingly independent from AT&T UNIX. At the same time the cost of the AT&T license continued to increase as well. Starting out at a cost of $10000 and reaching north of $250000 in the late 80’s.¬†According to Kirk McKusick there was pressure to release the independently developed components of the CSRG so the community could benefit from the use of things such as the network stack without purchasing a costly license. This resulted in several release, comprised mostly of the code developed outside of AT&T such as 4.3BSD-Net/1, Net/2,¬†4.4BSD-Lite & Lite2. “Mostly” in that with the release of Net/2 AT&T file a lawsuit against the University of California for alleged code copying and theft of trade secrets.
During its lifetime, BSD saw itself being run on several CPU architectures from the DEC PDP-11, VAX to the MIPS, HP 9000 and Motorola 68000 to name a few. These ports along with the  Power 6/32 helped to improve the portability of the code base. The code base was deemed to be 90% platform independent, the remaining 10% being mostly related to the VM subsystem which was platform specific. As with AT&T UNIX, portability & migration between different systems was part of the nature of the code base, from the beginning.

The 4.3BSD-Net/2 code base was used as the basis for a port to the Intel 386, resulting in 386BSD (free) & BSD386 (commercial) releases.

The Modern BSD variants
At the time of writing there are many BSD variants in existence, each with its own area of focus. Everything still leads back to 2 major variants.

NetBSD was the first of the modern variants that is still actively developed. It started out life as a fork of 386BSD. The focus of NetBSD is portability which not only makes porting to new hardware easier (currently supporting over 60 different ports across may CPU architectures).
Everything from a VAX, ARM & MIPS Windows CE based PDAs to a Sega Dreamcast and many other systems are supported and able to run the latest version of NetBSD. There’s even a toaster which run NetBSD
The focus on portability also makes reusing components on other operating systems easy. For example the packaging system (forked from FreeBSD (which we’ll talk about next)) supports over 20 operating systems.
This enables a consistent toolset to be used regardless of operating system.

Some of the highlights of NetBSD include ATF, unprivileged builds and portable build infrastructure using

ATF, as the name suggests is used for automated tests of the source code to discover regression in the code base in an automated manner. Results can be found on the NetBSD release engineering page.

Unprivileged builds allow a user to not only build a copy of the operating systems without elevated privileged, but they can also build and install software from pkgsrc in a location they have write access to (by default, in a prefix under their home directory)., the build framework, allows NetBSD to be built on any modern POSIX compliant operating system. Freeing the person to use a operating system of their choice to build releases.

04/05/2016- Note Ollivier’s comment, I made a mistake when I was gathering info and looked at the source for head and checked the history for the COPYRIGHT file there, not noticing the repository started with v2.0.

Forked from the 4.4BSD Lite code base, 6 months after NetBSD was started. The focus of FreeBSD was performance on i386 systems. Over time support was added for the DEC Alpha as this meant porting the code base to a 64bit systems and addressing any bugs which would prevent the code base from running on a 64bit system. Many years later the project branched out and introduced support for additional platforms. Today the project boasts support for CPUs such as ARMv8, RISC-V and BERI.

Forked from NetBSD, the focus of OpenBSD is security. The project is home to many components which see wider use outside of OpenBSD, such as OpenSSH, PF (firewall), LibreSSL and others.

Forked from FreeBSD, the focus of DragonFly BSD is scalability & performance. Taking the operating system in a new direction with regards to how SMP is implemented and from there, developing a new files system called HAMMER.

No matter the flavour, documentation is a key part of the development process for the BSD’s.
Whether it is the Design & Implementation series which started with covering 4.3BSD in 1989 and more recently FreeBSD 10 in the fourth instalment of the series, or each projects own set of documentation. Documentation is important as it distinguishes intent & implementation as well as save a lot of question and answers emails.
FreeBSD has handbooks, NetBSD has guides, OpenBSD has FAQs and all projects make their man pages available online as web pages. There is even  a teaching course based around the  The Design and Implementation of the FreeBSD Operating System, 2nd edition.

Frameworks for building embedded images
Each operating system release is a complete, self contained bundle, containing the documentation and necessary toolchain required for building a copy of the operating system from source. release(7) on FreeBSD & NetBSD, release(8) on OpenBSD, nerelease(7) on DragonFlyBSD

For the purpose of embedding the operating system it may not be desirable to build a full blown release. Depending on the choice of variant, either the functionality is built in as standard or a project exists to assist with generating customised images with ease.

FreeBSD had PicoBSD which is now superseded by NanoBSD.
OpenBSD has flashrd and resflash.
NetBSD has a target for generating an image in, customisations controlled by variables set in mk.conf.
DragonFlyBSD has nrelase.

RetroBSD / LiteBSD
RetroBSD is a port of 2.11BSD (originally targeted for the PDP-11) to the MIPS M4K core found on the PIC32 micro-controllers. LiteBSD is a port of 4.4BSD to the PIC32MZ micro-controllers with a MIPS32 core. Due to the limited resources available, RetroBSD does not offer a network stack, Of the 128KB of RAM, 96KB are available for user space applications. A compiler, editor & various utilities come bundled with the OS so software could be developed on the PIC itself. Variants of common software titles are available to extend the system, such as an Emacs like editor.
LiteBSD is based on a more recent version of BSD, taking advantage of the availability of more RAM (512KB) and MMU on the targeted micro controller. It features a network stack.

Projects such as these take advantage of prior effort and offer the user a consistent environment from the microcontroller to desktop to server. With the extensive documentation and availability of source history, it is possible to realise at which stage in the evolution of the code base the currently running system is and if a desired feature is implemented.

The development of BSD is closely tied with that of the internet. BSD’s modern variants are some of the oldest communities who have¬†collaborated over the internet to develop a software project. The workflow of the projects has transpired to become the standard way of developing open source software on the internet, whether it’s adhering to a style guide or developing with a publicly accessible source repository¬†or holding a hackathon.

For a newcomer interested in an operating system to run on your hardware, it is a great opportunity to be a part of a tech savvy community working to evolve an idea started almost 40 years ago.

As a business, each project produces a mature and robust operating system that has seen many applications from running on devices such as game consoles, mobile phones, cars, satellites and the international space station. Nearly all projects are backed by a non-profit foundation which can act as a liaison for businesses and assist with enquiries regarding development.

Adventures in Open Source Software: Dealing with Security

I had the opportunity to give this talk at the London chapter of DefCon, DC4420 and censecutivily at London Perl Mongers technical meeting last week.
The subject of the talk was all the factors outside of doing security resceach which can make the process of dealing with advisories a daunting or a seamless process. As observed during the last year, while working as part of a security team.


pkgsrc is a crossplatform packaging system by the NetBSD project, forked from the FreeBSD ports in the late 90’s, initially the primary target was NetBSD but with the portability focus of the project, the list of supported flatforms has grown to a list of 23 operating systems (16 out of those 23 are currently actively worked on). Within the pkgsrc project, there is a dedicated security team whose responsibility is to audit published vulnerabilities and ensure that those which apply to software we offer packages for are listed in a file. Users download this file & use it¬†to check their installed packages.
Other open source projects have teams who are more involved and participate in the security research process and publish their own advisories, such as Debian or Redhat but that is not  main focus of our team.
There may be several reasons for this, during my talk I refered to the aquisition of a security company by Redhat, but looking up Redhat on Wikipedia, there doesn’t appear to be anything to suggest that.
I can say that for the pkgsrc-security team, the role is focused on filtering information and ensuring that items are listed in the pkg-vulnerabilities file, maintainers are notified (if there is one) and co-ordinating with the release engineering team so that necessary commits are pulled into the relevant branches. This is because we try to avoid dealing with development within our tree and opt to co-ordinate with upstream to submit fixes. Majority of our changes focus on removing assumptions to ensure things are built in a consistent manner and allow the software to be packaged how we like.

Dealing with advisories:
The advisories which we receive range in quality / detail.
A personal favourite are the drupal advisories. We offer the drupal core as a package but not any of the 3rd party modules. Their advisories clearly indicate if they apply to core or any published 3rd party modules and which scenario is needed for the vulnerability to be exploitable eg the user must be able to upload content.

The opposite of that is independently published advisories without any co-ordination with affected parties or independently published advisories which are disputed or not acknowledge from upstream. In these situation the role becomes more involved in order to work out if there is clearly an issue or not.

Then there’s Oracle advisories, we can confirm there’s one or more problems in the following versions of software, no more details than that. Upgrade to this version at a minimum to fix said issue(s). Here’s a chart so you can evaluate the risk.

It can be that upstream has actually made an announcement with the details of an issue in public but the mitre website will still lists the CVE as reserved. Ideally you’d like to list the mitre site in pkg-vulnerabilities as it’s where IDs are assigned and it’s self referencing (url will contain the CVE id). But it’s a terrible thing to do to a user. “You have a package installed which is vulnerable to the following type of issue follow this link to not find out any more information about it. Go fish”. Or maybe you have no choice.

Project Websites:
If the published advisory come via a Linux distribution it can be common that the fix references a binary package for users to install or perhaps further information required. In the 2000’s Soureceforge was a popular host for open source projects usually complimented with a separate web page of some kind, it’s now common to have projects which solely exists as an authoritative repo on github. In either scenario, a dedicated section for publishing security information is usually not found. This trend¬†is also prevelant in large commercially backed projects, which play an extremely critical role. Projcts such as ICU (International Components for Unicode), a project by IBM which deals with unicode, an issue in ICU can mean an issue in chrome/chromium, java.

There are also projects like Qemu which have a security page for submitting vulnerability information but never publish advisories themselves. It is common for advisories to reference a git commit email. KVM completely lacks any links related to security. Qemu has a strong link with Xen & KVM which rely on Qemu in one way or another.
While we do not offer KVM as a package, we do at present have four different versions of Xen in our tree and Qemu. This becomes a bit of a timesink when there are multiple advisories to address.

Commercial Repositories:
There are opensource projects with no publicly accessible source code repo. This makes the evaluation of the range of effected verisons difficult if the project only chooses to cover their supported versions.
ISC up until recently (past two years?) required paid membership to access BIND’s repo.
In the talk I refered to the ICU project here, this was incorrect. ICU advisories are either reserved or the bug report access blocked from public view.

Relationships with projects are important and they play a critical role in not only sharing information but code as well. Changes for 3rd party software really needs to be passed to the 3rd party to take care of. If relationships have tourned sour, it makes sharing somewhat difficult and has further implications when developing a project with the support for other projects in mind.
The LibreSSL project published a patches page which covered the changes needed to get affected software built but also co-ordinated with upstream projects to get the fixes integrated. Some projects needed more pressure^Wpersuasion than others to accept the patches.

Key components & deadware:
As mentioned previously, you want changes to go back up stream and not to carry changes in your own tree. But there are scenarios where this is possible, for example, the project is no longer developed. This is a huge problem if the project is widely used because you end up carrying local patches which hinders progress when auditing for vulnerabilites by consumers of the software downstream. It’s no longer a case of ensuring you have a specific version number but which patches are also applied to that baseline version, that then opens further questions about the patches, have you created a new issue that didn’t exist previously??
The widely used unzip utility is such an example,  are you patched for CVE-2015-7696?

An example fragmentation of fixes being carried locally is libwmf, with the announcement of some CVEs earlier this year, Jason Unovitch from the FreeBSD project discovered that there were unpatched vulnerabilities in this library going back to 2004, with patches spread across different Linux distributions, none carrying fixes for all advisories, in one case a hunk of the patch didn’t even apply. Development for libwmf stopped in the early 2000’s but it still exists as a project on sourceforge.

Jasper is another commonly used graphics library, this time for jpeg-2000, again development ceased long ago. In this case Slackware put out an advisory for their package to cover vulnerabilities from the past, going back to 2008, at which point we realised that we didn’t have the fixes either. The version in OpenBSD ports was vulnerable to the issues listed from 2014 but the vulnerabilities from prior (2008) were fixed because they’d been flagged up by the compiler in OpenBSD.

Widely Deployed:
Popular projects which have a large install base can greatly increase impact of a mistake, hence local changes should be kept to a minimum to ease maintanence and auditability.
Projects can see a very fast release cycle, especially ones which have advisories published about them regularly.
Keeping local changes to a minimum reduces the necessary effort to update. With projects which rely on downstream consumers to publish information it makes the process more difficult. Both KVM & QEMU projects do not publish any advisories themselves, at best you may have a git commit email which may be the patch you carry locally. Thankfully the Xen project publish advisories on Qemu as it can be a dependency. They are able to flesh out the details of the issue a little better than a vague commit message.
I’m unsure what happens if you’re not a Linux distro and utilise KVM.

Co-ordinating with upstream:
As I mentioned, relationships are important. An understanding and tolerrance for difference is absolutely essential in the world of software just as it is in day to day life. A common topic of disagreement is licensing, the terms expressed by said licenses and the strong opinions expressed by the participants in the disagreement. Whatever ones belief, the need to co-ordinate with people from different groups is absolutely necessary.
Of the fixes upstreamed from LibreSSL, the author of stunnel rejected a fix initially but eventually changed his mind. The change in question was a 2 line addition to add an ifdef statement so that RAND_egd function was only used if the SSL library being linked to offered such a function (detected by autoconf already). The author rejected the change based on terms of licensing of his project when the change was submitted.

Taking bigger leaps in a software project by trying to clean up a popular target can amass a large collection of local patches which need to make their way upstream. As observed by the Alpine Linux project, a Linux distribution with a new libc called musl libc. While it’s possible to build over 13000 packages with Debian 8 on pkgsrc, the package count is less than 9000 on Alpine, despite both being Linux distributions.

The submission process can be quite daunting depending on the project, to filter out submissions which may not be sound and reduce the workload of developers working on a project, some opt to requiring certain things such as results from a test suite or alike. It doesn’t help matters if project has multiple branches developed in parallel without changes being in sync.
Dealing with GNU toochain such as GCC can be very much like this. Again, local changes amass, slow transforming the local version of the toolchain to an extended version of upstream. While the toolchain may offer security features such as SSP (stack smashing protection), it’s not just the simple case of being able to switch it on, in some cases it either doesn’t work or worse, it results in broken binaries. Work to enable some of these¬†features in pkgsrc began in the summer.

While not specifically security related, I was reminded of an incident on the OpenBSD mailing lists with the author of the once popular ion window manager. The ion developer was frustrated with operating system projects packaging older versions of his software with local changes as users where following up with him for support (in this case he was referring to RedHat). So he came on to the OpenBSD mailing list to ask about updating the software to the latest version available but that was not possible due to incompatible license changes. As usual, the combination of making demanding + licensing drama didn’t work out to well.


  • Organise the information on your site, for an Open Source software project your source control repo should be a single click away from your websites main page.
  • Provide a dedicated security page or section where you post advisories.
  • Write descriptive commit messages.
  • Participate in the community and help your neighbours
  • Even if you stop developing software, the code may live on longer than envisioned, think about what happens if/when you decide to stop (who become the authoritative repo).
  • Don’t make changes locally which do not go upstream by default, for it’ll surely bite you or a member of the project later down the line.
  • Publish actual advisories for your project, don’t pass the buck.
  • Technical problems are best solved with technical solutions eg a bug can still continue to exist despite adhering to a license.
  • Make the submission process to your project effortless for both parties, not just one or the other.

Running BSDi BSD/OS on VirtualBox

By default, the BSD/OS kernel recognises the CPU of a VirtualBox guest VM as a Pentium II. The kernel is able to boot correctly but performing any I/O results in failure due to memory errors. Adjusting the CPU mask of the VM from the host resolves this issue.

Note the name or GUID of the VM to be modified
% VBoxManage list vms
"BSDi BSD/OS" {36772f8c-ec06-4f37-a995-662fc38ad103}

Adjust the CPU of the VM
VBoxManage modifyvm "BSDi BSD/OS" --cpuidset 1 0x4a7 0x7100800 0x17bae3ff 0xbfebfbff

Obtained from OS2 Museum.

EuroBSDcon 2015

I unfortunately will not be presenting my talk at EuroBSDcon 2015 later this week. A family emergency that developed while I was in Ottawa earlier this year came to a head in early August. Things had been pretty hectic up until this point and I didn’t feel up to buttoning down for the next two months to work so I decided to cancel my talk as I just wanted to switch off. Life is now back in motion again as of earlier this month and I intend to pick up from where I left off with this project next month to resubmit next year. I’m sorry I will not be there in Sweden to enjoy the conference with some of you but hopefully see you in 2016 for the next round!

Book review: The Design and Implementation of the 4.3BSD UNIX Operating System

The Design and Implementation of 4.3BSD UNIX Operating System
According to my photographs, I picked up this book in February of this year. With a 105 sections spread over 13 chapters I’ve been working through the book slowly at a section a day. Despite being a technical subject, the book does a very good job of explaining the operation system at a high level without becoming a study of the source code. There are snippets of source code & pseudo code to compliment the text and an extensive list of papers for reference at end of each chapter for those that wish to dig deeper.

I had previously attempted to complete the Minix book, Operating Systems: Design And Implementation but struggled with the extensive source reference. switching back and fourth between chapters or the requirement for a computer to view the source code was not a viable option. I took a chance on this book as used copies are available on Amazon for the cost of a postage which is less than a couple of pounds. The book is well written and enjoyable to read, while implementation details may not be completely applicable to modern BSD variants The fundamental details may still hold true in most cases if not providing a historical background around the technical challenges they faced at the time. What I liked with the Minix was that it provided lots of background to accommodated a beginner and get a reader up to speed though I much preferred the ability to read this book by itself without requiring access to the source code.

I found some of the details in the interprocess communication part a little unclear at times but enjoyed the filesystem and memory management chapters the most and the terminal handling chapter the least though I did learn of Berknet there, aswell as many other historical artefacts throughout the book, some of which I tweeted under the hashtag di43bsd.

Berknet is an obsolete batch-oriented network that was used to connect PDP-11 and VAX UNIX systems using 9600-baud serial lines. Due to the overhead of input processing in the standard line discipline, a special reduced-function network discipline was devised.

The 4.3BSD kernel is not partitioned into multiple processes. This was a basic design decision in the earliest versions of UNIX. The first two implementations by Ken Thompson had no memory mapping at all, and thus made no hardware-enforced distinction between user and kernel space. A message-passing system could have been implemented as readily as the actually implemented model of kernel and user processes. The latter was chosen for simplicity. And the early kernels were small. It has been largely the introduction of more and larger facilities (such as networking) into the kernel that has made their separation into user processes an attractive prospect — one that is being pursued in, for example, Mach.

The book breaks down the percentage of components in each category (such as headers) which are platform independent and platform specific. With a total of 48270 lines of platform independent code versus 68200 lines of platform specific code, the 4.3BSD kernel was largely targeted at the VAX.

From the details on the implementation of mmap() in the BSD memory management design decisions section it was interesting to read about virtual memory subsystems of old

The original virtual memory design was based on the assumption that computer memories were small and expensive, whereas disk were locally connected, fast, large, and inexpensive. Thus, the virtual-memory system was designed to be frugal with its use of memory at the expense of generating extra disk traffic.

It made me think of Mac OS X 10.4 (Tiger) as that still struggled with the same issue many years on which I have to suffer when building from pkgsrc. Despite having a system with 2GB of RAM, memory utilisation rarely goes above 512MB.

The idea of having to compile the system timezone in the kernel amused me though it was stated that with 4.3BSD Tahoe, support for the Olson timezone database that we are now familiar with was first added, allowing individual processes to select a set of rules.

I enjoyed the filesystem chapter as I learnt about the old berkley filesystem and the “new” which evolved into what we use today, the performance issues with the old filesystem due to the free list becoming scrambled with the age of the filesystem (in weeks), resulting in longer seek times and the amount of space wasted as a function of block size.

Although the old filesystem provided transfer rates of up to 175 Kbyte per second when it was first created, the scrambling of the free list caused this rate to deteriorate to an average of 30 Kbyte per second after a few weeks of moderate use.

The idea of being rotationally optimal to reduce seek times and implementing mechanisms to account for that was very interesting to read about.

To simplify the task of locating rotationally optimal blocks, the summary information for each cylinder group includes a count of the available blocks at different rotational positions. Eight rotational positions are distinguished, so the resolution of the summary information is 2 milliseconds for a 3600 revolution-per-minute-drive.

Though this is not so valid today with traditional spindle disks as there is not a 1:1 mapping between the physical location & logical representation of the blocks on disk.

The book is a bargain second hand and worth it for the BSD archeology.

Two months after the beginning of the first implementation of the UNIX operating system, there were two processes, one for each of the terminals of the PDP-7. At age 10 months, and still on the PDP-7, UNIX had many processes, the fork operation, and something like the wait system call. A process executed a new program by reading a new program in on top of itself. The PDP-11 system (first edition UNIX) saw the introduction of exec. All these systems allowed only one process in memory at a time. When PDP-11 with memory management (a KS-11) was obtained, the system was modified to permit several processes to remain in memory simultaneously, in order to reduce swapping. But this modification did not apply to multiprogramming, because disk I/O was synchronous. This state of affairs persisted into 1972 and the first PDP-11/45 system. True multiprogramming was finally introduced when the system was rewritten in C. Disk I/O for one process could then proceed while another process ran. The basic structure of process management in UNIX has not changed since that time.

Captive Portals & Brighton

Yesterday I gave a talk at SANE user group on my history with wireless networks as part of the project in Brighton (now defunct) and my experimentation with captive portal software which I began revisiting this time last year. I thought it would be a good opportunity to develop my programming skills by tidying up and modernising parts of the codebase which caused problems, such as things preventing builds on a modern system with clang which is now the default compiler for FreeBSD on i386/AMD64 architectures and OS X. The slides from my talk can be found here.


Back in the early to mid 2000’s there were 2 initiatives to provide public access wifi in Brighton. Loose Connection and, each had a different focus & approach.
Loose Connection was a commercial venture which could be deemed a VAR, they resold a ADSL connection along with a draytek router & that was it. Individual wireless networks with the loose connection SSID dotted around drinking holes in Brighton, the founding(?) company Metranet lives on as a WISP today. was a community driven effort with a technical team of volunteers, predominantly from a service provider / telecoms / networking background. Each node on the network was sponsored by a host who’d buy and run the equipment while the project members managed it.

The project started off based around the fanless VIA mini-itx boards, Prism2 chipset wireless cards, booting linux with hostapd and nocatauth/nocatsplash off a CF card in a IDE to CF adapter.
This was a very flexible platform, if there was no package for it you could build it with ease, problem was that it had a high cost for entry, £250 to £300? so from the start we were looking to reduce costs.
The other issue was though we’d eliminated moving parts, the casing was not suitable for outdoor use.

With the availability of 3rd party firmware and promotional sales of the WRT54G, PierToPier switched hardware platforms as the low cost solution for new nodes.
Tom Grifiths discovered Chillispot around the same time frame and we adopted it due to enhanced functionality it provided, such as RADIUS accounting and working captive portal. We’d previously ran into issues with browser support running nocatauth which by that point was no longer maintained and stability issues with nocatsplash.

Glastonbury 2005
VIA motherboard again this time with CF adapter & mini-pci slot onboard
2x Atheros A/B/G mini-pci cards, one on mini-pci to PCI bridges and second onboard
Stuck to a pelican case with epoxy
Two holes drilled in the side of the case for external antennas

It was early times for support of the cards and the wireless standards. In this era OpenBSD was leading the way in terms of support of hardware and development of their ieee80211 wifi stack, they were the first to reverse engineer the Atheros binary blob HAL (years before anyone?) but late to the game for the 802.11g, 11a was enabled from the start but didn’t appear to work – bringing the interface up in .11a hostap mode wouldn’t necessarily work.
Looking at alternatives there was a short lived live environment named WifiBSD which was based around FreeBSD but later moved to NetBSD before development ceased. The support for the Atheros cards was not as good as OpenBSD, hence not wasn’t much use.

The hardware for the Glastonbury nodes were truly terrible, all functionality had been wired to a single bus which caused the system to lock hard in most configurations. e.g. you booted from a CF card and tried to bring up a wireless interface. The only way to use the system was to disable everything that wasn’t needed in the BIOS including VGA, if there was an issue, you’d have to factory reset the BIOS before diagnosing.

For the wireless network at Glastonbury, the 11a 5GHz network was used as the backhaul while the 11b/g interface was used for connecting wireless clients. No captive portal, connect to the AP & off you go.

We arrived onsite on Tuesday, starting getting things running, Thursday morning the rain and lightning started things went downhill from there. loss of connectivity between the backhaul links meant things fell apart.

By the time we’d discovered Chillipot the project had a 1.0 release out which had preliminary support for FreeBSD. The website claimed only FreeBSD 5 and up were supported, I created a port and submitted it for inclusion in the tree, net-mgmt/chillispot was born, Edwin@ from the ports team fixed the code so that it’d work on previous releases. I then moved onto creating a OpenBSD port, this was slightly harder and the final peace was actually resolved by a Steve Davies. I got the code to build on OpenBSD but networking wouldn’t work. This turned out to be because an additional 4 bytes needed to be allocated which Steve fixed. It never made it into the OpenBSD tree (only tested it on i386 and SPARC, it used strcpy() everywhere and didn’t run on SPARC) but it can be found in ports-wip. I then moved onto creating a live CD environment based on FreeBSD 6 using freesbie for advocacy purposes named BrightonChilli. The idea was to remove the hurdle of going through the installation process and provided an environment that just needed the configuration of network interfaces and chillispot. A person with previous experience of running chillispot would be familiar and a new user would not be too out of place.
This was in the days of X configuration being a part of sysinstall which could hang if Xconfigure was run and you’d have to start the install process again as the install was incomplete (for a newcomer). I was interviewed on BSDtalk #73 regarding BrightonChilli.

PierToPier also produced its own Linux image named Muddy Linux targeted for x86 hardware that ran the necessary stack to serve as a node on the network.
After Chillispot 1.1.0, the project went quiet, there was no answer from the founding developer for quite a while and eventually the web hosting stopped and the domain expired.
The community rehomed to and development continued in Coovachilli which was founded by David Bird, a contributor to Chillispot.
Coovachilli initially lacked support for FreeBSD but it was eventually added in by David and net-mgmt/coovachilli was born in ports. Not much else was done after that until a year ago. With FreeBSD 10 and the switch to clang, the codebase needed attention, first step was to get it to build correctly with GCC. The use of error_t from glibc caused the build to fail as it’s not available in FreeBSD, ensuring this was declared allowed the build to complete successfully. To resolve build issues with clang, nested functions were separated out. Any function with missing prototypes & parameter lists were addressed next.
struct ifreq had been marked as deprecated since 2000 and was finally removed in FreeBSD 10. The *BSD specific sections of Coova were switched out to the new struct ifaliasreq & Linux was left to use the pre-existing method. There was extensive use of macros for the logging functionality, these were dropped in favour of using the existing standard syslog(3) with the correct log level defined. This had the benefit of revealing issues which were not detected previously such as incorrect format specifiers.
There are still many things that need to be cleared up, the 3rd party functions added in are particularly problematic and will probably be my next task to replace with standard components.
CoovaChilli & Chillisport have seen large scale deployments thanks to use by Fon, o2 and Google which now owns Coova.


Cross-platform software packaging with pkgsrc

I recently gave a brief talk titled “cross-platform packaging, from Oracle Solaris to Oracle Linux & many more Operating systems using pkgsrc” at Solaris SIG (formerly London OpenSolaris User Group), where I tried to describe the benefits of pkgsrc for a system administrator.¬†This was the second talk on the subject of software packaging at these meetings, the first was by Mandy Waite back in 2009 on the OpenSolaris SourceJuicer which used the spec files usually offered for generating rpm packages.¬†There were no slides and I was quite nervous so I thought I’d write this post to clarify things.

pkgsrc is a portable framework for building software with more than 12000 packages available (with variants of packages, more than 15000) on 22 different platforms (see platform notes for level of support).

pkgsrc is inspired from FreeBSD ports. As with ports, each unique package is assigned a sub-directory in a parent directory named after the relevant category. This directory contains a minimum of a Makefile, a file containing cryptographic checksums of the source files, a file containing a description and a packing list which contains a list of files which will be installed. Sometimes there may be additional files which need to be installed or patches which need to be included as well. Though patches exist in the tree, we strive to upstream changes as much as possible to a) be good netizens b) make maintenance easier going forward.

Packages which are available to be built from pkgsrc generally do not have changes outside what’s needed to integrate with the framework or build correctly on various systems where additional flags need to be specified. There is a mechanism in place for packages to offer build time options to enable or disable functionality.

We try to use components offered by host operating system if viable but offer the facility to rebuild against versions available from pkgsrc.

If you’re mandated to use specific packages for system wide use but require tools for use personally, there is a unprivileged mode where packages are installed within ~/pkg (or another prefix of your choice) and the OS is left untouched.

There is a dedicated security team responsible for keeping a vulnerabilities file up to date so that packages installed from the tree that are affected will be flagged. There is also support for signed packages which utilises gpg for signing.

To start using pkgsrc you either need to run the bootstrap script in pkgsrc/bootstrap (fetch & uncompress pkgrc.tar.gz) or obtain a prebuilt bootstrap kit for your OS from If you bootstrap manually using the script you’ll end up with your own copy of the bootstrap kit which you can distribute amongst other systems running the same operating system.¬†With a system bootstrapped, you have the necessary tools to build software from the pkgsrc tree or fetch and install prebuilt binaries.

Assuming that your bootstrap kit is /usr/pkg & you uncompressed your pkgsrc archive to /usr/pkgsrc and you wanted to install nginx
cd /usr/pkgsrc/www/nginx
/usr/pkg/bin/bmake install

You can setup a build environment using the pkgsrc/mk/pbulk/ script which will allow you to build packages in bulk or individually for the purpose of distribution across multiple system.

Solaris SIG name badge

IBM ThinkPad X60s – Part 1 “headless brick”


Since the announcement regarding the X60 series becoming certified by the FSF I’ve been interested in getting my hands on one. Not because of the political statement but I really wanted to play with coreboot on something other than my Alix which at the time previously had ended disasterously.
Luckily my regular used computer hardware pusher told me he had an X60s albeit with a broken screen which I could have. I decided to put off collecting it until I had managed to source a replacement screen cheaply so that there’s one less computer to move around with, come summer time there were “compatible” panels being advertised on ebay for between ¬£40 to ¬£80, I collected the ThinkPad & eventually ordered a panel listed specifically for the model ThinkPad I had (1704-GL5).
Once the panel had showed up, I began stripping the display assembly to swap LCD panels, fairly straight forward to do apart from the bits of tape which stick to the side frame & LCD panel. With the old panel out, I began trying to fit the new panel in the frame, the screw holes lined up though things were slightly off alignment, the cover still fitted so I continued.
I connected the panel to the motherboard via the ribbon & to make sure the display actually worked before tightening the screws I switched the laptop on. The fan on the laptop spunup and the moment when the display is meant to become active the system switched off. Subsequent boot attempts resulted in even short runs before switching off. The panel the seller had sent was not compatible with the ThinkPad & the pinout on the panel was completely different.

I was pointed to the X60 fuse list & advised to check those fuses were ok. Stripping the board & checking the fuses with a multimeter for circuit continuity showed that fuse F5 had blown (responsible for the LCD inverter).
I was unsure of the state of the motherboard at this point and whether there was further damage caused to the board. As I had stripped the system to test the fuses, the CMOS battery had been disconnected which obviously caused BIOS settings to be reset. Not sure of which, but on the first power attempt after a reset the system either opts to run from the first connected display or the VGA out, either way the system completed post with an external monitor connected, pressing the hot key to switch to the LCD panel exhibited the previous behavior and the system switching off promptly.
I obtained a full refund for the LCD display from the seller & the replacement fuses were with me after a couple of weeks of shipping blunders from RS.


I’d never attempted to do surface-mount re-soldering before, attempting to remove & re-solder components on a scrap circuit board didn’t go too well, not sure if it was the soldering iron I was using or the solder used on the components but I didn’t make any component move. Thanks to the kindness of a member of the Hackspace with experience who offered to help with the use of 2 soldering irons the fuse came off & they were able to replace it with a new fuse. The problem was fixed, the system was able to drive the original (broken) panel and complete post successfully.


Returning to search on ebay, I sourced a complete display assembly for £21 which saved the hassle of trying to fit thing back again into the old assembly.
The system was fully functional once again & ready for playing with CoreBoot.


coreboot on PC Engines Alix 2c3

I’ve been following coreboot (formally LinuxBIOS) for quite a while but my last attempt to run it ended disastrously a couple of years back on the defunct coreboot 3 branch, requiring an RMA for a new Alix 2c3 board & the purchase of additional tools.
A ThinkPad X60s was offered up by my friendly local “old computer” pusher which I chose to try coreboot on but before I bricked that, I thought I’d try the Alix again as I was in possession of the everything needed to debug or recover the board if things went wrong.
A lot of time was wasted trying to get things built on OS X & diverged into briefly trying CentOS 7 (systemd, run away) before settling on Debian 7.6 to build on.

Running apt-get install gcc make libncurses-dev doxygen iasl gdb flex bison will install the necessary dependencies to build coreboot on Debian following the build guide.

Selecting a board from the menu should set the ROM size as well but I was advised to double check, on the Alix 2c3 the flash chip is located under the board, my board had a AMIC A49LF040ATY-33F.

Once the image was built, coreboot.rom was copied to the Alix running FreeBSD/i386 10-RELEASE & flashed with sysutils/flashrom built from ports.
Flashrom currently assumes that there’s a device node named /dev/cpu0 and sysutils/devcpu-data offers the necessary device but on FreeBSD it’s named /dev/cpuctl0, a symlink allows you to work around the hardcoded assumption once the micro_code service has been started.

service micro_code onestart
ln -s /dev/cpuctl0 /dev/cpu0

coreboot can then be programmed with flashrom -p internal:laptop=this_is_not_a_laptop -w ~/coreboot.rom

flashrom v0.9.7-r1711 on FreeBSD 10.0-RELEASE-p7 (i386)
flashrom is free software, get the source code at

Calibrating delay loop… delay loop is unreliable, trying to continue OK.
WARNING! You may be running flashrom on an unsupported laptop. We could
not detect this for sure because your vendor has not setup the SMBIOS
tables correctly. You can enforce execution by adding
‘-p internal:laptop=this_is_not_a_laptop’ to the command line, but
please read the following warning if you are not sure.

Laptops, notebooks and netbooks are difficult to support and we
recommend to use the vendor flashing utility. The embedded controller
(EC) in these machines often interacts badly with flashing.
See the manpage and for details.

If flash is shared with the EC, erase is guaranteed to brick your laptop
and write may brick your laptop.
Read and probe may irritate your EC and cause fan failure, backlight
failure and sudden poweroff.
You have been warned.
Proceeding anyway because user forced us to.
Found chipset “AMD CS5536”. Enabling flash write… OK.
Warning: unexpected second chipset match: “AMD CS5536”
ignoring, please report lspci and board URL to
with ‘CHIPSET: your board name’ in the subject line.
Found AMIC flash chip “A49LF040A” (512 kB, LPC) at physical address 0xfff80000.
Reading old flash chip contents… done.
Erasing and writing flash chip… Erase/write done.
Verifying flash… VERIFIED.

Power cycling the box will result in text output or garbage depending on if your console speed settings have changed or not, coreboot defaults to the speed of 115200bps, the factory default speed of Alix is 38400bps & the default console speed of FreeBSD is 9600bps.

When attempting to generate a new image with different settings be sure to make clean before starting.
In my image I removed the “PS/2 keyboard init” option from the “Generic Drivers” menu. Still todo is building an image with PXE support, test booting other BSD’s, try different payloads, in particular Open Firmware & generate status data for submission to change board support status on wiki.


Through the previously failed attempt to run coreboot 3 I ended up with a LPC.1a & a POST.5a mini-pci board, the LPC.1a is absolutely essential for testing if you do not have more advanced equipment which would allow you to re-program the flash chip. The LPC.1a is a secondary BIOS chip which can override the onboard the flash, allowing you to boot the system & reprogram the onboard chip again. J2 jumper allows you to select which chip to boot from. When reflashing the chip on the motherboard with flashrom with the LPC.1a inserted (and set to read only), flashrom trashed the image on the onboard chip. Rebooting the system & carefully removing the LPC.1a before re-flashing allowed the process to complete successfully.

GeekLAN is 10 years old

This blog started life with a zero dot release of wordpress on the end of a 512kb down/128k up cable modem connection 10 years ago. Originally I had intended¬†to host it on my AlphaStation which at the time was acting as my gateway, running OpenBSD. Unfortunately gettext was broken on Alpha at the time¬†which¬†meant though php was available I couldn’t build extensions such as the mysql one, I had a slot 1 PIII which was my previous gateway using RRAS on¬†Windows 2000 Server, it replaced the AlphaStation¬†and assumed the role of gateway again, this time on OpenBSD.

Up until 2009 this blog was served from my bedroom by then on a VIA C7 mini-its board with a ADSL connection. At some point it gained an SSL certificate from CAcert¬†& IPv6 connectivity. Through this domain I discovered that NTL overrode the TTL values for records in the early days, caching DNS records for a week by default.¬†Blocked several IP addresses from Thailand for excessive hits to the site. Of¬†all¬†of the computers which I collected over the years, most are now gone.¬†I still have the¬†Cobalt Qube2, some Macs and the ThinkPad X61s, the rest found¬†new homes or where thrown away. The most popular posts so far have been on Apple products, the post that’s still holds true is the Solaris installer misreporting disk failure if it finds a disk label other than the one it was expecting.


Book review: The Art of Unix Programming

I picked this book by mistake, assuming that it was going to be a technically detailed book in line with the Advanced Programming in the Unix Environment book written by the late Richard Stevens, it turned out to be much more high level than that but I was not disappointed, It’s been a pleasure to read whilst travelling over the last month.
The book is 20 chapters split across four parts (context, design, implementation, community) with commentary from some big names of the UNIX world. There are lots of great advice in the book but I would look at what’s now available in regards to software today if I was looking to implement something. It does explain why lots of software relies on some common (and heavy weight?) components. Let me explain, long ago I was unaware that packages for the -current branch of OpenBSD were being built, whenever I grudgingly tried a new snapshot I went through & built my packages from the ports tree after a fresh install, then something would depend on XML related components & then pull in a bunch of things which would involve building ghostscript, on a Sun Blade 100, between Firefox & ghostscript, 24 hours would easily be wasted, I now understand that all that wasted time was thanks to someone taking the advice of ESR on how to prepare documentation for a software project.
Besides the dubious software recommendation (11-year-old book?) everything is¬†explained in a clear manner that’s very easy to read.

Rule of Robustness: Robustness is the child of transparency and simplicity.
Rule of Generation: Avoid hand-hacking; write programs to write programs when you can.
Rule of Optimization: Prototype before polishing. Get it working before you optimize it.
Rule of Diversity: Distrust all claims for ‚Äúone true way‚ÄĚ.
Rule of Extensibility: Design for the future, because it will be here sooner than you think.

The Pragmatic Programmer articulates a rule for one particular kind of orthogonality that is especially important. Their ‚ÄúDon‚Äôt Repeat Yourself‚ÄĚ rule is: every piece of knowledge must have a single, unambiguous, authoritative representation within a system. In this book we prefer, following a suggestion by Brian Kernighan, to call this the Single Point Of Truth or SPOT rule.

The book is critical of Microsoft & their approach to software, explaining some of the design decisions (some inherited from the world of VMS).

From a complexity-control point of view, threads are a bad substitute for lightweight processes with their own address spaces; the idea of threads is native to operating systems with expensive process-spawning and weak IPC facilities.

the Microsoft version of CSV is a textbook example of how not to design a textual file format.

Criticisms of MacOS are of version 9 and prior which don’t really apply to OS X e.g. single shared address space. There are explanations of why things are such in the world of Unix and lots of great advice.

The ‚Äôrc‚Äô suffix goes back to Unix‚Äôs grandparent, CTSS. It had a command-script feature called “runcom”. Early Unixes used ‚Äôrc‚Äô for the name of the operating system‚Äôs boot script, as a tribute to CTSS runcom.

most Unix programs first check VISUAL, and only if that’s not set will they consult EDITOR. That’s a relic from the days when people had different preferences for line-oriented editors and visual editors

When you feel the urge to design a complex binary file format, or a complex binary application protocol, it is generally wise to lie down until the feeling passes.

One of the main lessons of Zen is that we ordinarily see the world through a haze of preconceptions and fixed ideas that proceed from our desires.

Doug McIlroy provides some great commentary too

As, in a different way, was old-school Unix. Bell Labs had enough resources so that Ken was not confined by demands to have a product yesterday. Recall Pascal‚Äôs apology for writing a long letter because he didn‚Äôt have enough time to write a short one. ‚ÄĒDoug McIlroy

I’d recommend the book for anyone involved with computers and not necessarily involved with Unix or open source variants/likes. The author does a great job of explaining the theory of an approach to developing software and the operating it typically runs on, It’s accessible, easy to read and doesn’t require a computer to work through. You may need one however¬†if you want to read it online for free.

My ideal for the future is to develop a filesystem remote interface (a la Plan 9) and then have it implemented across the Internet as the standard rather than HTML. That would be ultimate cool. ‚ÄĒKen Thompson

Bricking & Unbricking a Dell Inspiron Mini 9

I dusted off an old Dell Mini 9 netbook I had lying around, I’d stopped using it as the netbook refused to charge its battery once it’d run down completely (easily do-able if the machine is used lightly & could go up to a month without use), after the second battery It happend to, I gave up on it.
A side effect of the battery being run down was that it was not possible to flash the BIOS through windows, Phoenix WinPhlash refuses to write to the flash if there is no battery detected & exits with error code: -144.
This is not an issue on DOS using the phlash16 utility. Here lies a different kind of madness, as it’s attempting to write to the flash it stops using the mains & switches to battery power source, which if you have an unchargable battery results in bricking the netbook.
At this point, you can perform a flash recovery using a boot floppy containing a special boot sector, the phlash16 utility, some library and a copy of the image to flash named as BIOS.WPH generated with the BIOS recovery tool.
To initiate the recovery mode, the power and battery need to be removed, a USB floppy drive connected (of the left hand side ports only the USB port closest to the SD card reader worked on mine). With the Fn & B keys held down on the keyboard, connect the power whilst continuing to hold the keys down. At this point the power light should switch on. Press the power button & when you hear a beep, let go of the keyboard keys.
After a moment the system will begin reading the floppy & once reflashing commences, the system will begin beeping, once it has finished the system will reboot & startup normally.
I was unable to find working links for prepared images which I could write using dd so instead had to resort to finding another machine running Windows XP & a USB floppy drive but I’ve imaged the floppies I created so hopefully It wont be a repeated exercise.

The images can be written to a USB flash drive & used to recover a Mini 9

A04 Bios recovery image
A07 Bios recovery image

If you have a faulty or uncharged battery & you intend to use the phlash16 utility, remove the battery before attempting to flash the BIOS.