How to open source: going from NetBSD to Linux

TL;DR: some BSD user tries something other and wonder why things are different.

This post has sat in draft form for quite some time. At first it was written with highlighting the NetBSD project in mind and I started thinking about revisiting it recently due to frustration with running a mainstream Linux distribution when investigating

  • how some critical libraries I was running were built
  • what, if any changes were made to them
  • wondering why the source repositories for components were buried away if at all available.

The recent article on LWN titled Toward a “modern” Emacs mentioned the frustration with distributions provided sufficient confirmation bias to get this together and posted. Note: this is not intended as a bragging contest about NetBSD or pkgsrc or a put down for Linux, but perhaps things I’m not grasping and expecting one to be like the other.

Each technology community has a set of norms around how they interact with their technology, with regard to obtaining software for example mobile users obtain theirs from an “app store”. Mac OS/Windows users traditionally would install packaged software but now mostly obtain their software from a store again. It would be odd to be given a source archive and asked to compile the software for yourself as a user on these platforms (if the source code was even available to users). Unix was the opposite, it was common to receive software in source form and have to compile it yourself. By association and nature (Open Source software) so do GNU/Linux distributions, however binary packages are provided and encouraged for use. The packages save a great deal of compilation time and lower the barrier for users which again is a good thing. I get the impression the details regarding source code and changes do not get the same spotlight especially in a security context, for example as I edit this post, among the most recent advisories on the Debian security page is an advisory for ModSecurity, fairly short, lists the CVEs and states “We recommend that you upgrade your modsecurity packages.”. If I’m interested in the actual changes to the package it’s buried five pages away from the advisory. The GUI update manager on my distro goes as far as collapsing the description panel for the updates which I find amusing.

Software updater in Ubuntu

I agree hiding technical detail from a user is a valid case. Actually, while trying to take this screenshot I visited the bug report of the GCC update and with a bit of clicking around, I found a link to a diff of changes. Why can’t the advisories document both paths (build your own or obtain the packages) and allow the user to choose.
I was hoping for something a bit more flexible which would allow me to use what’s in place and also allow me to rebuild the system or parts with ease should I wish/need to.
Relying on a distribution as a means of obtaining gratis binaries to use, at best, isn’t very appealing.
Use of Open Source software in such a way while completely acceptable overlooks the opportunity to mould software to your requirements should you be inclined.
Given a piece of software, to consume provided binaries, avoiding any customisation is akin to bending around an implementation and is actually heading in the opposite direction of what Open Source software is able to allow you to do.
Let me clarify, I’m not saying just because a piece of software is Open Source it must be compiled by every user by them self for maximum benefit (a talk I gave in 2019 was torpedoed by the objection that one should build their own version of Chrome or Firefox 🙂 ).
I’m suggesting that if you are relying on tools of an Open Source nature, you are best off owning your stack.
That is, you take active participation in projects, for you are able to help shape the evolution of your tools through participating and get insight into upcoming changes.
This makes upgrades and maintenance smoother because you are not reliant on a 3rd party and their release cycle for updates, potentially resulting in long gaps between upgrades which could also mean big jumps between major versions when you do upgrade, bringing about many changes since the previous version you were running.
You become familiarised with the process to assemble your tools which helps when you are reasoning about your stack during debugging.
Questions like “are there local changes from your distribution?” are off the table. e.g Linux From Scratch
Bad tools harbour bad habits.
The shortcomings of a bad tool are pushed on to the user/operator who is then forced to tolerate them and work around accordingly, leading to a clumsy workflow. See Poka-yoke
With a system composed of many such components, it becomes harder and harder to think about new ideas or existing problems in a new way because of the mental burden of coping with what is currently in place and adapting, leading to paralysis and surviving in maintenance mode where the system remains static and is kept running because no one dares make a change.

The enjoyment of one’s tools is an essential ingredient of successful work.

Vol. II, Seminumerical Algorithms, Section 4.2.2 part A, final paragraph

Enter NetBSD and pkgsrc which is where I was coming from as a user.
NetBSD is an open source operating system with a focus on portability.
It has been around since the early 1990s and is the oldest Open Source operating system which is still actively developed as well as one of the oldest active source code repositories on the internet today. The lineage of the code base is easily traceable back to the early days of UNIX thanks to the CSRG archive repository. This is not so important as a first port of call for a new user or for day to day operation but provides useful insight during debugging/troubleshooting.
Having the source code alone is not as useful as having access to the source repo and the history of the code base with commit messages (not that all commit messages are useful).
As with the other BSDs, the current source repository plays a prominent role on the front page of the website and very easy to find.
pkgsrc is NetBSD’s sibling packaging system project with a similar focus on portability.
pkgsrc provides a framework to package tens of thousands of pieces of software consistently across many different operating systems.
In combination of the two there is a complete stack to compose a system with, from operating system to a suite of 3rd party software (including Chrome and Mozilla based browsers, FYI! 🙂 ) or to take selected components and extend other systems with.
As an example, a feature of NetBSD is a tool called Rump Kernel. Rump allows you to instantiate an instance of the NetBSD kernel in the user space of another operating system instance. A common use of this in NetBSD is for testing, it is possible to perform tests on vital components of a system, safely, and on failure the result at worse is a failed system process, rather than a system crash. This saves valuable time between iterations when debugging, especially on larger systems where boot processes run into minutes (think about a server with a large number of disks attached, easily ~ 10 minutes or more to POST and probe a full shelf of disks before even getting to booting the operating system). Rump can also be used to supplement functionality on operating systems, saving development time of device drivers or subsystems. An example of this is the use of Rump in the GNU/Hurd operating system to provide a sound system and drivers for sound cards.
pkgsrc with its support for a range of operating systems means that it is possible to unify your workflow across a range of systems with relation to deploying software. This makes it possible to run the same variety of software with identical changes regardless of operating system. pkgsrc also provides the flexibility to select where dependencies are satisfied from, where possible. That is, if the host operating system provides a component as standard, pkgsrc could make use of it rather than building yet another copy of it, or as time goes on, with legacy systems it may be preferred not to use any such components provided by the host operating system but to only make use of components from pkgsrc, this is also possible. Like pkgsrc, NetBSD has its own build framework which makes it easy to build a release or cross build from another operating system which has a compiler installed. It feels very much like NetBSD comes to you and you work on it from your environment of choice rather that you having to change your environment to it in order to work on it, and the tools you become comfortable with you get to take with you to other platforms. You end up with a toolbox with for solving problems.
The GNU eco system itself is a vast toolbox to pick from also but I’m missing the integration and struggling with the fragmentation and the differences in project management if any. Source code up on a project hosting site alone is no good, neither is just a project site without access to the source code repository, you need both to engage with a project, to be able to track changes and to participate in the community. One doesn’t replace the other.

How I ended up here is that I installed Ubuntu because it provided ZFS support out of the box and I didn’t need to worry about things like pinning kernel versions to prevent kernel updates from rendering my machine un-bootable until I built new modules somehow and I thought it would be the easiest way to work through the BPF performance book. My experience with Linux has been with traditional distros, started on Slackware, then onto RedHat 5.x, Suse 6.x, Debian (Woody) and now Ubuntu 20.04. Tried Gentoo once about 15 years ago but never got past building an unbootable system from their live cd env I recall. I have not tried more recent distributions like Arch, void and such. I’m currently playing with Linux From Scratch.

May the source be with you

Marshall Kirk McKusick

5 Replies to “How to open source: going from NetBSD to Linux”

  1. Just to get down to the base, you’re coming from NetBSD, which from what I can tell is a rolling-release compile-everything-from-source distribution with it’s own BSD-based kernel. You needed ZFS, so you’re trying out the latest long-term Ubuntu (20.04)… which is pre-packaged binary with some package lag time, and the ability to rebuild packages locally and/or pick from a vendor built package.

    Which is fine if you need something stood up fast, or you’re burned out on the former.

    But if you need the latest and greatest and it’s not a very popular application… or you need a feature compiled in… you need to compile a version for yourself and maintain a local repo or grab the vendor and see if they compile their own versions. In the Debian/Ubuntu sense for the latter, looking for a “PPA” repo to add.

    Maddening, yes. But at least you got the tools. It’s a matter of how much time you’re willing to spend… and over time, that’ll get shorter.

    I’m burned out of Gentoo after being with it for about 15 years, formerly Slackware. I’ve tried Redhat/Fedora before, and had a hard time with RPM. Debian based distros are *sane* in that regard. After spending half-or-more of a day just updating a Gentoo system, only for it to break or have on-going problems, and “we don’t support it anymore” is an option… I had enough and switched.

    Linux From Scratch is exactly that… and you’ll need to provide your own package manager. Arch et all are also build-from-source distros, although I think they have some more pre-built capabilities. I’ve seen it, and right now… most everyone’s supporting Ubuntu or Fedora. I’ll stick with them. As I said, I’m burnt out.

  2. I don’t think any of the BSDs are rolling-release but NetBSD is definitely not and neither is pkgsrc. NetBSD cuts a major release roughly every 3 years, and pkgsrc has a quarterly release every three months. There are daily builds of the OS whether you are tracking the head of the tree (the bleeding edge where the next major release is cut) or a stable branch (the branch where the next minor release is going to be cut). It’s your choice which path you want to follow. If you don’t like any of that you can take the source code for whichever branch and roll your own release with ease (a single command build.sh).
    I wanted ZFS and then realised I needed to rebuild things with -fno-omit-frame-pointer when I was reading the BPF performance book.
    Initially it was libc but I wondered what if I wanted a release built with that, how would I go about it in the scheme of things (don’t just untar, configure, make, make install over the provided libc only to trip yourself with updates from packages).

  3. > Tried Gentoo once about 15 years ago but never got past building an unbootable system from their live cd env I recall.

    That’s a shame, because that’s the Linux distribution you were looking for, not Ubuntu.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.