Videos from OSHUG #46

Earlier this year, I gave a short talk at OSHUG about the BSD family of operating systems to a group of open hardware enthusiasts. I previously wrote about the event here
The videos from the event are now available on youtube


Book review: The Design and Implementation of the 4.3BSD UNIX Operating System

The Design and Implementation of 4.3BSD UNIX Operating System
According to my photographs, I picked up this book in February of this year. With a 105 sections spread over 13 chapters I’ve been working through the book slowly at a section a day. Despite being a technical subject, the book does a very good job of explaining the operation system at a high level without becoming a study of the source code. There are snippets of source code & pseudo code to compliment the text and an extensive list of papers for reference at end of each chapter for those that wish to dig deeper.

I had previously attempted to complete the Minix book, Operating Systems: Design And Implementation but struggled with the extensive source reference. switching back and fourth between chapters or the requirement for a computer to view the source code was not a viable option. I took a chance on this book as used copies are available on Amazon for the cost of a postage which is less than a couple of pounds. The book is well written and enjoyable to read, while implementation details may not be completely applicable to modern BSD variants The fundamental details may still hold true in most cases if not providing a historical background around the technical challenges they faced at the time. What I liked with the Minix was that it provided lots of background to accommodated a beginner and get a reader up to speed though I much preferred the ability to read this book by itself without requiring access to the source code.

I found some of the details in the interprocess communication part a little unclear at times but enjoyed the filesystem and memory management chapters the most and the terminal handling chapter the least though I did learn of Berknet there, aswell as many other historical artefacts throughout the book, some of which I tweeted under the hashtag di43bsd.

Berknet is an obsolete batch-oriented network that was used to connect PDP-11 and VAX UNIX systems using 9600-baud serial lines. Due to the overhead of input processing in the standard line discipline, a special reduced-function network discipline was devised.

The 4.3BSD kernel is not partitioned into multiple processes. This was a basic design decision in the earliest versions of UNIX. The first two implementations by Ken Thompson had no memory mapping at all, and thus made no hardware-enforced distinction between user and kernel space. A message-passing system could have been implemented as readily as the actually implemented model of kernel and user processes. The latter was chosen for simplicity. And the early kernels were small. It has been largely the introduction of more and larger facilities (such as networking) into the kernel that has made their separation into user processes an attractive prospect — one that is being pursued in, for example, Mach.

The book breaks down the percentage of components in each category (such as headers) which are platform independent and platform specific. With a total of 48270 lines of platform independent code versus 68200 lines of platform specific code, the 4.3BSD kernel was largely targeted at the VAX.

From the details on the implementation of mmap() in the BSD memory management design decisions section it was interesting to read about virtual memory subsystems of old

The original virtual memory design was based on the assumption that computer memories were small and expensive, whereas disk were locally connected, fast, large, and inexpensive. Thus, the virtual-memory system was designed to be frugal with its use of memory at the expense of generating extra disk traffic.

It made me think of Mac OS X 10.4 (Tiger) as that still struggled with the same issue many years on which I have to suffer when building from pkgsrc. Despite having a system with 2GB of RAM, memory utilisation rarely goes above 512MB.

The idea of having to compile the system timezone in the kernel amused me though it was stated that with 4.3BSD Tahoe, support for the Olson timezone database that we are now familiar with was first added, allowing individual processes to select a set of rules.

I enjoyed the filesystem chapter as I learnt about the old berkley filesystem and the “new” which evolved into what we use today, the performance issues with the old filesystem due to the free list becoming scrambled with the age of the filesystem (in weeks), resulting in longer seek times and the amount of space wasted as a function of block size.

Although the old filesystem provided transfer rates of up to 175 Kbyte per second when it was first created, the scrambling of the free list caused this rate to deteriorate to an average of 30 Kbyte per second after a few weeks of moderate use.

The idea of being rotationally optimal to reduce seek times and implementing mechanisms to account for that was very interesting to read about.

To simplify the task of locating rotationally optimal blocks, the summary information for each cylinder group includes a count of the available blocks at different rotational positions. Eight rotational positions are distinguished, so the resolution of the summary information is 2 milliseconds for a 3600 revolution-per-minute-drive.

Though this is not so valid today with traditional spindle disks as there is not a 1:1 mapping between the physical location & logical representation of the blocks on disk.

The book is a bargain second hand and worth it for the BSD archeology.

Two months after the beginning of the first implementation of the UNIX operating system, there were two processes, one for each of the terminals of the PDP-7. At age 10 months, and still on the PDP-7, UNIX had many processes, the fork operation, and something like the wait system call. A process executed a new program by reading a new program in on top of itself. The PDP-11 system (first edition UNIX) saw the introduction of exec. All these systems allowed only one process in memory at a time. When PDP-11 with memory management (a KS-11) was obtained, the system was modified to permit several processes to remain in memory simultaneously, in order to reduce swapping. But this modification did not apply to multiprogramming, because disk I/O was synchronous. This state of affairs persisted into 1972 and the first PDP-11/45 system. True multiprogramming was finally introduced when the system was rewritten in C. Disk I/O for one process could then proceed while another process ran. The basic structure of process management in UNIX has not changed since that time.

Book review: The Art of Unix Programming

I picked this book by mistake, assuming that it was going to be a technically detailed book in line with the Advanced Programming in the Unix Environment book written by the late Richard Stevens, it turned out to be much more high level than that but I was not disappointed, It’s been a pleasure to read whilst travelling over the last month.
The book is 20 chapters split across four parts (context, design, implementation, community) with commentary from some big names of the UNIX world. There are lots of great advice in the book but I would look at what’s now available in regards to software today if I was looking to implement something. It does explain why lots of software relies on some common (and heavy weight?) components. Let me explain, long ago I was unaware that packages for the -current branch of OpenBSD were being built, whenever I grudgingly tried a new snapshot I went through & built my packages from the ports tree after a fresh install, then something would depend on XML related components & then pull in a bunch of things which would involve building ghostscript, on a Sun Blade 100, between Firefox & ghostscript, 24 hours would easily be wasted, I now understand that all that wasted time was thanks to someone taking the advice of ESR on how to prepare documentation for a software project.
Besides the dubious software recommendation (11-year-old book?) everything is explained in a clear manner that’s very easy to read.

Rule of Robustness: Robustness is the child of transparency and simplicity.
Rule of Generation: Avoid hand-hacking; write programs to write programs when you can.
Rule of Optimization: Prototype before polishing. Get it working before you optimize it.
Rule of Diversity: Distrust all claims for “one true way”.
Rule of Extensibility: Design for the future, because it will be here sooner than you think.

The Pragmatic Programmer articulates a rule for one particular kind of orthogonality that is especially important. Their “Don’t Repeat Yourself” rule is: every piece of knowledge must have a single, unambiguous, authoritative representation within a system. In this book we prefer, following a suggestion by Brian Kernighan, to call this the Single Point Of Truth or SPOT rule.

The book is critical of Microsoft & their approach to software, explaining some of the design decisions (some inherited from the world of VMS).

From a complexity-control point of view, threads are a bad substitute for lightweight processes with their own address spaces; the idea of threads is native to operating systems with expensive process-spawning and weak IPC facilities.

the Microsoft version of CSV is a textbook example of how not to design a textual file format.

Criticisms of MacOS are of version 9 and prior which don’t really apply to OS X e.g. single shared address space. There are explanations of why things are such in the world of Unix and lots of great advice.

The ’rc’ suffix goes back to Unix’s grandparent, CTSS. It had a command-script feature called “runcom”. Early Unixes used ’rc’ for the name of the operating system’s boot script, as a tribute to CTSS runcom.

most Unix programs first check VISUAL, and only if that’s not set will they consult EDITOR. That’s a relic from the days when people had different preferences for line-oriented editors and visual editors

When you feel the urge to design a complex binary file format, or a complex binary application protocol, it is generally wise to lie down until the feeling passes.

One of the main lessons of Zen is that we ordinarily see the world through a haze of preconceptions and fixed ideas that proceed from our desires.

Doug McIlroy provides some great commentary too

As, in a different way, was old-school Unix. Bell Labs had enough resources so that Ken was not confined by demands to have a product yesterday. Recall Pascal’s apology for writing a long letter because he didn’t have enough time to write a short one. —Doug McIlroy

I’d recommend the book for anyone involved with computers and not necessarily involved with Unix or open source variants/likes. The author does a great job of explaining the theory of an approach to developing software and the operating it typically runs on, It’s accessible, easy to read and doesn’t require a computer to work through. You may need one however if you want to read it online for free.

My ideal for the future is to develop a filesystem remote interface (a la Plan 9) and then have it implemented across the Internet as the standard rather than HTML. That would be ultimate cool. —Ken Thompson

Issues with V6 UNIX

From the article Bringing up V6 Unix on the Ersatz-11 PDP-11 Emulator

V6 as distributed is strictly a 20th Century operating system. Literally. You can’t set the date to anytime in the 21st century, for two reasons.
First, the ‘date’ command only take a 2-digit year number. Second, even if you fix that, the ctime() library routine has a bug in it that makes it stop working in the closing months of 1999. (IIRC, it is that the number of 8-hour blocks since the epoch – January 1, 1970 – overflows a 15-bit integer at that point. ‘Vanilla’ V6 C doesn’t have unsigneds.)

I have ‘fixed’ copies of date.c and ctime.c; here and here. You can download them and install them; ctime.o goes in /lib/libc.a:

ar r /lib/libc.a ctime.o
The ‘date’ command has been extended to support 2- and 4-digit year numbers (to be upwardly compatible, the 2-digit ones assume 19xx). It has also been extended (for forgetful people like me 🙂 so that if you type:
date -
it will tell you what order the arguments go in.
The ‘fix’ in ctime() is really kludgy – sorry, I was in a hurry! It will also break in another 15 years or so (when the number of 8-hour periods overflows 16 bits). Someone else can fix that one!

Building RetroBSD on Mac OS X

Earlier this week it was announced on the RetroBSD forum that the official repository had moved to GitHub. Along with the move came changes which fixed previous issues with the build so now it is possible to build a RetroBSD image on Mac OS X.

As covered in the RetroBSD installation document, I installed the required dependencies from MacPorts & used the prebuilt PIC32 toolchain to build RetroBSD.

port install bison byacc flex libelf

The compiler was uncompressed to /usr/local to keep it inline with the code base & reduce the number of changes need to get thing to build.

I’d previously installed the command line tools from within Xcode, while this had installed clang & the rest of the toolchain into /usr, I was missing /usr/include, installing the command line tools manually with xcode-select --install fixed this.

Kernel built from the SVN code base on Google Code:
2.11 BSD Unix for PIC32, revision 892M build 2:
Compiled 2014-03-29 by xxx@xxx:
/xxx/retrobsd/sys/pic32/max32-eth
cpu: 795F512L 80 MHz, bus 80 MHz
oscillator: XT crystal, PLL div 1:2 mult x20
console: tty0 (5,0)
sd0: port SPI2, select pin C14
sd0: type SDHC, size 15339520 kbytes, speed 13 Mbit/sec
phys mem = 128 kbytes
user mem = 96 kbytes
root dev = rd0a (0,1)
root size = 102400 kbytes
swap dev = rd0b (0,2)
swap size = 2048 kbytes
temp0: allocated 30 blocks
/dev/rd0a: 588 files, 8752 used, 93247 free
temp0: released allocation
Starting daemons: update cron

2.11 BSD UNIX (pic32) (console)

Kernel built from the git code base on GitHub:
2.11 BSD Unix for PIC32, revision G38 build 1:
Compiled 2014-04-19 by xxx@xxx:
/xxx/retrobsd/sys/pic32/max32-eth
cpu: 795F512L 80 MHz, bus 80 MHz
oscillator: XT crystal, PLL div 1:2 mult x20
console: tty0 (5,0)
sd0: port SPI2, select pin C14
sd0: type SDHC, size 15339520 kbytes, speed 13 Mbit/sec
phys mem = 128 kbytes
user mem = 96 kbytes
root dev = rd0a (0,1)
root size = 102400 kbytes
swap dev = rd0b (0,2)
swap size = 2048 kbytes
/dev/rd0a: 588 files, 8752 used, 93247 free
Starting daemons: update cron

2.11 BSD UNIX (pic32) (console)

RetroBSD on chipKIT Max32 with Arduino Ethernet shield

I bought myself one of these boards to play with RetroBSD a few months back but with build issues & other distractions in real life I didn’t get very far because the stock MAX32 kernel configuration is for SD card slots wired to the board whereas I was trying to use the SD card slot on an ethernet shield. This resulted in the kernel not being able to detect the SD card, hence no root file system. I’d use prebuilt binaries without trying to build my own but for some reason there is no prebuilt kernel image for the configuration I require. Finally, tonight I managed to get a MAX32-ETH kernel built & flashed onto the board, still unable to build the userland but I was able to get an older prebuilt image from the RetroBSD Autobuild Server which currently has images for build 860 & 778.

Will follow up once I have everything building from scratch.
Building RetroBSD on Mac OS X

2.11 BSD Unix for PIC32, revision 892 build 1:
Compiled 2014-03-29 by xxx@xxx.xxx:
/usr/local/retrobsd/sys/pic32/max32-eth
cpu: 795F512L 80 MHz, bus 80 MHz
oscillator: XT crystal, PLL div 1:2 mult x20
console: tty0 (5,0)
sd0: port SPI2, select pin C14
sd0: type SDHC, size 15339520 kbytes, speed 13 Mbit/sec
phys mem = 128 kbytes
user mem = 96 kbytes
root dev = rd0a (0,1)
root size = 163840 kbytes
swap dev = rd0b (0,2)
swap size = 2048 kbytes
temp0: allocated 47 blocks
/dev/rd0a: 576 files, 8644 used, 154555 free
temp0: released allocation
Starting daemons: update cron

2.11 BSD UNIX (pic32) (console)

login:

UNIX/WORLD Review Sun 2 Workstation, October 1984

I was looking around for information on Lions’ Commentary when I came across this site which happened to be of a journalist who reviewed the Sun 2 workstation for the UNIX/WORLD magazine back in 1984.
The reference to a Sun employee demonstrating the dual head capability made me smile & reminded me of the Project Looking Glass demonstration.

What took me there was his derivative copy of the UNIX v6 source code but I also grabbed the v7 documentation which he posted up as updated PDF (changelog on the page)

a.out

From the paper “First edition Unix: Its creation and restoration

because the PDP-11/20 provided no memory protection against kernel corruption by the running process, it was considered a courtesy for a programmer to yell ‘‘a.out?’’—the name of the Unix assembler’s default output file—before running a new executable for the first time

WMI Window Manager

After a serious plugging by some of the peepz @ BSD Nexus (WIntellect!!) I decide to give the new window manager called WMI a try on my laptop running OpenBSD.
WMI is coming from the same angle as flux & black box in that its a totally stripped down window manager with heavy focus on keyboard shortcuts.
For OpenBSD the site links straight to the OpenBSD ports cvsweb page, where you can download the necessary files to build WMI, the build process went totally smoothely & a package was created in /usr/ports/packages/i386/all/ but as there is a error in one of the Make files, the package doesnt complete the install & outputs a error (will investigate this & keep you updated).

I wasnt particularly impressed with the default theme that WMI comes with (called industrial) but there are 2 aditional themes currently available on the WMI site which you can install, which I did (im currently running the DragonflyBSD theme) & judging from the contents of a theme if you dont like the themes available it wouldnt be very difficult to make your own one, a quick edit of my theme file to add some wallpaper & Its all systems go. 🙂

Automated Uploads with FTP using cron

Today I’ve mostly been messing with IRC bots & toys.
setup a bot (eggdrop) & irc stat generator (pisg) on my server here at home. Unfortunately the channels website is hosted elsewhere, using crontab I created a cron job to run pisg at a regular interval, then planned on using ftp in another job to upload the generated stats page. Unfortunately there no way of telling the ftp command to upload files.
To get around this I resorted to setting a cron job on a script which ran ftp & included the necessary ftp commands to upload the file piped to the ftp command.
Heres the script I used (thnx Dan!):

#!/bin/sh
ftp -i -n ftp.hostname.net <<EOF
user myusern mypasswd
cd /whereu/want2upload/
lcd /wherethe/fileis/locatedlocally/
put index.html
quit
EOF

the crontab entry looks like this
0 * * * * /pathto/sh /pathtothe/script >/dev/null 2>&1