Book review: The Design and Implementation of the 4.3BSD UNIX Operating System

The Design and Implementation of 4.3BSD UNIX Operating System
According to my photographs, I picked up this book in February of this year. With a 105 sections spread over 13 chapters I’ve been working through the book slowly at a section a day. Despite being a technical subject, the book does a very good job of explaining the operation system at a high level without becoming a study of the source code. There are snippets of source code & pseudo code to compliment the text and an extensive list of papers for reference at end of each chapter for those that wish to dig deeper.

I had previously attempted to complete the Minix book, Operating Systems: Design And Implementation but struggled with the extensive source reference. switching back and fourth between chapters or the requirement for a computer to view the source code was not a viable option. I took a chance on this book as used copies are available on Amazon for the cost of a postage which is less than a couple of pounds. The book is well written and enjoyable to read, while implementation details may not be completely applicable to modern BSD variants The fundamental details may still hold true in most cases if not providing a historical background around the technical challenges they faced at the time. What I liked with the Minix was that it provided lots of background to accommodated a beginner and get a reader up to speed though I much preferred the ability to read this book by itself without requiring access to the source code.

I found some of the details in the interprocess communication part a little unclear at times but enjoyed the filesystem and memory management chapters the most and the terminal handling chapter the least though I did learn of Berknet there, aswell as many other historical artefacts throughout the book, some of which I tweeted under the hashtag di43bsd.

Berknet is an obsolete batch-oriented network that was used to connect PDP-11 and VAX UNIX systems using 9600-baud serial lines. Due to the overhead of input processing in the standard line discipline, a special reduced-function network discipline was devised.

The 4.3BSD kernel is not partitioned into multiple processes. This was a basic design decision in the earliest versions of UNIX. The first two implementations by Ken Thompson had no memory mapping at all, and thus made no hardware-enforced distinction between user and kernel space. A message-passing system could have been implemented as readily as the actually implemented model of kernel and user processes. The latter was chosen for simplicity. And the early kernels were small. It has been largely the introduction of more and larger facilities (such as networking) into the kernel that has made their separation into user processes an attractive prospect — one that is being pursued in, for example, Mach.

The book breaks down the percentage of components in each category (such as headers) which are platform independent and platform specific. With a total of 48270 lines of platform independent code versus 68200 lines of platform specific code, the 4.3BSD kernel was largely targeted at the VAX.

From the details on the implementation of mmap() in the BSD memory management design decisions section it was interesting to read about virtual memory subsystems of old

The original virtual memory design was based on the assumption that computer memories were small and expensive, whereas disk were locally connected, fast, large, and inexpensive. Thus, the virtual-memory system was designed to be frugal with its use of memory at the expense of generating extra disk traffic.

It made me think of Mac OS X 10.4 (Tiger) as that still struggled with the same issue many years on which I have to suffer when building from pkgsrc. Despite having a system with 2GB of RAM, memory utilisation rarely goes above 512MB.

The idea of having to compile the system timezone in the kernel amused me though it was stated that with 4.3BSD Tahoe, support for the Olson timezone database that we are now familiar with was first added, allowing individual processes to select a set of rules.

I enjoyed the filesystem chapter as I learnt about the old berkley filesystem and the “new” which evolved into what we use today, the performance issues with the old filesystem due to the free list becoming scrambled with the age of the filesystem (in weeks), resulting in longer seek times and the amount of space wasted as a function of block size.

Although the old filesystem provided transfer rates of up to 175 Kbyte per second when it was first created, the scrambling of the free list caused this rate to deteriorate to an average of 30 Kbyte per second after a few weeks of moderate use.

The idea of being rotationally optimal to reduce seek times and implementing mechanisms to account for that was very interesting to read about.

To simplify the task of locating rotationally optimal blocks, the summary information for each cylinder group includes a count of the available blocks at different rotational positions. Eight rotational positions are distinguished, so the resolution of the summary information is 2 milliseconds for a 3600 revolution-per-minute-drive.

Though this is not so valid today with traditional spindle disks as there is not a 1:1 mapping between the physical location & logical representation of the blocks on disk.

The book is a bargain second hand and worth it for the BSD archeology.

Two months after the beginning of the first implementation of the UNIX operating system, there were two processes, one for each of the terminals of the PDP-7. At age 10 months, and still on the PDP-7, UNIX had many processes, the fork operation, and something like the wait system call. A process executed a new program by reading a new program in on top of itself. The PDP-11 system (first edition UNIX) saw the introduction of exec. All these systems allowed only one process in memory at a time. When PDP-11 with memory management (a KS-11) was obtained, the system was modified to permit several processes to remain in memory simultaneously, in order to reduce swapping. But this modification did not apply to multiprogramming, because disk I/O was synchronous. This state of affairs persisted into 1972 and the first PDP-11/45 system. True multiprogramming was finally introduced when the system was rewritten in C. Disk I/O for one process could then proceed while another process ran. The basic structure of process management in UNIX has not changed since that time.

Book review: The Art of Unix Programming

I picked this book by mistake, assuming that it was going to be a technically detailed book in line with the Advanced Programming in the Unix Environment book written by the late Richard Stevens, it turned out to be much more high level than that but I was not disappointed, It’s been a pleasure to read whilst travelling over the last month.
The book is 20 chapters split across four parts (context, design, implementation, community) with commentary from some big names of the UNIX world. There are lots of great advice in the book but I would look at what’s now available in regards to software today if I was looking to implement something. It does explain why lots of software relies on some common (and heavy weight?) components. Let me explain, long ago I was unaware that packages for the -current branch of OpenBSD were being built, whenever I grudgingly tried a new snapshot I went through & built my packages from the ports tree after a fresh install, then something would depend on XML related components & then pull in a bunch of things which would involve building ghostscript, on a Sun Blade 100, between Firefox & ghostscript, 24 hours would easily be wasted, I now understand that all that wasted time was thanks to someone taking the advice of ESR on how to prepare documentation for a software project.
Besides the dubious software recommendation (11-year-old book?) everything is explained in a clear manner that’s very easy to read.

Rule of Robustness: Robustness is the child of transparency and simplicity.
Rule of Generation: Avoid hand-hacking; write programs to write programs when you can.
Rule of Optimization: Prototype before polishing. Get it working before you optimize it.
Rule of Diversity: Distrust all claims for “one true way”.
Rule of Extensibility: Design for the future, because it will be here sooner than you think.

The Pragmatic Programmer articulates a rule for one particular kind of orthogonality that is especially important. Their “Don’t Repeat Yourself” rule is: every piece of knowledge must have a single, unambiguous, authoritative representation within a system. In this book we prefer, following a suggestion by Brian Kernighan, to call this the Single Point Of Truth or SPOT rule.

The book is critical of Microsoft & their approach to software, explaining some of the design decisions (some inherited from the world of VMS).

From a complexity-control point of view, threads are a bad substitute for lightweight processes with their own address spaces; the idea of threads is native to operating systems with expensive process-spawning and weak IPC facilities.

the Microsoft version of CSV is a textbook example of how not to design a textual file format.

Criticisms of MacOS are of version 9 and prior which don’t really apply to OS X e.g. single shared address space. There are explanations of why things are such in the world of Unix and lots of great advice.

The ’rc’ suffix goes back to Unix’s grandparent, CTSS. It had a command-script feature called “runcom”. Early Unixes used ’rc’ for the name of the operating system’s boot script, as a tribute to CTSS runcom.

most Unix programs first check VISUAL, and only if that’s not set will they consult EDITOR. That’s a relic from the days when people had different preferences for line-oriented editors and visual editors

When you feel the urge to design a complex binary file format, or a complex binary application protocol, it is generally wise to lie down until the feeling passes.

One of the main lessons of Zen is that we ordinarily see the world through a haze of preconceptions and fixed ideas that proceed from our desires.

Doug McIlroy provides some great commentary too

As, in a different way, was old-school Unix. Bell Labs had enough resources so that Ken was not confined by demands to have a product yesterday. Recall Pascal’s apology for writing a long letter because he didn’t have enough time to write a short one. —Doug McIlroy

I’d recommend the book for anyone involved with computers and not necessarily involved with Unix or open source variants/likes. The author does a great job of explaining the theory of an approach to developing software and the operating it typically runs on, It’s accessible, easy to read and doesn’t require a computer to work through. You may need one however if you want to read it online for free.

My ideal for the future is to develop a filesystem remote interface (a la Plan 9) and then have it implemented across the Internet as the standard rather than HTML. That would be ultimate cool. —Ken Thompson

Book review : Kerberos, The definitive guide

Kerberos & AFS have been two technologies I’ve wanted to deploy for a long time but based on my experience with Kerberos in Windows 2000 & and studies for MCSE I had made myself believe that it would be a painful task, I purchased this book a couple of years back but never got around to reading it properly until the start of the new year. The book is divided into 10 chapters, the first 3 explain how Kerberos works conceptually, from there on the book covers the practical aspects, how to deploy Kerberos using the MIT, Heimdal & Windows implementation, how to troubleshoot common issues, improve security, integrate applications & services, implement cross realm authentication, windows & UNIX integration, finishing off with the future of Kerberos.
The book uses FreeBSD as the OS which the UNIX examples are demonstrated on though Kerberos is built from source. I also used FreeBSD to perform my test installation but instead opted to use the Heimdal implementation which comes bundled as standard in the base OS of the BSDs. Implementation was really simple, once the KDC was up & the necessary SRV records were in place, telnet authentication worked seamlessly and after I’d set GSSAPIAuthentication yes in my ssh(1) & sshd(8) config files, SSH also worked seamlessly. Only thing that caught me out was Heimdal in FreeBSD base uses DNS where as the book assumes that this is switched off (not sure if this feature was switched off by default at the time & has now changed or it’s just the FreeBSD bundled version which has it on by default). The information for troubleshooting & some of security is still relevant but other than that it is badly outdated, discussing DES encryption & the lack of support for RC4 encryption which was the default cipher used by Windows 2000. Setting up a slave KDC has also change since this book was published in Heimdal, you now need a hprop/hostname principal for each slave server where as the book recommends host/hostname principals which doesn’t work with Heimdal anymore.

Looking around, you will still see references to Windows 2000 when doing Kerberos implementation eg in the current Heimdal documentation, I’m not sure if this is still applicable to the latest version of Windows or it’s there for historical reference.
If I were looking to learn about Kerberos, specifically Heimdal, I would use the official documentation & the Kerberos5 article on the FreeBSD handbook instead of buying this book as there is too much outdated advice in this book that no longer applies.
Ignoring the outdated best practices, the initial implementation information has remained the same over the year & it’s amazingly easy to deploy in a lab scenario for testing.

Book Review: Pro DNS and BIND

So this is not a new book by any means, bought in 2007, published in 2005, covering BIND 9.x & now succeeded by Pro DNS and BIND 10, I’m on a mission to try & clear as much of my book shelf of books as I can, with ebooks & daily deals from publishers the digital shelf in ibooks is by no means shrinking while I’ve stopped buying books in print. My back is thankful for it & large reference books happily sit in digital format in reach when onsite. Anyway, back to the book this post is about, the book is a polished up version of the DNS for Rocket Scientists Guide which you most certainly would’ve come across if searching for answers to BIND & DNS related questions on the web, with a chapter on DNSSEC which is not on the website for added value.
The book is split into six parts:

  • Principles and Overview
  • Get Something Running
  • DNS Security
  • Reference
  • Programming
  • Appendixes
  • I read the first eleven of fifteen chapters which took me to the end of the DNS Security part, the last three part are all reference material such as BIND API, RFCs & configuration file parameter lists.
    Like the online guide the book is full of useful information & a very easy read apart from the DNS Security part. The “Securing Zone Transfers” felt out of place and jumped into using the dnssec-keygen with no prior reference to it, I struggled with the DNSSEC chapter also but that may have been more to do with it being my first exposure to the topic. The only thing I found annoying was the repeated reference to the backslash representing the spanning to a new line for every paragraph proceeding a command snippet.

    With reading this book and a review of deploying DNSSEC in the Intro to DNSSEC video from BSDCan 2012 I am looking forward to deploying DNSSEC via both DS & DLV as I have registrar support for some TLDs but not ccTLDs.

    Book Review: Implementing Cisco IOS Network Security (IINS)

    So I wrote up a review on the Cisco Press self-study guide for the 640-553 exam which I finished readuing this weekend & while double checking things I noticed that the 640-554 exam topics has already been announced last month with the self-study guide for 640-554 due to be published at the end of August, the new exams will follow on from the 1st of October.
    The new book will again be authored by Catherine Paquet so I’m curious how much new content there will be in the new revision.

    There are seven chapters in the current 640-553 book

  • Introduction to Network Security Principles
  • Perimeter Security
  • Network Security Using Cisco IOS Firewalls
  • Fundamentals of Cryptography
  • Site-to-Site VPNs
  • Network Security Using Cisco IOS IPS
  • LAN, SAN, Voice, and Endpoint Security Overview
  • Chapter 1, “Introduction to Network Security Principles” was the most tedious of the seven to read, a long drawn out chapter covering ethics, risk analysis, lots of charts, graphs & cost figures (I managed to get through the chapter by thinking of brass eye every time I came across one), marketing info on Ciscos “self-defending network” & buried amongst all that was some introductory info to different types of attack.

    Chapter 2, “Perimeter Security” covers getting setup (ACS Server on Windows, logging, AAA, views) more product line info & navigating SDM.

    Chapter 3, “Network Security Using Cisco IOS Firewalls” covers the fundamentals of firewalls, quiet a large portion of the chapter is on ACLs & configuring them which didn’t make sense as this is covered on ICND2, followed by a shorter section on configuring the zone based firewall via SDM & the firewall wizard.

    Chapter 4, “Fundamentals of Cryptography” was a good but contained some mistakes, like “DES is considered trustworthy” & “Cryptography researchers have scrutinized DES for nearly 35 years and have found no significant flaws”. These statements are wrong, the DES Cracker proved it in the late 90’s or perhaps this is what they were refering to by “because DES is based on simple mathematical functions, it can easily be implemented and accelerated in hardware”.

    Chapter 5, “Site-to-Site VPNs” was enjoyable & lead on from the foundation laid in the previous chapter, setup was also covered from the console this time.

    Chapter 6, “Network Security Using Cisco IOS IPS” covers the fundamentals on the theory side, how to configure it via SDM & more product intro. This chapter is available as a free sample for download.

    Chapter 7, “LAN, SAN, Voice, and Endpoint Security Overview” was 50/50, I enjoyed the SAN section because it was new to me, so there was new information to learn, the endpoint security section covered various attacks & vulnerabilities mixed up with product line info, the voice section was brief covering fundamentals, threats & defence, I didn’t find it very interesting. The chapter finished up with mitigating L2 attacks.

    I didn’t particularly enjoy this book, the first three chapters were pretty tedious to read but it got better in the later ones, overall it lacked flow & felt thrown together.
    It was also disappointing to see the use of TFTP being encouraged in a security book
    “The system that you choose should support TFTP to make it easy to transfer any resulting configuration files to the router” &
    “The added layer of MD5 protection is useful in environments in which the password crosses the network or is stored on a TFTP server”.
    The book is a combination of marketing material on the product line, some technical theory & mainly instructions to navigate the SDM though the console is covered here & there (main focus is SDM but that looks to change for the new exam to CPP).
    As self-study guides go I thought it was better than Stephen McQuerrys 2 books for the R&S CCNA. I’m looking forward to seeing how the CCNA Security book is, I really enjoyed reading Odoms CCNA books & though I’ve not read any of Kevin Wallaces books before, I found the video content he’s published on youtube very good so I’m looking forward to reading his book to prepare for the 640-553 exam.
    If the exam certification guides are generally on parr with Odoms books then in the future I think I will skip the self-study guides & move straight on to the exam certification guides.