I unfortunately will not be presenting my talk at EuroBSDcon 2015 later this week. A family emergency that developed while I was in Ottawa earlier this year came to a head in early August. Things had been pretty hectic up until this point and I didn’t feel up to buttoning down for the next two months to work so I decided to cancel my talk as I just wanted to switch off. Life is now back in motion again as of earlier this month and I intend to pick up from where I left off with this project next month to resubmit next year. I’m sorry I will not be there in Sweden to enjoy the conference with some of you but hopefully see you in 2016 for the next round!
According to my photographs, I picked up this book in February of this year. With a 105 sections spread over 13 chapters I’ve been working through the book slowly at a section a day. Despite being a technical subject, the book does a very good job of explaining the operation system at a high level without becoming a study of the source code. There are snippets of source code & pseudo code to compliment the text and an extensive list of papers for reference at end of each chapter for those that wish to dig deeper.
Finished the design and implementation of 4.3BSD UNIX operating system book, now available for UNIBUS based multiuser system consultancy
— Sevan Janiyan (@sevanjaniyan) August 11, 2015
I had previously attempted to complete the Minix book, Operating Systems: Design And Implementation but struggled with the extensive source reference. switching back and fourth between chapters or the requirement for a computer to view the source code was not a viable option. I took a chance on this book as used copies are available on Amazon for the cost of a postage which is less than a couple of pounds. The book is well written and enjoyable to read, while implementation details may not be completely applicable to modern BSD variants The fundamental details may still hold true in most cases if not providing a historical background around the technical challenges they faced at the time. What I liked with the Minix was that it provided lots of background to accommodated a beginner and get a reader up to speed though I much preferred the ability to read this book by itself without requiring access to the source code.
I found some of the details in the interprocess communication part a little unclear at times but enjoyed the filesystem and memory management chapters the most and the terminal handling chapter the least though I did learn of Berknet there, aswell as many other historical artefacts throughout the book, some of which I tweeted under the hashtag di43bsd.
Berknet is an obsolete batch-oriented network that was used to connect PDP-11 and VAX UNIX systems using 9600-baud serial lines. Due to the overhead of input processing in the standard line discipline, a special reduced-function network discipline was devised.
The 4.3BSD kernel is not partitioned into multiple processes. This was a basic design decision in the earliest versions of UNIX. The first two implementations by Ken Thompson had no memory mapping at all, and thus made no hardware-enforced distinction between user and kernel space. A message-passing system could have been implemented as readily as the actually implemented model of kernel and user processes. The latter was chosen for simplicity. And the early kernels were small. It has been largely the introduction of more and larger facilities (such as networking) into the kernel that has made their separation into user processes an attractive prospect — one that is being pursued in, for example, Mach.
The book breaks down the percentage of components in each category (such as headers) which are platform independent and platform specific. With a total of 48270 lines of platform independent code versus 68200 lines of platform specific code, the 4.3BSD kernel was largely targeted at the VAX.
From the details on the implementation of
mmap() in the BSD memory management design decisions section it was interesting to read about virtual memory subsystems of old
The original virtual memory design was based on the assumption that computer memories were small and expensive, whereas disk were locally connected, fast, large, and inexpensive. Thus, the virtual-memory system was designed to be frugal with its use of memory at the expense of generating extra disk traffic.
It made me think of Mac OS X 10.4 (Tiger) as that still struggled with the same issue many years on which I have to suffer when building from pkgsrc. Despite having a system with 2GB of RAM, memory utilisation rarely goes above 512MB.
The idea of having to compile the system timezone in the kernel amused me though it was stated that with 4.3BSD Tahoe, support for the Olson timezone database that we are now familiar with was first added, allowing individual processes to select a set of rules.
I enjoyed the filesystem chapter as I learnt about the old berkley filesystem and the “new” which evolved into what we use today, the performance issues with the old filesystem due to the free list becoming scrambled with the age of the filesystem (in weeks), resulting in longer seek times and the amount of space wasted as a function of block size.
Although the old filesystem provided transfer rates of up to 175 Kbyte per second when it was first created, the scrambling of the free list caused this rate to deteriorate to an average of 30 Kbyte per second after a few weeks of moderate use.
The idea of being rotationally optimal to reduce seek times and implementing mechanisms to account for that was very interesting to read about.
To simplify the task of locating rotationally optimal blocks, the summary information for each cylinder group includes a count of the available blocks at different rotational positions. Eight rotational positions are distinguished, so the resolution of the summary information is 2 milliseconds for a 3600 revolution-per-minute-drive.
Though this is not so valid today with traditional spindle disks as there is not a 1:1 mapping between the physical location & logical representation of the blocks on disk.
The book is a bargain second hand and worth it for the BSD archeology.
Two months after the beginning of the first implementation of the UNIX operating system, there were two processes, one for each of the terminals of the PDP-7. At age 10 months, and still on the PDP-7, UNIX had many processes, the fork operation, and something like the wait system call. A process executed a new program by reading a new program by reading a new program in on top of itself. The PDP-11 system (first edition UNIX) saw the introduction of exec. All these systems allowed only one process in memory at a time. When PDP-11 with memory management (a KS-11) was obtained, the system was modified to permit several processes to remain in memory simultaneously, in order to reduce swapping. But this modification did not apply to multiprogramming, because disk I/O was synchronous. This state of affairs persisted into 1972 and the first PDP-11/45 system. True multiprogramming was finally introduced when the system was rewritten in C. Disk I/O for one process could then proceed while another process ran. The basic structure of process management in UNIX has not changed since that time.
It’s been a while since the last post in the series, the details of what was covered in these posts was the partial basis of my talk at BSDCan and I got to repeat the talk again in Berlin, I was much less nervous the second time, not having a fire alarm going off during the talk may have helped. I will cover briefly some things that were mentioned in the talks which I hadn’t written up here, for the sake of completeness.
Thanks to the DragonFlyBSD folks, I have access to a build server for doing regular bulkbuilds on. As I’m running these as a unprivileged user, there’s not much parallelism in the package builds, it’s one package at a time. The system aptly named Monster is a 48 Opteron CPU server with a 128GB of RAM so I can at least run with
MAKE_JOBS set to 96. At the start of the bulkbuilds some deadlock issues in DragonFlyBSD were revealed by pkgsrc which Mat Dillon addressed promptly.
On the Bitrig front, I managed to add support for the OS to
lang/python27 which was the package causing the biggest breakage and now in the process of trying to get the support added upstream, there appears to be a bug report from 2013 in the Python bug tracker to add support ubut it was marked as won’t fix, I’m hoping the decision will be changed but will have to wait and see.
With Python 2.7 built successfully it was onto the next set of breakages, gettext!
I had taken a patch from OpenBSD ports for getting
devel/gettext-tools building but was asked to back it out as it was not the correct solution to the problem. I decided to reapply the fix in my build just to progress to the next hurdle. The next major breakage was with
devel/p5-gettext which needed to be told to include
libiconv, I’m now stuck at getting
During this process I found that we were missing some necessary flags for creating shared libraries which were highlighted by clang:
relocation R_X86_64_32S can not be used when making a shared object; recompile with -fPIC
This turned out to be bug in the platform support, the necessary fPIC flags were defined but under a if statement for version of OS running with a.out binaries still.
mk/platform/Bitrig.mk was stripped of anything related to a.out and everything was rebuilt again from scratch.
OpenBSD and Bitrig probably have many more breakages due to the fact that their architecture is detected as amd64 and not under the x86_64 banner by the build system. One example is
x11/libdrm which is set to add
sysutils/libpciaccess as a dependency if the host is a i386 or x86_64.
At present libdrm fails at the configure stage with
checking for PCIACCESS... no
configure: error: Package requirements (pciaccess >= 0.10) were not met:
No package 'pciaccess' found
Consider adjusting the PKG_CONFIG_PATH environment variable if you
installed software in a non-standard prefix.
Alternatively, you may set the environment variables PCIACCESS_CFLAGS
and PCIACCESS_LIBS to avoid the need to call pkg-config.
See the pkg-config man page for more details.
Trying to add OpenBSD to the x86_64 arch list revealed a problem in pkgsrc, the culprit being
The problem is that there are three separate points where the architecture is defined. In the bootstrap script, in BSDMake’s on source and the settings it passes onto
pkgtools/pkg_install. Unfortunately the settings defined at the start in bootstrap are ignored at the bootstrap stage & are not necessarily what pkg_install is built with. To add to this, it’s possible that BSDMake may need to work out what the system is for itself rather than to be expected to have settings passed to itself. That is they should build with settings passed down in succession or independently.
With severe bludgeoning of code between
pkgtools/pkg_install, I managed to get it to
pkg_add: OpenBSD/x86_64 5.7 (pkg) vs. OpenBSD/amd64 5.7 (this host)
pkg_install performs a check of the OS it’s running on against the settings it was built with (the settings bmake passed it during bootstrap), removing the check revealed there was nothing else preventing things from working but the check needs to be there.
For OmniOS, a major components components in the OS which caused many packages to break was the bundled
gettext, failing during builds as it could not find the
libgomp from (the also bundled) GCC. As a temporary work around to see how the build would progress if
libgomp could be found, I added the lib directory to the search path of
Configuration file [version 4]: /var/ld/ld.config
Platform: 32-bit LSB 80386
Default Library Path (ELF): /lib:/usr/lib:/opt/gcc-4.8.1/lib
Trusted Directories (ELF): /lib/secure:/usr/lib/secure (system default)
crle -c /var/ld/ld.config -l /lib:/usr/lib:/opt/gcc-4.8.1/lib
It was possible to build 13398 packages out of 16536 possible packages with this workaround in place.
With the help of Joerg Sonnenberger, at pkgsrcCon I added support for fetching the OS version info in OmniOS & SmartOS for use in build build reports, this should mean that these operating systems will be reported correctly rather than as SunOS 5.11.
sevan.mit.edu is back online as a G4 Mac Mini with 128GB SSD. It’s yet to complete its first bulkbuild since the rebuild but it’s nearly finished as I type this.
It’s now possible to build more than 14100 packages on FreeBSD 10.1-RELEASE with pkgsrc.
I gave a talk titled “Adventures in building open source software” where I covered the last year of playing with CoovaChilli and pkgsrc. This was my first time speaking at a conference, sadly the recording is in two parts because the fire alarm went off and we had to evacuate the building.
Following on from last weeks post, I forgot to mention building on OpenBSD/sparc64 via a LDOM running on a Sun T5210, this was even more painful than the Solaris counterpart and took the best part of a month, some of this delay was initially caused by problematic packages which held up the build, not parallelising the builds and again issues with FTP mirrors.
devel/electric-fence was another of packages which was responsible for holding up the build that I didn’t mention in the previous post. During the build it runs a binary called
eftest and that’s it, it’s stuck there until killed.
The LDOM I was running in was allocated 4 vCPUs but the build was running as a single threaded build. Defining
pkg/etc/mk.conf and recompressing the bootstrap kit (
bootstrap.tar.gz) helped this situation. To work around FTP issues, bulkbuilds were switched to HTTP only thanks to a pointer from Joerg Sonnenberger. As defined in
# Same as MASTER_SORT, but takes a regular expression for more
# flexibility in matching. Regexps defined here have higher priority
# than MASTER_SORT. This example would prefer ftp transfers over
# anything else.
# Possible: Regexps as in awk(1)
# Default: none
MASTER_SORT_REGEX= http://.*/ in
pkg/etc/mk.conf and recompressing the bootstrap kit ensured builds use HTTP from there on.
The bulkbuild report showed lots of fallout from packages which hadn’t been updated yet to support LibreSSL e.g.
net/wget expects DES support.
lang/python27 with the changes due for subsequent releases of Python.
Bernard Spil fixed Heimdal which failed due to the lack of RAND_EGD in LibreSSL, these fixes will be in the next release of Heimdal (1.6.0?), back porting the changes to 1.5.3 which is the current release available resolved the issue with lack of RAND_EGD but then failed at building a kerberised telnet due to changes in the OpenBSD IPv6 stack which removed functionality telnet was expecting to be there. There is no fix for the issue in the OpenBSD ports as Heimdal is set to build without legacy and insecure protocols such as telnet and rsh.
Due to the connectivity issues on the OpenCSW build cluster, I erased the error report and restarted the bulkbuild on Solaris 10 SPARC and 11 x86 to re-attempt everything that had failed during the previous run for whichever reason. The Solaris 10 SPARC bulkbuild has now finished with a total of 7389 packages built, previously 5701. I discovered a particularly nasty bug with
lang/gcc3-c++ which cost 3 days as the configure stage ran over and over again before being killed manually.
My access to the AIX LPAR expired, taking with it what I had previously tried, I requested access again but this time also requested access to a LPAR running SUSE 12 on Power8 as well.
Still no further with the AIX LPAR but managed to getting bulkbuild going on the SUSE 12 one with a little bit of assistance.
The first thing which needed to be done was to specify the ABI and the suffix applied to the library search path. This is because the system is 64 bit without any 32bit libraries installed and by default pkgsrc opts for 32bit unless set otherwise. When attempting to bootstrap initially, it failed with
ERROR: bin/digest: missing library: libc.so.6. I initially set about the wrong path of trying to locate the glibc-32bit rpm for SUSE on Power before realising what was actually required. This may have been a knee-jerk reaction from the past before the days of
yum and such on Linux. With the necessary change to
pkgsrc/mk/platform/Linux.mk the bulkbuild environment setup continued before hanging on the installation of
pkg_add would hang and CPU utilisation would spike to 100%.
A backtrace of the running process in gdb revealed it was stuck on
#0 0x0000000010096650 in mpool_get ()
#1 0x0000000010093658 in __bt_search ()
#2 0x000000001009318c in __bt_put ()
#3 0x000000001000b614 in pkgdb_store ()
#4 0x000000001000430c in extract_files ()
#5 0x0000000010006fd0 in pkg_do ()
#6 0x00000000100075a4 in pkg_perform ()
#7 0x0000000010005650 in main ()
Turns out the issue also affects pkgsrc on Linux/ARM and was previously reported in a bug report from 2013 with a workaround. Setting the GCC optimisation level to 0 for
mk/pbulk/pbulk.sh to setup a buklbuild environment and a bulkbuild is currently in progress. The bulkbuild was initially aborted to added some critical missing components which caused major breakage.
zypper install libxshmfence-devel gettext-tools gcc-c++.
With Suse Linux on Power8, that bumps my operating system count to 9 across 5 architectures. Just need to get AIX going to round off the OS count.
The past few weeks have been pretty hectic, as the time for BSDcan gets shorter and shorter, I’m thinking about my talk and testing more and more in pkgsrc. Rodent@ added support for Bitrig to pkgsrc-current last month, his patches highlighted an issue with the autoconf scripts (which should be shared across core components) not being pulled in automatically. Joerg Sonnenberger resolved this issue and I regenerated the patch set again. With the system bootstrapped the next thing which was broken was Perl, applying the changes needed for OpenBSD resolved any remaining issues and the bulk build environment was ready. After three days, the first bulkbuild attempt on Bitrig was complete and a report was published. There is now a bulkbuild in progress with
archivers/unzip fixed, that should free over 8400 packages to be attempted to be built.
For Solaris, my first bulkbuild on Solaris 10 completed after 22 days. Mid-April I also started off bulkbuilds on Solaris 11 (x86 and SPARC) using the SunStudio compilers (It’s not possible to use GCC at the moment due to removed functionality that was previously deprecated). The Solaris 11 SPARC bulkbuild is still in progress and the x86 bulkbuild is running. Unfortunately the build cluster had some connectivity issues and needed rebooting during the bulkbuild but not until lots of packages had failed to fetch distfiles, hence the figures look a lot worse than they could be. Solaris 10 SPARC report, Solaris 11 x86 report.
Through bulk building on multiple operating systems another issue that’s surfaced is problematic packages that hold the build up. On Bitrig
mail/fml4 is an issue, on OpenBSD
www/wml, FTP mirror issues for ruby extension on Solaris, Xorg FTP mirror issues on OmniOS. Things need regular kicking, a brief glance into
pkgsrc/mk didn’t reveal any knobs which would allow the preference of HTTP for fetching distfiles. On Bitrig & OpenBSD I’ve excluded these packages from being attempted via
NOT_FOR_PLATFORM statement in their
Makefile until I have a look into the issue.
sevan.mit.edu completed another bulkbuild, pkgsrc-current now ships with MesaLib 10.5.3 as
graphics/MesaLib, version 7 has now been re-imported as
graphics/MesaLib7 by tnn@, the new MesaLib needed a patch for FreeBSD, similar to NetBSD to build successfully, due to
ERESTART not being defined. At present, it’s still broken on Tiger as I’ve not looked into yet.
I revisited AIX again to test out pkgsrc once again, this has turned into a massive yak shaving session. I’ve yet to run a bulkbuild successfully as the scan stage ends with a coredump.
I originally started off with using the stock system shell, bootstrap completed successfully but scan stage of a bulkbuild would just stop without anything being logged. Manually changing the shell used to
pbulk/etc/mk.conf resulted in the following error message:
bmake: don't know how to make pbulk-index. Stop
pbulk-scan: realloc failed:
This turned to be a lack of RAM, my shell account was to a AIX 7.1 LPAR running on a Power8 host with 2 CPUs and 2GB of RAM committed, unfortunately the OS image IBM provided came with Tivoli support enabled and a bug in the resource management controller which meant RMC was consuming way more resource than it needed to. I was running with less than 128MB of RAM.
Stopping Tivoli & RMC freed up about 500MB of RAM, attempting to bulkbuild again, caused the process to fail once again at the same stage. With a heads up from David Brownlee & Joerg Sonnenberger, I bumped the memory and data area resource limits to 256MB.
This allowed the scan to finish with a segfault.
/usr/pkgsrc/pbulk/libexec/pbulk/scan: 11272416 Segmentation fault(coredump).
pscan.stderr logged multiple instances of
bmake: don't know how to make pbulk-index. Stop.
The segfault generated a coredump but it turned out that
dbx, the debugger in AIX was not installed. IBMPDP on twitter helped by pointing to the path where some components are available for installation, unfortunately, while the dbx package was available there, some of its dependencies were not. Waiting on IBMPDP to get back to me, I fetched a new pkgsrc-current snapshot (I couldn’t update via CVS because it wouldn’t build) and re-setup my pbulk environment via
I should mention that initially when I setup, I’d explicitly set
CC=/usr/bin/gcc last time, then while trying to get various things to build subsequently, I’d symlink
/usr/bin/gcc. When I came to set thing up with the new snapshot, I did not pass
CC=/usr/bin/gcc this time round and found that I was unable to link Perl, not sure if this was the Perl build files assuming if on AIX &
/usr/bin/cc exists, it’s XLC or if
ld(1) takes on different behaviour but I had to remove this symlink.
Once everything was setup, the bulkbuild failed agin at the same place, except this time I had a different message logged.
/bin/sh: There is no process to read data written to a pipe..
I edited the
bootstrap/bootstrap script &
devel/bmake/Makefile to set
shells/pdksh as a dependency & rerun bulkbuild.
The scan stage again completed with a coredump with this time
pscan.stderr just contained
Memory fault (core dumped).
I’ve committed these changes so pkgsrc-current now defaults to using
shells/pdksh as its shell but have not been able to try anything else as this weekend the system is unaccessible due to maintenance.
At present, I’m attempting to bulkbuild pkgsrc-current on 8 Operating systems
OpenBSD (5.6-RELEASE & -current), FreeBSD, Bitrig (current), Mac OS X (Tiger), Solaris (10 & 11), OmniOS on 4 architectures (i386, AMD64, SPARC, PowerPC).
If I could get AIX going that would bump the OS & arch could up by 1. Maybe by the next post perhaps.
Thanks to Patrick Wildt for access to host running Bitrig and Rodent@ for adding support to pkgsrc.
Yesterday I gave a talk at SANE user group on my history with wireless networks as part of the PierToPier.net project in Brighton (now defunct) and my experimentation with captive portal software which I began revisiting this time last year. I thought it would be a good opportunity to develop my programming skills by tidying up and modernising parts of the codebase which caused problems, such as things preventing builds on a modern system with clang which is now the default compiler for FreeBSD on i386/AMD64 architectures and OS X. The slides from my talk can be found here.
Back in the early to mid 2000’s there were 2 initiatives to provide public access wifi in Brighton. Loose Connection and PierToPier.net, each had a different focus & approach.
Loose Connection was a commercial venture which could be deemed a VAR, they resold a ADSL connection along with a draytek router & that was it. Individual wireless networks with the loose connection SSID dotted around drinking holes in Brighton, the founding(?) company Metranet lives on as a WISP today.
PierToPier.net was a community driven effort with a technical team of volunteers, predominantly from a service provider / telecoms / networking background. Each node on the network was sponsored by a host who’d buy and run the equipment while the project members managed it.
The project started off based around the fanless VIA mini-itx boards, Prism2 chipset wireless cards, booting linux with hostapd and nocatauth/nocatsplash off a CF card in a IDE to CF adapter.
This was a very flexible platform, if there was no package for it you could build it with ease, problem was that it had a high cost for entry, £250 to £300? so from the start we were looking to reduce costs.
The other issue was though we’d eliminated moving parts, the casing was not suitable for outdoor use.
With the availability of 3rd party firmware and promotional sales of the WRT54G, PierToPier switched hardware platforms as the low cost solution for new nodes.
Tom Grifiths discovered Chillispot around the same time frame and we adopted it due to enhanced functionality it provided, such as RADIUS accounting and working captive portal. We’d previously ran into issues with browser support running nocatauth which by that point was no longer maintained and stability issues with nocatsplash.
VIA motherboard again this time with CF adapter & mini-pci slot onboard
2x Atheros A/B/G mini-pci cards, one on mini-pci to PCI bridges and second onboard
Stuck to a pelican case with epoxy
Two holes drilled in the side of the case for external antennas
It was early times for support of the cards and the wireless standards. In this era OpenBSD was leading the way in terms of support of hardware and development of their
ieee80211 wifi stack, they were the first to reverse engineer the Atheros binary blob HAL (years before anyone?) but late to the game for the 802.11g, 11a was enabled from the start but didn’t appear to work – bringing the interface up in .11a hostap mode wouldn’t necessarily work.
Looking at alternatives there was a short lived live environment named WifiBSD which was based around FreeBSD but later moved to NetBSD before development ceased. The support for the Atheros cards was not as good as OpenBSD, hence not wasn’t much use.
The hardware for the Glastonbury nodes were truly terrible, all functionality had been wired to a single bus which caused the system to lock hard in most configurations. e.g. you booted from a CF card and tried to bring up a wireless interface. The only way to use the system was to disable everything that wasn’t needed in the BIOS including VGA, if there was an issue, you’d have to factory reset the BIOS before diagnosing.
For the wireless network at Glastonbury, the 11a 5GHz network was used as the backhaul while the 11b/g interface was used for connecting wireless clients. No captive portal, connect to the AP & off you go.
We arrived onsite on Tuesday, starting getting things running, Thursday morning the rain and lightning started things went downhill from there. loss of connectivity between the backhaul links meant things fell apart.
By the time we’d discovered Chillipot the project had a 1.0 release out which had preliminary support for FreeBSD. The website claimed only FreeBSD 5 and up were supported, I created a port and submitted it for inclusion in the tree, net-mgmt/chillispot was born, Edwin@ from the ports team fixed the code so that it’d work on previous releases. I then moved onto creating a OpenBSD port, this was slightly harder and the final peace was actually resolved by a Steve Davies. I got the code to build on OpenBSD but networking wouldn’t work. This turned out to be because an additional 4 bytes needed to be allocated which Steve fixed. It never made it into the OpenBSD tree (only tested it on i386 and SPARC, it used strcpy() everywhere and didn’t run on SPARC) but it can be found in ports-wip. I then moved onto creating a live CD environment based on FreeBSD 6 using freesbie for advocacy purposes named BrightonChilli. The idea was to remove the hurdle of going through the installation process and provided an environment that just needed the configuration of network interfaces and chillispot. A person with previous experience of running chillispot would be familiar and a new user would not be too out of place.
This was in the days of X configuration being a part of sysinstall which could hang if Xconfigure was run and you’d have to start the install process again as the install was incomplete (for a newcomer). I was interviewed on BSDtalk #73 regarding BrightonChilli.
PierToPier also produced its own Linux image named Muddy Linux targeted for x86 hardware that ran the necessary stack to serve as a node on the network.
After Chillispot 1.1.0, the project went quiet, there was no answer from the founding developer for quite a while and eventually the web hosting stopped and the domain expired.
The community rehomed to coova.org and development continued in Coovachilli which was founded by David Bird, a contributor to Chillispot.
Coovachilli initially lacked support for FreeBSD but it was eventually added in by David and net-mgmt/coovachilli was born in ports. Not much else was done after that until a year ago. With FreeBSD 10 and the switch to clang, the codebase needed attention, first step was to get it to build correctly with GCC. The use of
error_t from glibc caused the build to fail as it’s not available in FreeBSD, ensuring this was declared allowed the build to complete successfully. To resolve build issues with clang, nested functions were separated out. Any function with missing prototypes & parameter lists were addressed next.
struct ifreq had been marked as deprecated since 2000 and was finally removed in FreeBSD 10. The *BSD specific sections of Coova were switched out to the new
struct ifaliasreq & Linux was left to use the pre-existing method. There was extensive use of macros for the logging functionality, these were dropped in favour of using the existing standard
syslog(3) with the correct log level defined. This had the benefit of revealing issues which were not detected previously such as incorrect format specifiers.
There are still many things that need to be cleared up, the 3rd party functions added in are particularly problematic and will probably be my next task to replace with standard components.
CoovaChilli & Chillisport have seen large scale deployments thanks to use by Fon, o2 and Google which now owns Coova.
Each of the major BSD’s have a different way of handling keyboard layouts on the console & X11. On OpenBSD X11 inherits the setting from wscons by default, on FreeBSD the console keyboard config is separate to the X11 config & depending on if you go down the hald route or not, you may find yourself writing XML to configure your keyboard. For NetBSD which I’ll cover here, wscons configuration is again separate from X11 configuration but everything is configured as per usual via the
xorg.conf keyboard layout.
The snippet below is from
xorg.conf which sets the keyboard model as a ThinkPad T60 (it should apply to X60 series apart from issues with media buttons), US Dvorak layout with the crtl & caps locks switched.
Option "XkbRules" "xorg"
Option "XkbModel" "thinkpad60"
Option "XkbLayout" "us"
Option "XkbVariant" "dvorak"
Option "XkbOptions" "ctrl:nocaps"
I didn’t know about the
ctrl:nocaps option and I happen to stumble across it in the X section of the NetBSD guide.
To apply the same layout to the console, edit
/etc/wscons.conf and set
I can't believe your Caps Lock key hasn't been remapped yet.
— Hipster Hacker (@hipsterhacker) February 24, 2013
Not sure how hipster this all is, managed to get sidetracked into NetBSD desktop config as I was working on updating a package in pkgsrc and remembered the tweet above. Seems like a common thing in the emacs world.
I recently gave a brief talk titled “cross-platform packaging, from Oracle Solaris to Oracle Linux & many more Operating systems using pkgsrc” at Solaris SIG (formerly London OpenSolaris User Group), where I tried to describe the benefits of pkgsrc for a system administrator. This was the second talk on the subject of software packaging at these meetings, the first was by Mandy Waite back in 2009 on the OpenSolaris SourceJuicer which used the spec files usually offered for generating rpm packages. There were no slides and I was quite nervous so I thought I’d write this post to clarify things.
pkgsrc is a portable framework for building software with more than 12000 packages available (with variants of packages, more than 15000) on 22 different platforms (see platform notes for level of support).
pkgsrc is inspired from FreeBSD ports. As with ports, each unique package is assigned a sub-directory in a parent directory named after the relevant category. This directory contains a minimum of a Makefile, a file containing cryptographic checksums of the source files, a file containing a description and a packing list which contains a list of files which will be installed. Sometimes there may be additional files which need to be installed or patches which need to be included as well. Though patches exist in the tree, we strive to upstream changes as much as possible to a) be good netizens b) make maintenance easier going forward.
Packages which are available to be built from pkgsrc generally do not have changes outside what’s needed to integrate with the framework or build correctly on various systems where additional flags need to be specified. There is a mechanism in place for packages to offer build time options to enable or disable functionality.
We try to use components offered by host operating system if viable but offer the facility to rebuild against versions available from pkgsrc.
If you’re mandated to use specific packages for system wide use but require tools for use personally, there is a unprivileged mode where packages are installed within
~/pkg (or another prefix of your choice) and the OS is left untouched.
There is a dedicated security team responsible for keeping a vulnerabilities file up to date so that packages installed from the tree that are affected will be flagged. There is also support for signed packages which utilises gpg for signing.
To start using pkgsrc you either need to run the bootstrap script in
pkgsrc/bootstrap (fetch & uncompress pkgrc.tar.gz) or obtain a prebuilt bootstrap kit for your OS from pkgsrc.org. If you bootstrap manually using the script you’ll end up with your own copy of the bootstrap kit which you can distribute amongst other systems running the same operating system. With a system bootstrapped, you have the necessary tools to build software from the pkgsrc tree or fetch and install prebuilt binaries.
Assuming that your bootstrap kit is
/usr/pkg & you uncompressed your pkgsrc archive to
/usr/pkgsrc and you wanted to install nginx
You can setup a build environment using the
pkgsrc/mk/pbulk/pbulk.sh script which will allow you to build packages in bulk or individually for the purpose of distribution across multiple system.
Or should that be a month of pkgsrc!
Since I wrote my last post I received the good news that I’ll be presenting at BSDCan this year in Ottawa. I’m really stoked about this and looking forward to presenting there. In preparation for the big day, I’ll be speaking about different aspects of what I’ve been working on at various events/meetings. My first speaking opportunities will be at Oracle’s SolarisSIG (LOSUG in a past life) tomorrow on cross platform packaging with pkgsrc.
I didn’t have any hosts running Solaris to attempt bulkbuilds on for this, so I asked for shell access at various places, this landed me with access to Solaris 9 to 11 on x86 & SPARC and OmniOS.
Shell access to a Solaris 10u9 host came first and I quickly found that mandoc would not build due to the use of
vasprintf() whilst attempting to setup a bulkbuild environment using the
mk/pbulk/pbulk.sh script in pkgsrc.
Following up with the mandoc developers regarding the issue meant the issue was fixed by Ingo Schwarze within a couple of days with access to Solaris via OpenCSW, the next release of mandoc due Summer time will contain these fixes. I’m undecided whether to back port the fixes to the current release or not. If you have a Solaris 10 host with a “recent” patch cluster (unsure how recent) you should be unaffected by this issue.
On OmniOS, the
shells/standalone-tcsh package reared its head again. OmniOS ships with tcsh as standard which
shells/standalone-tcsh clobbers again during bulkbuild. The solution to this may be that pkgsrc becomes aware of the various Illumos distros instead of treating everything as SunOS, but nothing has been decided for definite and needs further discussion.
The bulkbuilds on OmniOS using the toolchain published in IPS for that release show issues in pkgsrc where libgomp bundled with GCC is not being found by gettext, this causes lots of packages to fail.
The first bulkbuild which completed showed it was possible to build 9547 out 15245 packages without issue but the inability to build
databases/shared-mime-info due to the libgomp issue caused 2274 packages to fail. This was the large impact of the issue mentioned but there are more packages which suffer from the same problem.
With the resumption of bulk building pkgsrc on FreeBSD, I found lots of packages which would fail to package due to use of the
-o flag with
pax(1) which is not present in the FreeBSD version. I’ve not yet addresses the issue with those packages but I did raise a patch for FreeBSD-current so that it will be available in future versions of FreeBSD. It is now committed and will treacle down to 10-STABLE at some point for inclusion in FreeBSD 10.2-RELEASE.
Bulk builds have also resumed on OpenBSD/amd64 and Sparc64, there is a third bulkbuild in progress on amd64 while the sparc64 host is still running the first attempt. The use of unprivileged user account and the lack of nullfs support prevents the setup of parallel runs to speed up the process. Building on OpenBSD was extremely useful for indicating what software in pkgsrc is effected by LibreSSL without having to put effort into integrating LibreSSL into pkgsrc first. It also raised an instance of copying overlapping buffers destructively (using
memmove()), a fix for this issue is yet to be committed pending further investigation.
MAKE_JOBS was raised to 8 on OpenBSD/sparc64 to improve performance whilst compiling, the build environment is hosted on a Sun T5210 with a dedicated disk mounted with soft writes & the
kern.bufcachepercent value bumped from 20 to 50.
With the pkgsrc-2015Q1 release announced, the figures for packages indicate that Darwin/PowerPC exceeded Darwin/x86.
11224 binary packages built with gcc for Darwin 8.11.0/powerpc
10019 binary packages built with gcc for Darwin 10.8.0/i386
My first bulkbuild report sent to to the pkgsrc-bulk mailing list back in September 2014 shows a total of 8501 packages could be built. There is more work required to keep this position in the future, as a new version of MesaLib is to be committed at some point soon. There is a bulkbuild in progress on sevan.mit.edu, upon completion I’ll be placing a request for the host to be wiped and setup again, the bulk builds are getting slower and slower and I suspect the IDE disk may be failing.
Thanks to Phil Stracchino, Sascha Curth and the OpenCSW Project for access to hosts running Solaris. Ian Kremlin, Brandon Mercer & Rodent @ NetBSD for access to hosts running OpenBSD.
For testing changes related to OS X in pkgsrc I revisited trying to get virtual machines of the various releases of OS X running to improve test coverage. At present I’m confined to testing on Tiger and Mavericks though I also have machines running Leopard and Lion but they need setting up.
By default, it’s not possible to boot an instance of Mac OS X from a genuine install image, on a Mac host, running OS X using virtualbox.
Searching around reveals using modified images intended for building Hackintosh as the solution most people use. Virtualbox supports OS X guests but when following the usual steps in the wizard to create a new VM & pointing it to your unmodified OS image, nothing much happens.
Depending on the version of OS X you’re trying to boot you’ll either end up with a XNU hang/panic or just dropped straight to an EFI prompt.
Again, depending on the version of OS X being attempted the issue differs. I’ve managed to install 10.7 to 10.10 successfully on virtualbox so far. 10.5 & 10.6 remain to be done.
10.7 – Lion
With the release of 10.7, Apple changed the way OS was packaged, the digital distribution came with a disk image named
InstallESD.dmg nested inside an application named
Install Mac OS X Lion.app. It’s possible to use this disk image with virtualbox as-is without change however the system will not boot from the image because it fails a test by the boot loader to ensure the image is being booted on a genuine Mac. In my case it is, but unfortunately the cpuid virtualbox presents to the operating system is not one that the OS recognise & so it fails.
The solution to this is to tell virtualbox to mask the cpuid of the guest, unfortunately depending on the version of hardware? or virtualbox that you’re using you may have to experiment with which ID works. I first tried the ID
00000001 000306a9 00020800 80000201 178bfbff listed in the post by BitTorrent engineering but it did not work on a Mid-2012 MacBookAir5,1 with VirtualBox 4.3.22 r98236.
Searching around I found the ID
1 000206a7 02100800 1fbae3bf bfebfbff to try in a comment on another guide which did work.
To create a working VM of Lion in virtual
1) Create a VM in virtualbox named something, type Mac OS X, version Mac OS X 10.7 Lion (64 bit).
2) Before booting the VM, switch to
Terminal and change the cpuid of the guest by running
VBoxManage modifyvm something --cpuidset 1 000206a7 02100800 1fbae3bf bfebfbff
3) Right click on
Install Mac OS X Lion.app, select “Show Package Contents” and navigate to
InstallESD.dmg to a locate on your disk which is navigatable.
4) Start the VM & when asked for an install disk, point to the
InstallESD.dmg which you copied out in the previous step. The system should boot without any need for further modification (most guides recommend other changes such as switching to a PIIX chipset).
10.8 – Mountain Lion and newer
With Montain Lion,
InstallESD.dmg was changed once again, this time to contain multiple partitions (EFI, Boot/Rescue, Install), unfortunately it’s not possible to boot these images successfully as the notion of multiple partitions is not applicable to media such as optical so what happens is that the system is able to boot from the disk image & load the kernel but unable to continue to load the install environment.
What needs to happen is a new “flattened” image needs to be generated which is on a single partition & contains everything from the boot partition.
There is no need to modify any settings for the VM such as cpuid as previous or chipset as recommended by other guides like Engadgets
The instruction in the Engadget guide pretty much covers everything needed. Just make sure that the disk images are fully detached before issuing the
hdiutil commands, quickest way being to open
Disk Utility.app, selecting mounted disk images & pressing eject or checkout the output of
hdiutil info & using
hdiutil detach $devicename to detach all device names associated with the disk images.
Time again to write up what’s been happening on the pkgsrc front since last time, this time the focus hasn’t been so much around pkgsrc on Darwin/PowerPC but more about pkgsrc in general & not necessarily code related.
At the end of the last post I mentioned a gentleman who’d been working on pkgsrc/Haiku and posting videos of his progress, I managed to make contact with him (James) & discussed his work that he’d been doing on pkgsrc. He sent me copy of the repo he’d be working off so I could assist with the aim of getting everything upstreamed as in the current state everything would need to be reintegrated per quarterly release rather than only having to pay attention if a new issue has arisen.
After getting the correct version of Haiku installed in virtualbox, I discovered a nasty bug in the Haiku network kit, it was unable to detect when the end of the file had been reached & would continue (restart?), this was revealed when I tried to download the pkgsrc tar ball from via WebPositive, ftp from the terminal was not affected however. pkgsrc bootstrapped unprivileged without issue. Hint: use the nightly snapshots until there is a newer release than Alpha 1 available.
The integration of pkgsrc into the user-land on Haiku is not currently possible due to the way the user-land is constructed, from what I understood, each Haiku package contains a piece of the filesystem, all the packages are union mounted to construct the user-land dynamically when the system comes up. That aside, with my system bootstrapped, I attempted to build Perl and ran into another bug, it seems that the library path for libperl is not populated on haiku hence perl is able to “build” but unable to run, the workaround for this in the tree I was given was to symlink libperl into
~/pkg/lib & move on. I tried various things but was unsuccessful, I believe the problem is pkgsrc specific as the version of Perl available in haikuports do not need any special treatment and the
rpath is passed in correctly.
The problem was trying to isolate the required change to fix the problem, whereas in pkgsrc a policy file is passed to the build to set how Perl should be built, haikuports clobbers the source & patches in a replacement, I stopped at that point.
At around about this time I received the good news about the NetBSD Foundation membership and commit bit so my focus moved to reading the various developer documentation & getting familiar with processes.
sevan.mit.edu finished a bulkbulid attempt of the entire tree which took the longest time so far to complete, through all the build attempts I uncovered a new bug, the range to use for numerical IDs of UID/GID, is not sufficient to cover all the packages in the tree that need to create an account. On further discussion with asau@, it was suggested the IDs are allocated randomly and should be fixed for consistency across builds. I started doing bulkbuilds of the entire tree on FreeBSD 10.1-RELEASE and stumbled across a very nasty bug. There is a version of tcsh package in the pkgsrc tree called
shells/standalone-tcsh, this is tcsh built as a static binary & set to install to
/bin (the only package in the pkgsrc tree which violates the rules and places files outside of
$PREFIX by default?), this ended up overwriting the system bundled version of tcsh in FreeBSD & then deleting
/bin/tcsh when the package is removed, this was fixed promptly by dholland@. It was also discovered that Python’s configure script had not been changed when FreeBSD switched to elf binaries and so still trimmed the name of libraries to account for the old linker which could not handle a minor version in a libraries filename (libpython.so.1, not libpython.so.1.0). All versions of Python had been patched in the pkgsrc tree to remove this change so that it used a consistent naming convention across all platforms. After discussing this with bapt@ (FreeBSD) at FOSDEM, it turned out to be bug and should be fixed in future Python versions once the fixes are upstreamed.
The opportunity to join the pkg-security team came up & for the past few weeks I’ve been getting familiar with the processes of dealing with security advisories & listing them so that users who fetch the pkg-vulnerabilities database are notified if they have any vulnerable packages installed. The general advisory process is a little infuriating, based on my recent experience I’d say at the top of my list are the Oracle security advisories as they do not divulge any details other than “unknown” in version(s) X, PHP for the frequency, OpenSSL for the impact. On the one hand I was quite impressed that CVE IDs were becoming so familiar that I could spot, on the fly, an advisory that had been accounted for, but on the other hand quite upset that I was using brain capacity on this. The availability of information is quite frustrating too, issues which are assigned an ID but cannot by checked on Mitre’s site take extra effort to find the necessary information to include (Mitre are responsible for allocating the CVEs!), I should note that this is from public advisories, say from a distribution. Example, CVE-2015-0240 was announced today, the Redhat security team published a blog post covering the issue, the Mitre site at present says:
“** RESERVED ** This candidate has been reserved by an organisation or individual that will use it when announcing a new security problem. When the candidate has been publicised, the details for this candidate will be provided.”
The wording & the lack of information can also be frustrating because it’s not clear what is affected. Looking at it positively, the requirement for clarification on these discrepancies means I get lots of opportunities to approach new people in different communities to ask questions.
I created a new wiki article on the NetBSD wiki to start documenting the bootstrap process of pkgsrc on Solarish, Illumos based distributions. At present the article covers what’s required to bootstrap successfully on OmniOS, Tribblix, OpenIndiana and OpenSXCE.
One thing that’s clearly evident is my workflow needs attention, at the moment things are very clumsy, involving lots of switching around but hopefully that will be addressed in the coming month. The first thing I’ve done is setup templates for emails with the correct preferences specified so that I just need to fill the necessary information & hit send, the necessary settings are automatically applied. Still thinking about how to deal with the scenario where the system that work is being carried out on is different to the system where the patch is going to be committed from, this also happens to be a different system which a developer is using. How to deal with that in as few steps from reading, say a bug report, to generating a patch, testing it & committing a fix.
For testing patches on Mac OS X, I revisited running OS X as guest on a Mac running OS X with virtualbox. Attempts in the past had not been successful & it seemed from search results that the only approach taken was to use modified OS images for hackintosh which I did not want to take. I have a genuine machine & genuine license, I shouldn’t have to resort to 3rd party images to run this. After whinging on twitter & referencing some older links I was able to successfully virtualise Mac OS X 10.7 to 10.10 in virtualbox. Will follow up with the details on that in a separate post.
After many years of trying to get my hands on a NeXT machine, I finally received a turbo color slab as a gift from Joshua Elasser of the OpenBSD project last week. The machine with eventually made its way to my from Portland, OR after a lengthy delay at the customs here. The machine didn’t work out of the box, first I needed to find a sync on green LCD which the Dell UltraSharp 1901FPs could, then I discovered the system was net booting despite having a 4GB ST15150N SCSI disk installed. Following the manual I re-jumpered the disk to set it to delay start which fixed the issue. The system was running OPENSTEP which was configured to be connected to other machines on a private network which caused delays booting as the daemons waited to time out. Following the Apple KB article I entered the boot prom & boot the system in single user mode to reset the root password thanks to a post on the nextcomputers forum . The system has 32MB RAM (expandable to 128MB), according to
/etc/hostconfig is called
murphy and the keyboard is tagged “Fuller Brush CO.”. The Config.app is broken on the system so not sure as to wipe it & restart or just replace the affected binaries. I’d like to get my hands on a daydream rom box which’ll allow me to run classic Mac OS natively on this machine, there are plans on next computers to build one from scratch & progress is being slowly made.
Since the last post I’ve made some further progress with pkgsrc on Darwin/PowerPC again, the biggest achievement was fixing
lang/ruby21-base which accounted for the breakage of some 1500 packages (variation of 500 or so modules for each version of Ruby). This was caused by the failure to build the DBM module, which on OS X required the inclusion of
dbm.h as well as
ndbm.h otherwise all tests fail and the module is not built. The frustrating thing is that there appears to be no documentation for the build process of Ruby, luckily, by Ruby 2.0 there was a comment added to
ext/db/extconf.rb to shed some light on the issue:
# Berkeley DB's ndbm.h (since 1.85 at least) defines DBM_SUFFIX.
# Note that _DB_H_ is not defined on Mac OS X because
# it uses Berkeley DB 1 but ndbm.h doesn't include db.h.
pkg/49508 was committed prior to the 2014Q4 pkgsrc release but
pkg/49511 and pkg/49512 did not.
net/ser include some additional files which weren’t accounted for previously pkg/49473 & pkg/49474 pkg/49476 pkg/49478 pkg/49496 pkg/49498 fixed that.
devel/commit-patch used the
-a flag for
cp(1) which isn’t available on older operating systems, pkg/49475 switched to the use of
-pPR instead (which
-a is an alias of).
graphics/ivtools failed to build successfully due to a packing issue due to the explicit specification of operating system in the name of one of the generated files. pkg/49497 switched the use of the
LOWER_OPSYS & added missing item which addressed the issue.
security/CSP failed at the installation stage due to the target directory not existing, pkg/49499 fixed that.
guid_t but did not include
sys/types.h, pkg/49523 fixed that.
SIOCGIFALIFETIME_IN6 but did not include
netinet6/in6_var.h when building on OS X which broke the build. pkg/49524 fixed that.
failed to build on Tiger due to
sys/types.h, pkg/49526 fixed that.
lang/php55 bundles its own version of sqlite and requires the necessary flags to disable features not available pkg/49527 fixed that but the correct fix is to not build an entire new version solely for PHP’s use. I began to look but had flashbacks of dealing with the same issue in TCL.
graphics/MesaLib I looked to build it using a newer version of binutils but it appears that support for Darwin/OS X/iOS and Mach-o is rudimentary and hence missing support in most of the tools. Support began being added upstream to binutils back in 2011 but is still not complete.
devel/cmake supports the ability to specify the location of library & header files, this can be done by creating a file which includes the necessary declaration that is passed to the configuration process using the
--init flag. Indeed when the configuration process displayed the correct versions of OpenSSL, CURL, Zlib, BZip among others from pkgsrc rather than the older system bundled versions, unfortunately the build still failed when it came to the linking stage as the paths to the libraries was prefixed with
/Developer/SDKs/MacOSX10.4u.sdk, as a kludge just to progress with the builds, I symlinked
/Developer/SDKs/MacOSX10.4u.sdk/usr, the build then succeeded without issue. Next task is to work out how to drop the
/Developer/SDKs/MacOSX10.4u.sdk prefix correctly.
There are currently 11132 packages available on sevan.mit.edu for Mac OS X/PowerPC with a new bulkbuild of the entire tree in progress. There are also Intel builds of the entire tree being attempted by Save OS X (64-bit packages) and Jonathan Perkin (32-bit packages) which should further improve support for OS X in pkgsrc.
Whilst browsing I discovered a series of videos on youtube by DisneyDumbazz, he has also been covering his work on improving support for Haiku in pkgsrc at length.
He was also struggling with issue in Ruby, QT4 & Mesa it seems.
Definitely more than a week, I’ve not had a chance to devote much time to this over the past few months due but have made sufficient progress to qualify another post.
The most import thing is apart from one PR, all previously submitted patches have now been committed to pkgsrc-current, pkg/49082 still remains.
With the introduction of GCC 4.9, the same changes needed to be applied to
lang/gcc49 as with previous versions, pkg/49178 took care of that, however this highlighted another problem. 32bit & 64bit hosts running Darwin both identify themselves as
powerpc in the
uname(1) output which means that GCC is always built with multilib support disabled, even when building on a 64bit host.
Some of the cross compilation tools for micro controllers were hardcoded to use
ksh to build with when in fact it was only required for NetBSD >= 5, this caused the build to break on Tiger (assuming because of the old version of bundled ksh), pkg/49311 fixed that for
cross/binutils but the changes were also applied to
devel/binutils by the maintainer.
cross/avr-libc was previously broken because it was using the system compiler & headers instead of
avr-gcc and the headers installed in pkgsrc during builds. pkg/49316 fixed this issue and upgraded the version to v1.81 of avr-libc.
It’s no longer a requirement to declare the
MACOSX_DEPLOYMENT_TARGET environment variable to build
lang/perl5. By default Perl declares this to be 10.3 which is no longer applicable on modern systems and when building with clang
mmacosx-version-min is specified, making it redundant. This had been removed in pkgsrc via a patch and it broke the build for GCC users as without this variable the target defaults to 10.1 and Perl needs specific attention for versions prior to Panther. pkg/49349 added this variable back in for Darwin 9 and prior which were GCC only releases. Bug #117433 in the Perl RT was the source of the patch proposed to resolve the issue.
lang/ocaml now builds on Tiger, the workaround for the lack of support for
-no_compact_unwind in the shipped linker was applicable to prior releases and not just specifically Leopard, pkg/49417 fixed that.
devel/py-py2app previously failed to build on PowerPC OS X due to an error in the PLIST, the use of the
MACHINE_ARCH variable would expand to
powerpc which raised a packing error. pkg/49418 fixed that.
devel/cmake still remain broken in the pkgsrc tree for Darwin PowerPC, I was able to generate a MesaLib package successfully by forcing static binaries which allowed the previously unattempted packages to be tried in a bulk build of the entire tree. Unfortunately I hadn’t caught a merge conflict from when pkg/49077 was committed and so devel/icu was not built, this caused a another large subset of packages to not be built.
Thanks to the pointer from Jonathan Perkin, after I’d resolved the merge conflict I removed the entry in
/mnt/bulklog/meta/error and ran
bulkbuild-restart to re-attempt building devel/icu & those which depended on it.
With these changes, there were over 10000 packages available on sevan.mit.edu but unfortunately that included lots of duplicate packages from previous bulk builds.
pkgtools/pkglint has the ability to scan packages against a pkgsrc tree & remove duplicate/stale packages. Running
lintpkgsrc -K /mnt/packages -pr took the number of packages down to 9200. There is an AWK based solution but I’ve not had a chance to try it.
I was able to get
devel/cmake to build successfully by removing the references to
Modules/Platform/Darwin.cmake and subsequently build packages such as
databases/mysql56-client but I’ve not added the changes to the tree yet. Will look to add this in a future bulk build, I want to get MesaLib linking correctly first before adding more kludges into the mix. The next thing I want to try is using a newer version of linker from
devel/binutils instead of the one bundled with Xcode.
Since the announcement regarding the X60 series becoming certified by the FSF I’ve been interested in getting my hands on one. Not because of the political statement but I really wanted to play with coreboot on something other than my Alix which at the time previously had ended disasterously.
Luckily my regular used computer hardware pusher told me he had an X60s albeit with a broken screen which I could have. I decided to put off collecting it until I had managed to source a replacement screen cheaply so that there’s one less computer to move around with, come summer time there were “compatible” panels being advertised on ebay for between £40 to £80, I collected the ThinkPad & eventually ordered a panel listed specifically for the model ThinkPad I had (1704-GL5).
Once the panel had showed up, I began stripping the display assembly to swap LCD panels, fairly straight forward to do apart from the bits of tape which stick to the side frame & LCD panel. With the old panel out, I began trying to fit the new panel in the frame, the screw holes lined up though things were slightly off alignment, the cover still fitted so I continued.
I connected the panel to the motherboard via the ribbon & to make sure the display actually worked before tightening the screws I switched the laptop on. The fan on the laptop spunup and the moment when the display is meant to become active the system switched off. Subsequent boot attempts resulted in even short runs before switching off. The panel the seller had sent was not compatible with the ThinkPad & the pinout on the panel was completely different.
I was pointed to the X60 fuse list & advised to check those fuses were ok. Stripping the board & checking the fuses with a multimeter for circuit continuity showed that fuse F5 had blown (responsible for the LCD inverter).
I was unsure of the state of the motherboard at this point and whether there was further damage caused to the board. As I had stripped the system to test the fuses, the CMOS battery had been disconnected which obviously caused BIOS settings to be reset. Not sure of which, but on the first power attempt after a reset the system either opts to run from the first connected display or the VGA out, either way the system completed post with an external monitor connected, pressing the hot key to switch to the LCD panel exhibited the previous behavior and the system switching off promptly.
I obtained a full refund for the LCD display from the seller & the replacement fuses were with me after a couple of weeks of shipping blunders from RS.
I’d never attempted to do surface-mount re-soldering before, attempting to remove & re-solder components on a scrap circuit board didn’t go too well, not sure if it was the soldering iron I was using or the solder used on the components but I didn’t make any component move. Thanks to the kindness of a member of the Hackspace with experience who offered to help with the use of 2 soldering irons the fuse came off & they were able to replace it with a new fuse. The problem was fixed, the system was able to drive the original (broken) panel and complete post successfully.
Returning to search on ebay, I sourced a complete display assembly for £21 which saved the hassle of trying to fit thing back again into the old assembly.
The system was fully functional once again & ready for playing with CoreBoot.
I was recently looking for a link I thought I’d bookmarked on how to install recent versions of Mac OS X on EoL Apple hardware, specifically the Mac Pro. I was unsuccessful in finding the link I was looking for but I did find that you can re-flash a MacPro1,1 with a MacPro2,1 EFI firmware, main benefit being microcode updates. Turns out the hardware in the first & second generation Mac Pro is identical bar the model of CPU available. There’s also modified images to bring the MacPro4,1 to 5,1 which seems to provide much more benefit than the previously mentioned modification.
This got me thinking about some of the issues I’d experienced with older apple hardware and the work arounds, it has been a while since I’ve posted something here so I wrote this post.
On the old world SCSI Macs (pre biege G3?) the drive vendor on the disk firmware with be identified as Apple which the Drive Setup utility (predecessor of Disk Utility) would look for, if it was not found, you would not be able to format your drive as HFS and hence be unable to install Mac OS. Work around was either finding another platform to format the disk or modify a copy of Drive Setup utility with ResEdit & add the drive to the necessary table.
The first of blue & white PowerMac G3 systems logic board shipped with a buggy CMD IDE controller which would corrupt data when doing DMA transfer, Apple shipped the disks in these systems with the firmware tied to PIO mode which was lots of fun when you came to replace the disk with a newer/bigger/faster one. To complete the replacement successfully, the new disk with need to be connected to a PC first & using the firmware utility provided by the vendor, make the same change of restricting the disks operation mode to PIO, otherwise it would not be possible to rely on the disk as data would be corrupted as you began writing to it, there was a recall for the motherboard If you were aware of the issue at the time.
The Mid/Late 2007 MacBook Pro (per advisory?) has the SATA port on the ICH8-M south bridge locked to SATA I even though it is capable of SATA II.
Most systems with user replaceable RAM are capable of taking more than official specification documents list. MacTracker – an application which lists specs & information about Apple hardware provides advertised & actual maximum memory capabilities of system. Not so much a software based restriction but a documentation one.
Shortly after the last blog post I had access to a couple of AIX LPAR. This would be my first time on a IBM PowerPC system and AIX, I’d applied for two AIX 7.1 instances, one defined as “AIX 7.1 Porting Image” and the other as plain “AIX 7.1”. The difference at a glance seemed to be the porting image had more gnu / common open source tools e.g GNU/Tar though both images had a version of GCC installed.
Using built-in specs.
Configured with: ./configure --disable-multilib --with-cpu=powerpc --enable-debug=no --with-debug=no --without-gnu-ld --with-ld=/usr/bin/ld --with-pic --enable-threads=aix --with-mpfr=/opt/freeware/lib --with-gmp=/opt/freeware/lib --with-system-zlib=/opt/freeware/lib --with-mpc=/opt/freeware/lib --with-mpicc=mpcc_r --with-libiconv-prefix=/usr --disable-nls --prefix=/software/gnu_c/bin --enable-languages=c,c++
Thread model: aix
gcc version 4.6.0 (GCC)
The stock version came with GCC 4.2 built on AIX 6.1 whereas the porting image came with GCC 4.6.
Alongside the open source tools each instance also had proprietary tools installed including IBM’s compiler XLC,
cc without any options invokes a man page which describes the different commands that represent a language at a level.
c99 – Invokes the compiler for C source files, with a default language level of STDC99 and specifies compiler option -qansialias (to allow type-based aliasing). Use this invocation for strict conformance to the ISO/IEC 9899:1999 standard..
The pkgsrc bootstrap process didn’t work too well by trying to allow it to workout things out for itself via
cc so opted to use GCC specifically.
pkgsrc happily bootstrapped without privilege and I proceeded to install
shells/pdksh on AIX.
security/openssl comes with 4 different configuration settings for AIX, a pair of settings for the XLC & GCC compilers with a 32bit or 64bit target. It turned out that in pkgsrc it just defaulted to
aix-cc (XLC with a 32bit target), pkg/49131 is now committed so the correct configuration is used, XLC successfully builds OpenSSL with a 32bit or 64bit ABI but GCC is only able to manage a 32bit target.
To switch compiler to
xlc, declare it as the value to
PKGSRC_COMPILER in your
Over the week I attempted to compile components of GCC 4.8 without much success, starting off with
lang/gcc48-cc++ & falling back to
The build process was very unstable, again as with the Tiger/PowerPC, the build would spin off & hang, pegging the CPU until killed. Attempting to restrict the processor time via
ulimit didn’t have much effect.
Alongside trying to get GCC built on AIX, I kicked off building
meta-pkgs/bulk-medium on sevan.mit.edu, the previously reported unfixed components prevented some of the packages from building again (ruby, MesaLib, cmake).
I began looking into fixing
devel/cmake so that it would link against the correct version of curl libraries & use the matching header files,
Modules/FindCURL.cmake in the cmake source references 4 variables which provide some control but I was unsuccessful in being able to pass these to the pkgsrc make process. While trying to resolve this issue I also discovered that on more recent version of Mac OS, the dependencies from pkgsrc ignored, opting for the use of the Apple supplied versions even though the pkgsrc version would be installed.
-- Found ZLIB: /usr/lib/libz.dylib (found version "1.2.5")
-- Found CURL: /usr/lib/libcurl.dylib (found version "7.30.0")
-- Found BZip2: /usr/lib/libbz2.dylib (found version "1.0.6")
-- Looking for BZ2_bzCompressInit in /usr/lib/libbz2.dylib
-- Looking for BZ2_bzCompressInit in /usr/lib/libbz2.dylib - found
-- Found LibArchive: /usr/lib/libarchive.dylib (found version "2.8.4")
-- Found EXPAT: /usr/lib/libexpat.dylib (found version "2.0.1")
-- Looking for wsyncup in /usr/lib/libcurses.dylib
-- Looking for wsyncup in /usr/lib/libcurses.dylib - found
-- Looking for cbreak in /usr/lib/libncurses.dylib
-- Looking for cbreak in /usr/lib/libncurses.dylib - found
mail/mailman had a missing README in PLIST which was handled differently between Tiger & newer releases. pkg/49143 was committed to fix that.