My MNT Reform – almost a year on

MNT’s Open Computing Autonomy sticker, CC BY-SA 2.0

I assembled my Reform almost a year ago. This post is about how thing have gone on with the Reform since then.

When I assembled the laptop at the time, I refrained from making any changes and just assembled the system as instructed. The change I had in my mind was to toggle the switch on the CPU module in order to change the boot sequence so that it would be possible to boot from the storage onboard the CPU module (eMMC) as the primary boot device. I wanted to change it so that I could add the boot loader there & setup the root filesystem on the nvme drive with ZFS. Since the switch is tucked under the CPU’s heat sink, I was worried that with the slip of the hand, before I’d event started using the machine, I would have screwed it up and damaged something. So the stock setup it was, and I’ve been with it since.

I bought be MNT Reform kit to assemble with the ath9k wireless card, sleeve, a printed handbook, trackball & trackpad modules. I started off with the trackball installed, intending to switch over to the trackpad after some time. I’ve yet to switch to the trackpad. The buttons on the trackball module give a nice click and it feels nice to use, so I’ve stuck with it.

The wifi connectivity at my place isn’t that great and even with other hardware and operating systems, including mobile phones, throughput is lacking. For the Reform, the antenna struggles with maintaining connectivity, so I worked around it by using ethernet which is fine at home. Outside it’s not such an issue if I’m tethering since the device acting as the access point is usually within very close proximity or where the access point is in the same room as the machine, rather than on opposite sides of a property.

The laptop came with the v3 system image based on Debian sid (unstable train) on a Transend SD card which I’m still using. Performance of the SD card was good and this was my first time running Debian sid. Despite being the unstable train it’s been fairly painless. I vaguely recall a package bug which caused issues with updates in the early days but since then, it’s been fine (colour me impressed). Only thing is the sheer volume of package updates on a weekly basis, give it a week or so and there’s easily a couple of hundred packages to update. I installed ZFS via dkms which I then used to create a zpool on my nvme drive. The nvme drive has a swap partition on it and a zpool with my data; OS currently lives on the supplied SD card along with the root of my home directory.

The system image is Debian sid with a specific kernel and drivers for the Reform, alongside some tools, and customisations to make the system more welcoming such as a login banner which lists useful commands at hand (defined as shell functions) like chat – when called invokes an irc client & joins the mnt-reform channel, or reform-handbook which displays the system handbook in a browser. The system image comes with KDE, Gnome 3, Sway, WindowMaker all pre-installed. I tried Sway briefly since it is a performant window manager but soon rolled back to Gnome 3 due to familiarity & comfort.

Screenshot of a terminal showing the output of the reform-help command

The system image comes with a bunch of apps preinstalled such as KiCad, ScummVM, FreeCAD, Inkscape. It sets a nice tone for the machine as a fun creative space and that’s pretty much how I’ve used the machine in this time. I have kept it separate from my daily environment and used it as a get away from the usual where I want to focus on something.

Of the reform specific packages which are installed, there is the reform-check utility which performs a sanity check on configuration and makes suggestions for new changes which been integrated into newer builds such as missing packages shipped with system image and outdated u-boot, so it is easier to maintain an installed OS image and reduces inconsistencies when it comes to debugging.

If the pace of change of bleeding edge Debian sid is too much, there is now a stable image available currently based on bookworm. There is a path to switch an existing unstable system image to the stable one but since you’re going back in package version it gets a bit messy with old and new version so probably safer to go for a fresh install of the stable image, especially if you’ve been keeping your unstable system image up to date with package updates.

It’s been nice to be able to provide minor fixes and changes to the components which compose the system image. When I first got the laptop, u-boot lacked support for initialising the display, when the system booted, Linux would then initialise the display. As development progressed u-boot grew support to init the display, it even displays the MNT logo! I believe these improvements were a community effort.

Reform sitting at the u-boot boot prompt.

With Debian’s support for running different ABIs it has been possible to run Steam on the Reform. Since you’re running a duplicate userspace, it has made the number of updates balloon but apart from that it just works. The hardware is sufficient to play Thimbleweed Park but the games needing more advanced OpenGL support will run but won’t display so for example Monkey Island 4 runs but there’s just a black window. Since the system is using the open source driver for the Vivante GC7000 GPU, I wondered if it could be made to work using the vendor’s closed binary blob driver since that supposedly has support for newer OpenGL, but I’ve not tried. The swap on nvme was necessary here since Steam will use up all the 4GB RAM and work its way through the swap too (it really needs 8GB RAM) but that’s fine, there’s no noticeable frame rate drop in the high pace action of a point & click adventure. Outside of Steam, the GPU is capable of handling Monkey Island 1 & 2 in ScummVM and the Minecraft clone Minetest which ships with system image works just fine (I’m not really a gamer, so don’t take that list as all the machine is capable of, I haven’t investigated what works and what doesn’t extensively).

I still enjoy looking at the machine just as much as using it. The Japanese keyboard layout with the backlight looks beautiful. The ability to take the machine apart easily has made a world of difference since it’s not a chore to investigate issues or perform maintenance. Need to reflash the keyboard, unscrew 6 screws, and lift the bezel. Need to reflash new firmware on the motherboard, 10 screws, and lift the base panel.

Overhead shot of the MNT Reform showing the keyboard and trackball.

Since I built the laptop, I made two hardware changes.

Originally, the keycaps did not have a notch for the home row index keys, and since the layout it is a little different to usual, it was a bit disorienting switching between machines with different keyboards since muscle memory was lacking. That’s no longer an issue as there are notched keycaps available which I purchased and installed.

The battery board which came with the laptop originally had a couple of issues which were addressed in the updated battery board. With the original battery board, I needed a full battery charge if I was going to compile the ZFS kernel modules otherwise the batteries would not sustain the prolonged surge in use. With the upgraded battery board this is no longer an issue.

I’m still using the battery cells which came with the laptop, the laptop is fast to recharge them. There’s a short delay of around 30 seconds before the system detects the charger is connected and switches over, which has caught me out when I’ve realised at the very last moment that I’m about to run out of power and hurried to connect the charger to the power outlet.

Unfortunately I don’t have numbers on the runtime off batteries as the system gets switched off in between uses, and I’ve not bothered with fighting sleep/resume, the original battery board caused issues with systems sleeping and later, kernel bugs prevented the system from resuming correctly.

Of the hardware features, I’ve yet to use the HDMI port to connect it to an external display. The builtin LCD panel is nice to look at. Since I’m not using full disk encryption on the SD card, I use a Yubikey for SSH keys. The orientation of the USB ports means that the Yubikey touchpad is facing down which is a little annoying to use, but since it flashes when you need to touch, it’s not something that’s going to go forgotten, though it is somewhat clumsy to have to lift the laptop up to touch. I have a small USB hub which I tried connecting the Yubikey to, so that I wouldn’t have to lift the laptop up, didn’t really work, the Yubikey remained facing down. Really need to use a Yubikey nano, rather than the standard one.

The headphone socket is fine for headphones with a cylindrical connector like the Apple earphones, but if your headphone uses another shaped jack like the L shaped, 3 pole one, it wont be able to go fully into the socket. This is due to the socket being positioned ever so slightly back, and with the side panel installed, the panel blocks the L shaped jack from going fully in. That was a trivial fix, use a 15cm 3.5mm 4 pole male to female stereo aux audio extension cable.

3 chords with 3.5mm jacks on top of Reform case.

One completely cosmetic mistake I made when assembling the machine on the first day is sticking the label with the units details on the perspex cover directly above where the CPU heatsink is positioned, that was probably not the best place for it in the long term.

The way the LPC driver for Linux interacts with the kernel currently means that there’s insufficient time for the nvme to complete its own internal shutdown procedure resulting in the unsafe shutdown S.M.A.R.T counter being incremented every time the system is powered down via Linux. Workaround is to issue a reboot instead & switch off the machine at u-boot.

There have been some new additional components added to the MNT Store for the Reform since last year. The most recent addition is the CM4 adapter which allows the compute modules by Raspberry & Banana Pi to be installed in the Reform as a CPU module replacement and games like Monkey Island 4 to be played without issue. When I can afford it, I really want to get the higher end LS1028A CPU module which has 16GB RAM and faster CPU cores. I don’t really strain the machine CPU wise currently, but more RAM is always good. There are now FPGA modules so you can replace the ARM CPU with an FPGA running a softcore which is so exciting and unique, however the modules are more than the price of the laptop itself. I would definitely go for it if I had the cash spare. For now, I’ll have to live with the potential idea of having a laptop form-factor which I could experiment with different CPUs by just synthesizing them.

There’s a new Laird wifi antenna and anti-flexing bars for the keyboard which are going to be my next items to purchase, curious the difference the bars will make.

From the software side I’m actually happy running Debian on this machine as there’s binary packages of anything that I think of wanting to try but I really want to switch to a root on ZFS setup and don’t feel that the DKMS path of rebuilding a module as a separate component works since a kernel upgrade and a failed ZFS module will render a system unbootable (not catastrophic but hassle to recover). I have hit this several times over the last year, for example when the license on the functions in the kernel were changed to GPL only, that broke the ZFS build. Not an immediate failure but something that flags up at the linking stage if I recall correctly which resulted in the module not building during the apt upgrade process. Workaround was to make a local modification to ZFS license ID so it would build successfully and forcing the rebuild of the package again. So I’m dragging my feet, pick up the ball again and integrate ZFS support into Viewpoint Linux and add an aarch64 build or just continue with the convenience of a maintained Debian system. I’d need to refresh everything in Viewpoint Linux and before that, I need to get the framework into place so that it’s easy to repeat the process and maintain it going forward. Ahh!

close up black and white photo of MNT Reform keyboard LCD display showing the MNT Reform logo and above on screen is the Debian logo.

Not having to focus on OS / packaging side of things has definitely been appreciated over the last year. Starting off with an idea of trying something without a lengthy detour into compilation and actually getting on with what I had in mind is great and something I would have to maintain otherwise. Hmm, perhaps this could be a good excuse to buy more hardware for the build infrastructure. There is a new Reform motherboard on the way, if I upgraded the CPU module with the motherboard, I would have spare hardware for the build system.
You have been reading the fantasy of someone with high end CPU taste, and microcontroller money. 🙂

Reform v0.4

Switching from Zevo to OpenZFS on OS X

I recently moved my last Mac from Greenbytes Zevo to OpenZFS on OS X, the reason for both sticking with Zevo & switching to OpenZFS were one and the same, CPU usage.
Prior to the development of OpenZFS on OS X, the two choices for using ZFS on OS X where Zevo or MacZFS, Zevo originally started out as a commercial product but switched to a freebie after Greenbytes picked it up. Zevo had much better integration with OS X e.g disk would be automatically mounted when connected to system just like any other disk with a supported file system and it supported a v28 of the filesystem whereas MacZFS supported a much older version.

When the OpenZFS on OS X development began just over a year ago, I ran the test builds that where made available, though these supported new features through feature flags it was very early days, attempting to scrub a zpool on a i7 MacBook Air with a USB 3 disk would spike the CPU for the duration and again the integration was still missing, you manually had to import & export pools. I continued to try newer builds on my MacBook Air but stuck with Zevo on my 2007 MacBook Pro.

The two things which where annoying about Zevo was that it was a dead end, development had stopped, the last version available wasn’t compatible with Mavericks available and its conservative memory setting meant that disk performance wasn’t that great, during playing audio files it would break to buffer audio in iTunes for example (luckily not in Serato as mid set would’ve been embarrassing).

As the MacBook Pro was running low on disk space I tried to move around 40GB of files in several chunks in parallel to my external USB3 disk & noticed the CPU pegged and fans started up with Zevo too. OpenZFS on OS X is fairly robust now (though still rough around the edges) so I decided to switch over.

The OpenZFS on OS X disk image comes with uninstall scripts for Zevo & though the main script was unable to detect the installed copy of Zevo, I was able to run the subsequent scripts individually to remove Zevo from my system and reboot (eject the disk containing the filesystem beforehand (export the zpool)).

The integration with OS X is still missing though it seems that on boot zpools are imported, I’ve not worked out if that’s because the system caches the state from previous boot or this is the preliminary support for auto mounting???

If you want to eject a disk, you still have to export the pool manually from terminal, pressing the eject button in finder will remove the disk icon but the filesystem is still mounted. That aside, OpenZFS on OS X performed well, scrubbing the zpool on the 2007 MacBook Pro did not cause the CPU to spike at all, there is now a shorter delay in iTunes when starting to play a track but haven’t noticed any drops in audio yet, so things are looking positive.

Scrubbing the zpool on a 2007 17″ MacBook Pro with 4GB RAM

pool: tank
state: ONLINE
scan: scrub in progress since Fri Jul 25 18:58:48 2014
28.2G scanned out of 579G at 30.1M/s, 5h11m to go
0 repaired, 4.87% done
tank ONLINE 0 0 0
disk1s2 ONLINE 0 0 0

errors: No known data errors

All properties on the zpool I was using:
tank type filesystem -
tank creation Mon Jul 29 5:00 2013 -
tank used 579G -
tank available 1.22T -
tank referenced 579G -
tank compressratio 1.00x -
tank mounted yes -
tank quota none default
tank reservation none default
tank recordsize 128K default
tank mountpoint /tank default
tank sharenfs off default
tank checksum on default
tank compression off default
tank atime on default
tank devices on default
tank exec on default
tank setuid on default
tank readonly off default
tank zoned off default
tank snapdir hidden default
tank aclmode discard default
tank aclinherit restricted default
tank canmount on default
tank xattr on default
tank copies 1 default
tank version 5 -
tank utf8only on -
tank normalization formD -
tank casesensitivity sensitive -
tank vscan off default
tank nbmand off default
tank sharesmb off default
tank refquota none default
tank refreservation none default
tank primarycache all default
tank secondarycache all default
tank usedbysnapshots 0 -
tank usedbydataset 579G -
tank usedbychildren 4.48M -
tank usedbyrefreservation 0 -
tank logbias latency default
tank dedup off default
tank mlslabel none default
tank sync standard default
tank refcompressratio 1.00x -
tank written 579G -
tank logicalused 578G -
tank logicalreferenced 578G -
tank snapdev hidden default
tank on default
tank off default

Upgrading the zpool with OpenZFS on OS X
This system supports ZFS pool feature flags.

Successfully upgraded 'tank' from version 28 to feature flags.
Enabled the following features on 'tank':

Zvol backed bhyve guest

Things have moved forward in the world of bhyve since I last looked at it over a year ago, support for zvol backed guests where fixed long ago among other things such as the birth of vmrc by Michael Dexter.
To run a guest with a ZFS zvol as its disk is no different to running with a disk image, only thing is that my version of /usr/share/examples/bhyve/ (11.0-CURRENT r267869 at the time of writing) fails to start from the disk once the OS has been installed.

A typical deployment scenario would be

Create a zvol of some size, e.g. 10GB

zfs create -V 10g zroot/guest0

Start a guest which’ll boot from the FreeBSD install CD iso & install onto the zvol

# sh /usr/share/examples/bhyve/ -c 4 -m 1024M -t tap0 -d /dev/zvol/zroot/guest0 -i -I FreeBSD-10.0-RELEASE-amd64-disc1.iso guest0

Use the “ZFS – Automatic Root-on-ZFS” option from the Partitioning menu

As instructed in the bhyve section of the handbook, before rebooting, drop to the shell & edit /etc/ttys & replace the console line with

console "/usr/libexec/getty std.9600" xterm on secure

Shutdown the operating system
halt -p

Kill the guest
/usr/sbin/bhyvectl --destroy --vm=guest0

Create a new guest
bhyveload -m 4G -d /dev/zvol/zroot/guest0 guest0

Boot the new guest from the zvol
bhyve -c 1 -m 4G -A -H -P -s0:0,hostbridge -s 1:0,virtio-net,tap0 -s 2:0,ahci-hd,/dev/zvol/zroot/guest0 -s 31,lpc -l com1,stdio guest0

These instruction skip the creation of networking which is covered in the FreeBSD handbook as linked to above.

Copyright (c) 1992-2014 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-RELEASE #0 r260789: Thu Jan 16 22:34:59 UTC 2014 amd64
FreeBSD clang version 3.3 (tags/RELEASE_33/final 183502) 20130610
CPU: Intel(R) Core(TM) i7-3770 CPU @ 3.40GHz (3399.54-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x306a9 Family = 0x6 Model = 0x3a Stepping = 9
AMD Features=0x20100800
AMD Features2=0x1
Standard Extended Features=0x200
TSC: P-state invariant
real memory = 5368709120 (5120 MB)
avail memory = 4103143424 (3913 MB)
Event timer "LAPIC" quality 400
ioapic0 irqs 0-23 on motherboard
random: initialized
module_register_init: MOD_LOAD (vesa, 0xffffffff80cfa5e0, 0) error 19
kbd1 at kbdmux0
acpi0: on motherboard
acpi0: Power Button (fixed)
atrtc0: port 0x70-0x71 irq 8 on acpi0
Event timer "HPET" frequency 10000000 Hz quality 550
Event timer "HPET1" frequency 10000000 Hz quality 450
Event timer "HPET2" frequency 10000000 Hz quality 450
Event timer "HPET3" frequency 10000000 Hz quality 450
Event timer "HPET4" frequency 10000000 Hz quality 450
Event timer "HPET5" frequency 10000000 Hz quality 450
Event timer "HPET6" frequency 10000000 Hz quality 450
Event timer "HPET7" frequency 10000000 Hz quality 450
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: port 0x408-0x40b on acpi0
pcib0: port 0xcf8-0xcff on acpi0
pci0: on pcib0
virtio_pci0: port 0x2000-0x201f mem 0xc0000000-0xc0001fff irq 16 at device 1.0 on pci0
vtnet0: on virtio_pci0
virtio_pci0: host features: 0x1018020
virtio_pci0: negotiated features: 0x18020
vtnet0: Ethernet address: 00:a0:98:7f:5a:41
virtio_pci1: port 0x2040-0x207f mem 0xc0002000-0xc0003fff irq 17 at device 2.0 on pci0
vtblk0: on virtio_pci1
virtio_pci1: host features: 0x10000044
virtio_pci1: negotiated features: 0x10000044
vtblk0: 40960MB (83886080 512 byte sectors)
isab0: at device 31.0 on pci0
isa0: on isab0
uart0: port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)
uart1: port 0x2f8-0x2ff irq 3 on acpi0
sc0: at flags 0x100 on isa0
sc0: MDA
vga0: at port 0x3b0-0x3bb iomem 0xb0000-0xb7fff on isa0
atkbdc0: at port 0x60,0x64 on isa0
atkbd0: irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
ppc0: cannot reserve I/O port range
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 10.000 msec
random: unblocking device.
Netvsc initializing... Timecounter "TSC-low" frequency 1699769676 Hz quality 1000
Trying to mount root from zfs:zroot/ROOT/default []...

System fails to boot with root on ZFS

I’d installed FreeBSD on my ThinkPad X61s when the head branch of the source tree was at 9-CURRENT, multi-booting it with Windows 7 & OpenBSD.
At the time I was not aware that it was possible to boot FreeBSD from a root file system on a ZFS volume from a disk partitioned using a MBR scheme. Instead, I opted to store /boot on a UFS filesystem.
This install existed for a couple of years, the ThinkPad got a roasting every once in a while to build a new release to install for updates. At some point support for 4K sectors in ZFS was improved, zpool status began to report degraded performance as the disk had been using 512byte sectors where in fact it could support 4K sized sectors.

Eventually, I deleted the existing slices in the FreeBSD partition & attempted to reinstall but found this time the system would not boot.
Booting from the install CD & issuing zpool import reported the new pool & old pool from the previous install.
Destroying a pool before deleting slices stopped this problem from re-occurring but the system still wouldn’t boot from a ZFS volume on a MBR partition.
The next step was to see if things would work if the whole disk was dedicated to FreeBSD, with a GPT partition scheme, things worked but switching to MBR, again, it failed to boot, hanging at a flashing cursor.
Over the next four months, many installs were attempted. On a MBR partitioned disk
FreeBSD failed to boot but PCBSD could by using GRUB.

I stopped trying any further at this point & took a break from it, one thing that had been raised at BSDCan was the possibility it could be lingering metadata, I’d thought as zpool(8) wasn’t reporting any existing pools when asked to import that this wasn’t the case. To give the benefit of a doubt, I ran dd on the disk with no difference in result.
This approach to clearing old pools seemed a little rough so over the weekend I looked into what other options are available.

The zpool(8) man page documents the labelclear option as

zpool labelclear [-f] device

Removes ZFS label information from the specified device. The device
must not be part of an active pool configuration.

-v Treat exported or foreign devices as inactive.

I still had the FreeBSD snapshot from the last attempt which I booted the X61s with, headed to the shell, deleted the existing partitions & issued
zpool labelclear -f /dev/ada0

Everything worked as intended after that.

Thanks to Allan Jude & everyone who chipped in at BSDCan.
Through the trial of getting this working Allan added the option to use a BSD partition scheme to the FreeBSD installer as well as MBR & GPT, which was previously unavailable.