HFS “Incorrect number of thread records” error

Note: this post doesn’t provide a solution to address the “Incorrect number of thread records” file system issue but documents what I went through to see if I could when I faced it. I think your best bet would probably be to try ALSOFT’s Disk Warrior if you need to fix this issue.

I was sitting in front of a Mac Pro which had previously suffered from boot issues due to the expansion boards holding the memory not making contact with the logic board properly. The boot issue manifested itself as the system not POSTing and the fans spinning loudly. Luckily the issue was resolved by reseating things. The system also had a failing Cinema Display which would work fine after a “cold” power cycle having removed the power connector from PSU and reattached, but would fail to wake from sleep if it had been left on. The user was not aware of this issue and assumed whenever the system wouldn’t display anything on boot, it was the POST issue and would power cycle the system, hoping to strike it lucky. Through doing this, the HFS+ journaled file system ended up with problems where fsck_hfs(8) was unable to fix the issue.

Since the filesystem had problems, the system would try to boot, the grey screen with the Apple logo would appear along with a progress bar which would move from left to right and then after some time the system would switch off.

Booting the system in verbose mode by holding command + v showed that the system was running fsck on boot which is what the progress bar was an indicator for normally. It would attempt to repair the file system 3 times, fail, and switch off.

Connecting to another Mac via TDM, it was possible to mount the filesystem as read only and take a closer look. It would have been possible to also boot the system in rescue mode but the Mac I connected to via TDM was running a newer version of macOS and I was hoping that the tooling would be better able to deal with such an issue though fsck_hfs(8) bugs section states “fsck_hfs is not able to fix some inconsistencies that it detects“, regardless.

% sudo fsck_hfs -ypd /dev/disk2s2 
/dev/rdisk2s2: starting
journal_replay(/dev/disk2s2) returned 0
	Using cacheBlockSize=32K cacheTotalBlock=32768 cacheSize=1048576K.
   Executing fsck_hfs (version hfs-522.100.5).
** Checking Journaled HFS Plus volume.
   The volume name is Macintosh HD
** Checking extents overflow file.
** Checking catalog file.
   Incorrect number of thread records
(4, 21962)
	CheckCatalogBTree: fileCount = 421327, fileThread = 421326
** Checking multi-linked files.
   Orphaned open unlinked file temp2639983
   Orphaned open unlinked file temp2651454
   Whole bunch of these temp files trimmed from listing
** Checking catalog hierarchy.
** Checking extended attributes file.
** Checking volume bitmap.
** Checking volume information.
   Verify Status: VIStat = 0x0000, ABTStat = 0x0000 EBTStat = 0x0000
                  CBTStat = 0x0800 CatStat = 0x00000000
** Repairing volume.
 	FixOrphanedFiles: nodeName for id=2671681 do not match
	FixOrphanedFiles: Created thread record for id=2671681 (err=0)
	FixOrphanedFiles: nodeName for id=2671681 do not match
	FixOrphanedFiles: Created thread record for id=2671681 (err=0)
	FixOrphanedFiles: nodeName for id=2671681 do not match
	FixOrphanedFiles: Created thread record for id=2671681 (err=0)
	FixOrphanedFiles: nodeName for id=2671681 do not match
	FixOrphanedFiles: Created thread record for id=2671681 (err=0)
** Rechecking volume.

Repeat again a second time

** Checking Journaled HFS Plus volume.
   The volume name is Macintosh HD
** Checking extents overflow file.
** Checking catalog file.
   Incorrect number of thread records
(4, 21962)
	CheckCatalogBTree: fileCount = 421327, fileThread = 421326
** Checking multi-linked files.
   Orphaned open unlinked file temp2639983
   Orphaned open unlinked file temp2651454
   Whole bunch of these temp files trimmed from listing
** Checking catalog hierarchy.
** Checking extended attributes file.
** Checking volume bitmap.
** Checking volume information.
   Verify Status: VIStat = 0x0000, ABTStat = 0x0000 EBTStat = 0x0000
                  CBTStat = 0x0800 CatStat = 0x00000000
** The volume Macintosh HD could not be repaired after 3 attempts.
	volume type is pure HFS+ 
	primary MDB is at block 0 0x00 
	alternate MDB is at block 0 0x00 
	primary VHB is at block 2 0x02 
	alternate VHB is at block 975093950 0x3a1ec0be 
	sector size = 512 0x200 
	VolumeObject flags = 0x07 
	total sectors for volume = 975093952 0x3a1ec0c0 
	total sectors for embedded volume = 0 0x00 
	CheckHFS returned 8, fsmodified = 1

It was possible to find out the offending files by looking up the inodes listed in the fsck_hfs output. Note the id reported in “FixOrphanedFiles: nodeName for id=2671681 do not match” and use find(1) to look it up

% find /tmp/p -inum 2671681
/tmp/p/private/var/audit/20210215092939.crash_recovery
/tmp/p/private/var/audit/20210215093035.not_terminated

Ironically, the files causing the issue are auditd(8) logs from crash reports?

I thought perhaps turning off journaling would help sidestep the issue by causing fsck to remove the offending files, rather than trying to make use of journal data to replay. hfs.util(8) which is tucked away in /System/Library/Filesystems/hfs.fs/Contents/Resources let’s you do that.

% sudo /System/Library/Filesystems/hfs.fs/Contents/Resources/hfs.util -N /dev/disk2s2 
Turned off the journaling bit for /dev/disk2s2

It didn’t help.

hfs.util‘s supposedly supports a -M (Force Mount) but I was unable to get this to work. I was hoping to force mount the file system read/write & delete the 2 files.

I ended up wiping the disk and reinstalling macOS.

As an aside, the history section of the hfs.util(8) claims it was “Derived from the Openstep Workspace Manager file system utility programs“. The sources for hfs v106 package on Apple’s site shed some more light. Oldest entry in “change history” section of hfsutil_main.c states

13-Jan-1998 jwc 		first cut (derived from old NextStep macfs.util code and cdrom.util code).

Note the description of what main() does in a comment block inside hfsutil_main.c.

Purpose -
This our main entry point to this utility.  We get called by the WorkSpace.  See ParseArgs for detail info on input arguments.
Input -
argc - the number of arguments in argv.
argv - array of arguments.
Output -
returns FSUR_IO_SUCCESS if OK else one of the other FSUR_xyz errors in loadable_fs.h.

There are icons for HFS formatted disks from Rhapsody in the directory: hfs_RHD.fs.tiff, hfs_RHD.openfs.tiff. These live on in hfs v556.60.1 which is the most recent version available on the site as I write this.

Overlapped Extent Allocation Errors (File 123465x) Fix

Due to a lack of disk space I decided to zap my /System & /Library folders while my PowerBook was in Target Disk mode & connected to my G4 PowerMac, only to find out that I couldnt install Tiger when I got the PowerBook home 🙁
Running Disk Utility & trying to repair the disk brought up several Overlapped Extent Allocation Errors (File 123465x)

After laying a few bricks here & there as I hadn’t backed up (surprise! surprise!!!) I googled around & came up with this

As Keoneo mentions in the 3rd tip, if you make a note of the file numbers DiskUtility complains about, fire up terminal.app & run find / -inum 123456 (dont worry about any of the letters at the end of the file number)
You’ll be given the file name of the offending file number, unfortunately you cant actually delete the individual file, you have the delete the folder which the file is in by typing the following command in terminal.app rm -rf /path/to/offending/file/
If you are doing all this from the OS X install Disk or if you have connected the broken disk/System to a healthy Mac add /Volumes/*disk_label*/ to the beginning of the path e.g if your broken disk is called "Macintosh HD" you would issue the follwing command
rm -rf /Volumes/Machintosh\ HD/path/to/offending/file/
note the backslash between the word Macintosh & HD, you need to have that in so the shell understands that whatever you are referring to has a space in otherwise it treats the text either side of the space as two seperate things which wont work!!
Once you have removed the offending files, run repair disk in Disk Utility again & all should be well again!! 😀

Verify and Repair disk "Macintosh HD"
Checking HFS Plus volume.
Checking Extents Overflow file.
Checking Catalog file.
Incorrect number of thread records
Checking Catalog hierarchy.
Checking Extended Attributes file.
Checking volume bitmap.
Volume Bit Map needs minor repair
Checking volume information.
Invalid volume file count
(It should be 29573 instead of 466435)
Invalid volume directory count
(It should be 5662 instead of 80007)
Invalid volume free block count
(It should be 5920335 instead of 1269081)
Volume Header needs minor repair
Repairing volume.
Rechecking volume.
Checking HFS Plus volume.
Checking Extents Overflow file.
Checking Catalog file.
Checking Catalog hierarchy.
Checking Extended Attributes file.
Checking volume bitmap.
Checking volume information.
The volume Macintosh HD was repaired successfully.
Mounting Disk

1 HFS volume checked
Repair attempted on 1 volume
1 HFS volume repaired