-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 MDJ 2002.11.14 (November 14, 2002) ================================== Copyright 2002, GCSF Incorporated. All rights reserved. Top of the Day -------------- **The big big story** Where have we been since Sunday (Monday was a US holiday)? Conducting a three-day investigation into journaling, the biggest new feature in Apple's Monday release of Mac OS X 10.2.2. In a bug-fix update, journaling is clearly the highest-priority new toy, and everyone has questions about it since it's barely documented and unsupported on non-server versions of the operating system. Not anymore! MacCyclopedia steps up today with an issue-length look at the technology that developers first saw only three weeks ago, and even then only if they were paying close attention. From a refresher on what a journaled file system does to _exactly_ how Journaled HFS Plus works, including the two new special files and changes to the Volume Header Block, MacCyclopedia has the information. We put disk utilities through their paces and found what they did to journaled volumes - and why. We spell out the troubleshooting agenda that works with journaling and the one that ignores it, and what specific circumstances could lead journaling to damage your disk (unlike some of our first thoughts, a program doing so would need to violate Apple's HFS Plus specification, at least the later versions). This is why we're MDJ_, as you'll know without a doubt after reading today's issue. **Still to come** A three-day investigation with disk editors is massive, but we still have news, products, and the rest of Mac OS X to document for you - we just took what we knew was the most important part first and did it right. There's a lot more to come in Friday and Saturday issues; we're just getting over the hump now! MacCyclopedia[tm]: Journaled HFS Plus ------------------------------------- **What the new capability means for real people** Among dozens of bug fixes and minor improvements in Monday's release of Mac OS X 10.2.2 [1], none is garnering more attention than the embryonic journaling file system. The user-level release notes [2] on Monday said that the OS update "provides a foundation for the journaling file system (JFS), which may currently be enabled via Disk Utility on Mac OS X Server systems." By Tuesday night, even that note had been removed from the documentation, but not before users around the world drooled at the prospect of more reliable disk systems. [1] [2] _The_Weekly_Attitudinal_ hit the basics of journaled file systems a couple of weeks ago (MDJ_ 2002.10.25), and that background remains sound. The implementation in Mac OS X 10.2.2 is the more conservative version that the Attitudinal posited might actually make it into a future version of Mac OS X: a simple journaling mechanism that makes sure all of the changes that should have been written to disk actually were. It is not Rewind [3] from Power On Software, nor does it "provide corporate Mac sites with a new, historical view of your data." [4] It doesn't provide any view - it merely ensures what might have been cached got written to disk. [3] [4] So now we have a conundrum. Here's a feature that would seem to be of major import that Apple is removing from the release notes, even though it's clearly present in the operating system. Meanwhile, Apple is quietly advertising [5] the feature for Mac OS X Server, calling it a "robust new journaling feature ... that enhances server availability and fault resilience." Apple says, "Journaling protects the integrity of the file system on Xserve and other Mac OS X Server-based systems in the event of an unplanned shutdown or power failure. It also helps to maximize the uptime of servers and server-attached storage devices by dramatically expediting repairs to the affected volumes when the system restarts." A three-page PDF file [6] explains how to enable journaling on existing or new Mac OS X Server HFS Plus volumes. [5] [6] So should you use it or not? Is it tested? Does it provide any benefits? The MDJ_ Staff spent two days exploring the journaling file system and its changes to HFS Plus. Our research encompassed published changes to Darwin code related to journaling, as well as conversations with technical experts, and examining block-level changes to journaled volumes using tools like Sedit [7], comparing the results of Mac OS X 10.2.2's journaling to the documented HFS Plus file structures [8]. What we found is reassuring at times and completely frustrating at others, pointing to a journaling capability that is probably safe to use and yet without significant merit except in specific cases. It's no wonder Apple is targeting it only at Mac OS X Server owners at present. Read on and judge for yourself. [7] [8] **Starting a journal** The open secret about Mac OS X Server is that it's Mac OS X with extra files. There may be one or two exceptions, but to our knowledge, there's no file in Mac OS X Server that's different in Mac OS X. The server version simply has more programs and more configuration files that the OS recognizes if it finds them. The valuable parts of Mac OS X Server are the programs like Workgroup Admin, Server Admin, the unlimited AppleShare file server, and more. The core parts of both Mac OS X and Mac OS X Server are all but identical. It's therefore no surprise that, like the Apache Web server and the NetBoot distributed information system, the journaling file system from Mac OS X Server is actually part of Mac OS X 10.2.2. It's there, and the word is spreading: by Monday night, more than a dozen people had already submitted the magic incantation "sudo diskutil enableJournal /" to MacOSXHints.com [9] for publication. In Mac OS X Server, you enable journal through Disk Utility, and since diskutil is the command-line equivalent for remote administration, that's where you find it in Mac OS X 10.2.2. The diskutil command takes several verbs, each of which operates on a mounted file system (you can also use it to verify or repair permissions, just as in the Mac OS X 10.2 version of Disk Utility). The magic incantation grants root permissions to diskutil and then enables journaling on the root file system (the startup disk), the one named "/" in Mac OS X. [9] To review, since some people don't expect technical information from the _Attitudinal_: a journal records a list of changes to a disk. For performance reasons, most modern operating systems delay some requests to write to disk, caching the information in RAM (and returning it if you try to read the data back from "disk"), but not actually writing the data to physical media until it's time to write lots of data, or until a file is closed, or other opportune time. If the system crashes (such as through a kernel panic or loss of power) before any cached blocks are written to disk, the physical media may be left in an inconsistent state. It's more than just losing data; the directory structure may be corrupted, prohibiting access to other files that weren't even open when the crash happened. A journaled file system records a copy of these changes to the file system to a separate part of the disk called a _journal_, or in some implementations, records the changes solely to the journal and then commits them to the regular disk structures in the background. If the system unexpectedly halts, the journal contains all of the write commands that programs had issued through the time of the halt, and the data they should have written to disk. Startup code that finds a journaled volume unmounted properly simply _replays_ the journal, making sure all the changes it contains are properly recorded on disk. When that's done, the volume structures are consistent again. The journal is designed solely to protect against improperly unmounted volumes. There are no provisions to manually "roll back" a disk to a previous state, or to replay the journal on command. Apple's implementation of journaling, in fact, does not journal any changes to the _contents_ of files, but only changes to HFS Plus file system structures. Apple's PDF "white paper" on the technology says, "The journaling capability in Mac OS X Server is designed to protect file systems' data structures and facilitate rapid system recovery in the event of a failure. Because journaling does not safeguard the actual contents of files or prevent data loss resulting from a hardware failure, it should be used in conjunction with protected RAID storage and an appropriate backup strategy." Journaling can never make file system operations faster, as, at a minimum, it requires writing file system structures to disk twice (once to the journal and once to their proper location). It only makes them more secure. The only real point is to avoid checking a large volume's file system structures on startup. In Mac OS 9, a small version of Disk First Aid embedded in the System file handles the task if you have it enabled in the General Controls control panel (and you should); in Mac OS X, the system runs the file system checking command 'fsck' as part of the startup process. The larger the volume, the longer it takes to validate each directory and check it for damage. Apple's white paper says, "On a multi-terabyte volume, this process can take hours, resulting in an unacceptable period of server downtime." The company has reason to be concerned. In May, when introducing the Xserve rackmount computer, Apple also announced a product called "Xserve _RAID_" (yes, the italics are part of the product name). The device is a 3U rackmount storage system that uses up to fourteen of the Xserve's hot-swappable drive modules, combining for (at demonstration time) up to a 1.68TB disk volume (MDJ_ 2002.05.17). No matter how carefully you use monitoring software or uninterruptible power supplies, sometimes systems fail, be it through power outages or kernel panics. Maybe once a year, maybe once every two years, but it can happen to any system. Since a server needs to validate a file system before it starts letting thousands of clients read and write files, the file system check is not optional. What's more, the task is more I/O-bound than CPU bound, so faster processors won't really help it much, only faster drives. Alas, Apple has chosen standard ATA/100 drives for its RAID solution, running cheaper, slower drives in parallel for speed instead of relying on single drives that are inherently fast. File system structures must be validated block-by-block, negating much of the parallel advantage, so for repair purposes, the Xserve _RAID_ probably won't be much faster than a single disk. So if it takes a machine about one minute to check 8GB of file system structure (a back-of-the-envelope estimate that seems right to us), it would take about three and a half hours to check a 1.68TB RAID array. If Apple boosts the storage by moving beyond 120GB drives in the removable modules, it could take over four hours to reboot an Xserve with attached Xserve _RAID_ device. That's not really what marketing wants to tell customers. But if that volume is journaled, well, that's a disk of a different stripe. Even if _four_thousand_ file system blocks were unwritten when the system crashed, fixing the volume requires nothing more than writing those four thousand blocks - around 32MB, or just a few seconds' worth of work. Since the journal mechanism insured that all other file system structures were consistent, there's no need for any other validation, and the disk is ready to go. That's eight times worse than reality - the default journal size in Mac OS X 10.2.2 is 4MB, and the chances that any but the very busiest servers could have that many uncommitted journal entries are small indeed. So remember these points in counter to hype you may read. Journaling does not make a system faster, smarter, or more stable. It does not "provide a historical view" of data, or let you look at past versions of files. Mac OS X 10.2.2's journaling doesn't even preserve file contents, only file system structures. Its sole purpose is to eliminate what could be a lengthy file system check when first mounting a volume that was unmounted improperly. If you don't have problems with lengthy file system checks, there's little benefit in Mac OS X 10.2.2 journaling for you. By the way, Apple has yet to name this implementation anything catchy, so to avoid calling it "Mac OS X 10.2.2's journaling implementation" a billion times, MDJ_ hereby dubs it Journaled HFS Plus until a more official name comes along. **How the journal works** The information in this section comes from examination of Journaled HFS Plus volumes as well as from HFS Plus documentation and source code changes made public on the Internet. Specifically, many of the changes to HFS Plus structures are found in the changes [10] to newfs_hfs, the command that creates a new HFS Plus file system on a given device under Mac OS X. Don Brady, a long-time Apple engineer, committed these changes to the Darwin CVS repository on 2002.09.17. Brady, Mark Day and Derek Horn were the principal designers and engineers of HFS Plus back in 1997 (MWJ_ 1998.02.02). They're definitely among the good guys, and any code they write for HFS Plus is authoritative by definition. [10] Brady's changes to newfs_hfs show the differences in creating a plain HFS Plus volume and in creating a Journaled HFS Plus volume, something newfs_hfs now does if passed the "-j" flag. Other changes [11] made by former BeOS File System engineer Dominic Giampolo, who modified the fsck_hfs command to be aware of Journaled HFS Plus volumes and not repair them by default, reinforce Brady's changes. [11] By the way, these documents eviscerate the myth that journaling changes were common knowledge among Darwin developers: as of press time, neither Apple's live Darwin CVS repository [12] nor the OpenDarwin CVS repository [13] had these changes checked in where they're supposed to be. In fact, neither even has open projects for HFS-related tasks in the /src/live hierarchy; they're in the /src/notlive hierarchy. If you just browse the code and look, you'll be told that these files haven't changed in eight months. However, if you disbelieve that and actually look at the files in the Web interface, you'll see they were checked in eight weeks ago. We checked them out via CVS, though, and got the old pre-journaling versions. The CVS metadata just has, as the movie says, a "gift for fiction." Since the changes do not appear to be in either advertised home of current "Darwin" sources, and can only be found (and only on the Web) if you don't believe the version numbers you see, there's little way to rationally argue that all this is "obviously" available or documented. [12] [13] As with other systems that add journaling to a well-known file system, Journaled HFS Plus keeps the journal in a special file in the root directory of the HFS Plus volume. The journal file is named ".journal", and when properly created, has file type 'jrnl' and creator type 'hfs+'. By default, the journal file takes 4MB of disk space, but it must be _contiguous_ disk space. If you open the journal file and look inside, you'd find what looks like a series of directory entries, with file names and file types and other metadata. These are the file system structures that journaling protects. Every time any program changes a file's directory entry, or through normal operations forces HFS Plus to expand or contract a file system structure, the journal records it. The journal begins with a header that points to the first and last valid entries in it. Although we haven't found conclusive proof in the source code, the journal appears to wrap around - when entries reach the end of the 4MB chunk of disk allocated to it, new entries start again at the beginning of the chunk, just after the header. That's one reason the disk space must be contiguous. Journaled HFS Plus uses a second file to hold information about the journal, a structure called the _journal_info_block_. The file is named ".journal_info_block", has the same 'jrnl' file type and 'hfs+' creator type, but is only one allocation block long. The code that makes new file systems reveals that the journal information block contains information about the journal, including the byte-level offset to the journal file, the journal's size, and some flags describing how it works. One flag says the journal needs to be initialized, no surprise for a brand-new file system; another seems to indicate that the journal is placed on the device being journaled. (Other source files imply that the specification allows for journals on devices other than the one being tracked, but that this option is not implemented in Mac OS X 10.2.2.) **Structural integrity** -- You may be wondering why the journal information block points to the journal when it's already in a file. After all, wouldn't it be easier to just open ".journal" than to point to its structures on disk? What if the file moves through optimization, or expands, or something like that? Why bother to update two sets of pointers to the same information? The answer is part of the reason that journaling breaks _some_ existing disk utilities. Remember how the journal reuses its 4MB of disk space, wrapping around instead of expanding if it reaches the end of the chunk? That's a critical part of Journaled HFS Plus's design. Although the journal is technically stored in a file, that's only true to preserve the disk's structure. 4MB of data that lived outside any file would completely confuse today's disk utilities. HFS had some information (like the volume bitmap) that lived outside of any file, even the catalog file, but HFS Plus puts everything in a file, even the list that tracks which blocks on disk are in use and which are free. The journaling code, however, doesn't treat the journal like a file. It uses the 4MB of contiguous disk space as a wrap-around buffer, and it would unacceptably sabotage performance if the code had to seek parts of that buffer all over the disk. For sanity reasons, it's extremely unlikely that journaling uses normal file access to write to the journal file anyway, as that would require opening and seeking a file while in the middle of other file operations. No, this code is part of the HFS Plus implementation itself; it writes directly to the _blocks_ in the journal file. To avoid normal file access, it caches the location of all of its information in its own structures. Journaling keeps to itself. This is rapidly approaching what a technojoker might call a cache-22. Journaling doesn't go through normal file I/O to access its own files, but without going through normal file I/O, how can it find the journal and journal info files? The answer is that it does look at the disk to find the files, but once it finds them, it doesn't look again. Journaled HFS Plus changes the definition of the HFS Plus Volume Header Block [14]. As documented and since inception, the four-byte field at offset 12 in the Volume Header Block has been reserved. Starting with Mac OS X 10.2.2, that field now belongs to journaling. The code to create a new file system clearly shows that Journaled HFS Plus caches the address of the journal info block in this field. That's the same single allocation block found in the ".journal_info_block" file, you'll recall. In turn, the journal info block points to the location of the journal on disk, and the journal's headers point to where the next journal entry goes. [14] HFS Plus keeps the Volume Header Block in RAM at all times because it contains fields that must be updated constantly, including a "write count," a number that's incremented every time anything is written to disk (with the possible new exception of a journal entry). Since journaling is part of the HFS Plus implementation, it has access to HFS Plus's data structures. It can't necessarily assume that the rest of HFS Plus will bother to cache journal information, but as long as it caches the Volume Header Block, journaling has all the information it needs to find all its own structures on disk. The Volume Header Block points to the journal info block, which in turn points to the journal itself. Now there's just one more problem. Suppose you get a little nasty - like we did - and run Speed Disk or another disk optimizer on a journaled volume after unmounting it and remounting it in Mac OS 9. Chances are very, very good that any optimizing will move the ".journal" and ".journal_info_block" on disk. That's not the end of the world, but it means the value in the Volume Header Block no longer points to the journal info block on disk. The journaling code can check for this - if it knows that it should. When the code sees that the formerly-reserved field in the volume header block no longer points to a valid journal info block (by comparing it to the sole block in the ".journal_info_block" file), what conclusion does it draw? It could mean that journaling has been disabled but the files are still around, or it could mean that journaling is enabled but the files have moved. Should it reinitialize journaling with the new location of the files, or should it shut itself off? Journaling doesn't have to guess, thanks to one more change to the Volume Header Block. The four-byte field starting at byte four of the volume header block (the first byte is always byte zero) is defined as a series of flags describing attributes [15] of the HFS Plus volume, including the flags that say a volume was unmounted improperly (so the OS knows it needs a file system check before mounting it again). Bit 13 of the attributes was previously reserved, but now it is set when journaling is enabled. That allows Journaled HFS Plus to know if journaling is supposed to be on and repair any damaged journaling structures if necessary, instead of wondering if you meant to stop journaling altogether. [15] **Simplicity** -- That's pretty much all there is to Journaled HFS Plus. This isn't enough detail for disk utility developers, of course. Those folks need to know the exact format of the journal file and the journal info block. We've confirmed much of how it works now, but that's not a guarantee it will continue to work that way. Only documentation provides that assurance, and while we expect Apple to update the HFS Plus volume format documentation soon, it hasn't happened yet. If journaling looks kind of glued onto HFS Plus, that's because it is. Other than the changes to the Volume Header Block, the HFS Plus volume format changes none to support this simple journaling mechanism. By performing its own block level access, it avoids huge performance penalties by making every HFS call into two calls (one to write information and one to write the journal). By restricting its scope to directory structures and not the contents of data files, Journaled HFS Plus protects against damaged disks while not adding huge performance penalties for tracking every bit changed in every file (just journaling "console.log" alone in Mac OS X could at times make a file system implementation blink). It doesn't overreach and it doesn't under-deliver: a solid feature set for a first attempt on Apple's part. **Responsible journaling** Since Apple first unleashed this feature on the Mac OS X public this week, there has been a lot of sound and fury, most of it signifying less than nothing. Media outlets that should have known better rushed to declare journaling a wonderful thing for all Mac OS X users without having the slightest concept about how it works or what it does to HFS Plus disks. Others dogmatically decided not to install it at all because it was mysterious, undocumented, and apparently unsupported. Let's look at the facts. First, Journaled HFS Plus is only supported on Mac OS X Server. This is neither myth nor opinion, it is fact. If you enabled Journaled HFS Plus on Mac OS X 10.2.2 and have problems, Apple's support people will say, in so many words, "don't do that." There is no easy human interface to manage journaling, and according to information obtained by MDJ_, Apple has no intention to add journal-managing features to Mac OS X (non-server) in the future, either. The only reason it's released now is to allow Apple to sell the Xserve RAID later this year. Requiring hours of file system checking to remount a RAID array is simply not feasible. Second, journaling is in Mac OS X to stay. It was wise of Apple to include the feature in Mac OS X 10.2.2 even with no way to manage it in the human interface. Adding the feature only in the server version would require forking HFS Plus into "journaling" and "non-journaling" branches, introducing a much larger chance of problems (even when managed by conditional code). What's more, server owners can be sure that if they unmount a volume on a server and remount it on a Mac OS X 10.2.2 machine for upgrading or testing, the journal is maintained, avoiding a long file system check when remounting it on the server machine. Even if you never enable journaling for your own volumes, the compatibility story makes it worth including in client machines. Third, Journaled HFS Plus changes file system structures. Some people who have pined for journaling want to believe this is not true, and some of those people are in acute stages of denial. It is nonetheless fact: Journaled HFS Plus changes the meaning of 33 bits of the Volume Header Block, and those changes have consequences. It does not change the B-tree or catalog file structure, and in every significant way, HFS Plus and Journaled HFS Plus volumes are completely interchangeable. But there are issues with how the journaling code changes the Volume Header Block, and you should know about them. Finally, journaling is not a passive option. Due to Apple's bungled release of developer information on the topic, a program written today using all available documented information about journaling could interfere with it, either trampling the journal or worse, causing it to replay when it should not (though the journaling code is pretty defensive about this, and Mac OS 9 helps). If you're going to use a journaled volume under any operating system other than Mac OS X 10.2.2 - including booting the machine to use disk utilities - you must be aware of how the changes affect existing programs. Let's settle those rumors once and for all. **Unopened journals** -- Suppose you hold in your hands a Journaled HFS Plus disk that's been disconnected from a Mac OS X Server 10.2.2 machine. There are only two possibilities for the state of the journal on that volume: either the disk was properly unmounted and the journal is empty, or it was improperly unmounted and the journal contains information necessary to preserve the integrity of the file system. In the first case, you're relatively safe no matter what happens. The file system is intact and the volume is ready to be mounted on any new computer. The only question is if something you do to the volume disrupts journaling, such as by moving the journal files or messing with the Volume Header Block. We now know what happens in each case: if a utility clears bit 13 of the HFS Volume attributes, the journaling code assumes you've disabled journaling and does not journal the next time you mount the volume on a Mac OS X 10.2.2 or Mac OS X Server 10.2.2 machine. If a utility tampers with the journal files, the journaling code conservatively assumes they're invalid (since they are) and reinitializes them. Either way, since there was nothing vital _in_ the journal, your disk is safe. The latter case is more troublesome. If the volume was unmounted improperly (for example, by unplugging a FireWire cable), you must presume the journal contains entries that haven't been committed to their final location on disk - in other words, that some entries must be replayed to bring the file system to a consistent state. You _may_ be in trouble if the journal should be replayed but is not, depending on why it doesn't replay. You _are_ in trouble if the journal shouldn't replay but does, since that overwrites newly valid structures with older journal entries. Two more parts of the HFS Plus specification bear on how any HFS Plus implementation treats volumes. One field in the Volume Header Block is called lastMountedVersion and is supposed to tell any code that implements HFS Plus what implementation _last_ mounted this volume. All versions of the classic Mac OS that implement HFS Plus (that's 8.1 through 9.2.2) use the four-byte constant named kHFSPlusMountVersion, which equates to the ASCII string "8.10" (the first version of Mac OS to implement HFS Plus). A third-party utility that mounts an HFS volume just to repair it is supposed to put its four-byte creator type in the lastMountedVersion field. Journaled HFS Plus seems to use the value 'HFSJ', though we've had trouble confirming that. The point of the field is to allow new code to know if older, troublesome code has been using the volume. For example, if Journaled HFS Plus sees the value "8.10" in the lastMountedVersion field, it knows that the volume has been mounted under classic Mac OS since the last time Journaled HFS Plus saw the disk. It therefore has reason to question the validity of the journal and probably checks it for consistency. The other relevant part of the specification is the definition of reserved [16], a more specific term than "unused." The official HFS Plus documentation says that bit 13 of the attributes (the new "is journaling enabled" bit) and the four-byte pointer to the journal info block are both reserved, designations that will obviously change in the next revision of the specification. Any code working with HFS Plus structures, however, has very specific instructions on how to treat reserved fields. Any HFS Plus implementation that sees a value in a reserved field must ignore it, explaining why Apple can start using those fields without freaking out every utility. Any implementation that modifies an existing HFS Plus volume must preserve values in reserved fields - copy them without modification. Finally, any program that creates new HFS Plus structures must put zeroes in all fields that were reserved at the time the code was written. (Zeroing fields that were reserved in the future violates those pesky laws of physics.) [16] Got that? (Reread it if you need to, we can wait.) Now we can explain when things might go wrong. **The complex cases** -- All Journaled HFS Plus volumes that weren't unmounted properly share some characteristics. The attributes in the Volume Header Block show that the disk is journaled and was not unmounted cleanly. The lastMountedVersion field contains Journaled HFS Plus's signature (we think it's "HFSJ"), and the formerly reserved field of the Volume Header Block points to the journal info block. It's only when all of these conditions combine that, upon next seeing the volume, Journaled HFS Plus attempts to replay the journal and restore inconsistent file system structures. That alone almost guarantees you'll never get an unwanted journal replay after repairing the drive under Mac OS 9. Unless the drive is so damaged that Mac OS 9 can't mount it, simply plugging it in should mount the drive. That replaces lastMountedVersion with the traditional kHFSPlusVersion signature, "8.10", and makes sure the journal will not be replayed. Journaled HFS Plus knows that you've mounted the volume elsewhere since the journal was last updated and backs off. What's more, if you mounted it under Mac OS 9, chances are good you unmounted it properly, so the volume's attributes wouldn't even say it needs repair by the time Journaled HFS Plus sees it again. Therefore, if you choose to take such a volume and mount it under Mac OS 9, you'll need to run your own disk utility program to repair any damage. By mounting it under another version, you've told journaling not to replay the journal next time, and taken the responsibility for repairing the volume on yourself. Don't think Mac OS X will do it for you, either: although the OS still runs fsck on newly-mounted volumes (at least during startup), Giampolo's changes to fsck_hfs for version 10.2.2 clearly show that it entirely skips journaled volumes. You can force it to make the check [17] with the "force" option, but the OS doesn't do that for you. Once you mount the volume elsewhere, repairs are _your_ responsibility. [17] In fact, that's true even if the disk is so damaged that only a utility can mount it. Under the HFS Plus rules, such utilities must put their creator type in the lastMountedVersion field. Therefore, the next time Journaled HFS Plus sees the volume, it should have a lastMountedVersion other than "HFSJ", meaning the journal won't replay. This prevents a potentially disastrous situation where a disk utility repairs almost everything just to have Journaled HFS Plus ruin it by trying to replay any journal it finds. That's why we said you _may_ be in trouble if the journal should replay but doesn't. If the journal doesn't replay, chances are it's because you mounted the volume under Mac OS 9 or on another machine and ran a repair utility on it. Such repairs won't be as good as if the journal had replayed - you may lose access to some files that journaling could have saved - but the volume will be consistent and ready to use. If whatever repair program you use fixes all damage, or if the journal was empty anyway, you've lost nothing in the process. Putting all this together, the best plan for any badly-unmounted Journaled HFS Plus volume is to remount it on any Mac OS X 10.2.2 or Mac OS X Server 10.2.2 machine before doing anything else. You don't necessarily need to boot from a damaged volume - try using FireWire Target Disk Mode and just mounting it on some other machine. That will replay the journal and restore the file system structures. You can then unmount it safely and try any further disk repair you want. Until you have a disk utility that you know is completely journaling-savvy, this is the easiest and safest path to repairing any journaled volume. This only works, however, if all implementations and utilities follow the rules. If any implementation of HFS Plus or any mounting utility does not set the lastMountedVersion field properly, there's going to be structural drama. The next time Journaled HFS Plus sees the volume, it may find a valid journal and its own "HFSJ" signature in the lastMountedVersion field. Journaled HFS Plus should refuse to replay the journal if it sees that the volume was properly unmounted by such a utility or implementation, but we haven't yet been able to verify that. The point, and the bad news, is that a buggy HFS Plus implementation could fool Journaled HFS Plus into incorrectly replaying a journal under specific circumstances. The good news is that it would be hard to recreate this situation - we tried. Every popular HFS Plus utility mounts volumes after it finishes repairing them, and as noted, that should cancel journal playback. To set up a bad journal playback situation, a utility would have to leave lastMountedVersion untouched (a bug) and unmount a volume without marking it as cleanly unmounted (another bug). Apple's older documentation may not have been as strident about setting these fields properly as current documentation is, but the current documentation has been out for nearly two years. It shouldn't be a problem. **Specific utility notes** -- We tried getting two popular and current HFS Plus utilities to destroy a journaled volume without success, regardless of whether it was unmounted properly or improperly. Although each program had different effects, neither provoked really bad things on the disk. Norton Utilities 7 [18] repairs disk damage in place. Per the definition of "reserved" fields mentioned earlier, the HFS Plus specification requires the program to preserve all values it finds in reserved fields, including the new attributes flag and four-byte journaling block info pointer that are no longer reserved. Norton Utilities 7 obeys the specification, preserving these values, even though it means the journal info block now often points to the wrong block on disk (particularly if you've optimized the disk with Speed Disk). Fortunately, the journaling code is conservative: when it sees that another implementation has mounted the volume, it checks to make sure the pointer in the Volume Header Block has the same block as the one in the ".journal_info_block" file. After Speed Disk, it finds them to be different, so journaling resets the pointer in the Volume Header Block to point to the block in the ".journal_info_block" file. As noted earlier, that prevents journal playback, but since Norton Disk Doctor probably fixed what it could (or Speed Disk moved everything the journal would touch), that's a good thing. [18] DiskWarrior 2.1 [19] takes a different tactic, creating an entirely new optimized directory structure and writing it out to disk. The definition of "reserved" fields requires DiskWarrior to write zeroes for all values that were documented as reserved when the program was written. DiskWarrior therefore must write zeroes to the "is this volume journaled" attribute bit as well as to the field that points to the journal info block. The latter is not a problem, because as noted, journaling resets that during disk mounting if it's invalid. The former, however, clears the flag that definitively tells Mac OS X 10.2.2 that journaling is enabled on the volume. When DiskWarrior 2.1 writes a new Volume Header Block with that flag clear, it effectively turns off journaling on the volume. Alsoft confirms this on DiskWarrior's Web page: "DiskWarrior will rebuild disks with Mac OS X 10.2.2 installed and journaling turned on, but it will deactivate journaling. You will need to turn journaling back on once DiskWarrior has finished its rebuild." [19] Other utilities should behave like one of these two popular ones. If a utility rebuilds a volume in place, it should leave journaling enabled when it's done even though any unplayed journal entries are lost. Utilities that replace directory structures with new ones will turn journaling off completely. You'll get error messages when you re-enable journaling, telling you that the ".journal" and ".journal_info_block" files are already hidden, but you can ignore those. **The final page** How much easier do you think life would have been for users and developers if the information you've just learned had been included in all of the Mac OS X 10.2.2 seed notes, and was published concurrently with the release of the update? Arguments that the journaling code was "widely seeded" are without merit. In the course of a three-day investigation, MDJ_ learned that the existence of journaling was disclosed only in the seed notes that accompanied Mac OS X Server 10.2.2 build 6F12. The journaling code was not included in the previous seed, build 6F6. All this was very close to the end of the testing cycle as well, since the release build is numbered "6F21." Even though journaling is included in Mac OS X 10.2.2 and obviously affects how Journaled HFS Plus volumes are mounted and used, the non-server seed notes _never_ mentioned it. The server seed notes never mentioned it again after build 6F12, released on 2002.10.21. That was just _three_weeks_ before the software was released to customers, giving no possible way for developers to have tested and revised their code for problems - particularly since Apple did not include documentation of how journaling works in the seed. You've learned more here than most developers know about it now. Preliminary documentation on journaling's data structures was sent to developers that Apple knew were writing disk utilities, but only just before the 6F12 seed. Disk utility developers were reluctant to speak to MDJ_ about journaling and its effects on their products, possibly because they feared getting even less information from Apple in the future. Unless you're a paid member of Apple's developer programs and the company knows you're making HFS Plus manipulation software, you can't get access to journaling information today. It will be available soon, no doubt, but it's not there today even though it's not that complicated. A programmer who goes to Apple's Web site today and downloads documentation for HFS Plus can write a program that completely follows the specification and yet, like Norton Utilities and DiskWarrior, interferes with journaling. At the very least, following the outdated specification assures that journal entries are lost if you mount a Journaled HFS Plus volume using anything other than Mac OS X 10.2.2 or later. Utilities should recognize the journal and replay its contents, not ignore it and perform less-complete repairs on their own. That's impossible today, because Apple doesn't allow it yet. Compare this to late 1997, when the new HFS Plus file system structures were documented and seeded to all developers months ahead of their release in Mac OS 8.1. Developers may not have had their code ready in January 1998, but they knew what to do and how to do it, and most had any necessary changes well underway by that point. Today, only Apple's handpicked disk utility developers even have a chance at changing their code. It's like changing a portion of the Autobahn to be one way in the opposite direction without marking it, but brushing aside criticism because you told the top five drivers. Arguments that journaling was openly discussed on Darwin development mailing lists are equally without merit. The day after Mac OS X 10.2.2's release, MDJ_ searched all of Apple's Darwin mailing list archives [20] for "journaling" in various spellings, with no relevant matches. But even that is beside the point: every hard disk Apple has shipped in every Macintosh for over four years has been an HFS Plus volume. The code change affects everyone, not just Darwin developers, and programmers shouldn't have to join dozens of mailing lists to find vital documentation. HFS Plus's volume format is readily available from Apple's developer Web site; the journaling changes should be no less accessible. [20] The HFS Plus release in 1998 was largely handled by Apple's experienced Mac OS managers and went quite well. This time, Mac OS X 10.2.2 was overseen by Apple's NeXT imports, and their lack of experience and maturity in large-scale software releases is showing. The same group of managers had to be persuaded at high volume during Mac OS X 10.2 testing that the operating system must maintain binary compatibility: to introduce some new features, Apple's upper engineering management wanted to change the operating system such that existing programs would have to be recompiled before they'd run under Jaguar. Having only experience shipping an operating system to a small number of academic and corporate customers that were all developing code, the team had no clear concept that telling users to update _every_piece_of_software_ was a bad thing. This time, an unnecessary penchant for secrecy has kept vital information from users and developers that needed it. It was unnecessary because customers aren't going to delay purchasing machines until a journaled file system ships, especially when it's just three weeks away. They'll upgrade the OS and get it when it's released, just like you did. Any corporate, academic, or in-house developers manipulating HFS Plus structures are left in the cold with journaling until Apple releases the real documentation, and there's absolutely no reason it should have been kept secret. Because no utility today can properly process journal files, you must take special precautions if you move a Journaled HFS Plus disk to another operating system, even for repairs. To recap: if the disk was unmounted improperly, try to mount it on a Mac OS X 10.2.2 machine first to let the journal replay. Then take it wherever you want for repairs. Remember that mounting a Journaled HFS Plus disk on anything earlier than Mac OS X 10.2.2 will invalidate the journal, losing any information it may have contained. Journaling corrects errors introduced by Norton Utilities, but using DiskWarrior 2.1 disables journaling on a volume. You must re-enable journaling in Mac OS X 10.2.2 and later if you want the volume to remain a Journaled HFS Plus disk. Should you use journaling? Not unless you have large HFS Plus volumes that you mount and unmount regularly, including those permanently attached to computers you reboot often. Replaying even a large journal is faster than checking the entire file system of a big volume. However, by "big," we're talking over 200GB in a single volume. Smaller volumes, or those that stay mounted for long periods of time, won't benefit that much from journaling. It extracts a small performance penalty (between 5% and 10% according to early estimates), but makes recovery much faster. It does not make your system "more stable" - journaling can't prevent crashes, only get you past them more quickly. Journaled HFS Plus is not a bad first step. It is more "near" HFS Plus than "of" it, though. It's a quick implementation that neither survives mounting in older implementations nor works with disk quotas [21]. It's not intended for use anywhere but Mac OS X Server, the only place it's easily configurable and fully supported. It can't protect data files, and unless it gets a chance to replay its journal before you try repairs in other programs, it won't protect even directory structures. It's not a passive option: you must be aware of journaling and follow slightly different troubleshooting steps if a journaled volume goes offline. [21] Such changes probably don't bother server administrators, the target audience for Journaled HFS Plus. Those folks will rest a little easier knowing that the inevitable and unexpected glitches won't keep their large volumes offline for hours while fsck checks them out. They're less likely to boot into Mac OS 9 or be unaware of journaling's consequences. Corporate or in-house developers who have directly manipulated HFS Plus structures are probably aware of it and can stop their customers from using journaling-unfriendly applications for now. For these hard-hitters, journaling is a great new feature. For everyone else, it's a curiosity that adds maintenance baggage without substantially strengthening the system, and is best left disabled for now. If your disks ain't broke, don't fix them. ----------------------------------------------------------------- MDJ_, The Daily Journal for Serious Macintosh[tm] Users, is published by GCSF, Incorporated. Publisher: Matt Deatherage Staff: Justin Seal Nathaniel Irons Jerry Kindall John C. Welch MDJ_ contains news, information, strong opinion, parody, biting sarcasm, and things you need to know. Those easily offended should seek information elsewhere. Humans often answer the telephone between 10 AM and 6 PM Central (US) Time, Monday through Friday. Voicemail is available at any hour. This file is formatted as setext. For more information, send email to . A file will be returned shortly. It is also digitally signed using PGP technology to verify the integrity of the transmission. Our DH/DSS corporate PGP key may be obtained at . Copyright (c) 2002 GCSF, Incorporated. All rights reserved. All trademarks are the property of their respective holders and owners. GCSF, Incorporated. P.O. Box 1021 El Reno, OK 73036-1021 (405) 262-1399 -----BEGIN PGP SIGNATURE----- Version: PGP 8.0 (Build 349) Beta iQA/AwUBPdUENbH4QMSEyVHCEQJfAwCfRpl/IucQpNiJPeQ2vMKi0LUYDL8AoP13 2/hUAgz99MECCTEn35xy01CH =IKhE -----END PGP SIGNATURE-----