What we learned after spending 20 hours repairing a server

It's amazing how Mac OS X almost, but not quite, works correctly if /var is a real, empty directory and not merely a pointer to /private/var.

You would think "repair permissions" or an OS updater would detect such a mistake and correct it.

You would be wrong.


# - Posted to Administrivia on 9/27/07; 2:08:58 AM

Tools we use: LaunchBar for $10

We're probably a little late to this party, but we wouldn't think of using a system without LaunchBar, and it's on sale today for half price through MacZOT. Go stock up.


# - Posted to Administrivia on 9/10/07; 6:48:27 PM

Updated Acrobat FDF certificate

Three months ago, we explained how to download our self-signed Acrobat certificate and use it to verify the signatures on PDF issues of MDJ and MWJ.

As it turns out, self-signed certificates are good for five years, and we started using Acrobat's own technology to sign PDF issues five years ago last week (2002.07.17). We can't sign any new issues with it, although it's still valid for verifying issues we signed with it.

Today we've issued a new self-signed certificate, available here or from our certificates and security settings page here.


# - Posted to Administrivia on 7/24/07; 5:29:43 AM

MDJ on iPhone is really cool

Well, we think so, at least. We can't really take sharp pictures of it, but try mailing yourself a PDF issue of MDJ or MWJ, making sure you don't compress it with Zip or StuffIt or anything else. Then read the message on iPhone - at the bottom of the page, you'll see an attachments button that lets you open and read the PDF right there on the phone, with the proper font rendering and everything.

Frankly, this surprises us, but we're not complaining. OK, we're complaining about two things:

  1. iPhone's PDF reader does not recognize hyperlinks within a PDF document. You can see that something is a link from the blue text, but tapping it does nothing.

  2. Even though MDJ is presented in two columns, iPhone's double-tap-zoom metaphor does nothing but zoom the full page to fit the iPhone's screen. We even tried testing an older issue of MDJ that had PDF articles defined for the text, allowing Acrobat and Acrobat Reader to follow stories across columns and pages automaticcally. No dice - iPhone's PDF viewer knows nothing about them, so they don't provide any advantages.

    (Ironically, we stopped including the "article" features in MDJ and MWJ in 2002 with the new design because Adobe InDesign has no way to generate them from columns and text frames on the page. InDesign has its own PDF export that does a good job in many areas, but this has been a glaring omission since version 1.0.)

We've always compressed MDJ and MWJ issues for delivery for a few reasons:

  • The #1 error we used to get in delivery was "mailbox full," so naturally we want the issues to be as small as possible.

  • In the days of Mac OS 9, compression was necessary to include HFS metadata, like the file type 'PDF ' and the creator type 'CARO', necessary to allow double-clicking the file to open Acrobat.

  • When we started this 11 years ago, most people didn't have broadband services, and those outside the US were slower than those here. Downloading big files could take a long time.

We've long considered ditching the compression and sending the file as MIME type "application/pdf" because Mac OS X's "Mail" application can display uncompressed PDFs inline, but that would have left people who want compressed files without options, on top of rewriting our software to do the new thing.</P.

But now, since mail is so spammily broken to begin with, we have ZIP-compressed RSS feeds for people who want compressed files, and Apple continues to improve the experience for people using its products if we mail uncompressed PDF files. The RSS feeds are irrelevant to the iPhone - it redirects display of any RSS URL to Apple's "reader.mac.com" Web application, but reader.mac.com cannot access or display secure RSS feeds like ours, so at present, you can't view MDJ or MWJ RSS on the iPhone.

Therefore, starting around 1 August (2007.08.01), we're going to change our delivery system to mail PDF versions of MDJ and MWJ without compression, as MIME type "application/pdf", encoded with Base-64. We'll also add a new "no E-mail" type of subscription for people who prefer compressed files in RSS - when a new issue is published, we won't send you E-mail at all, just let you find it in the RSS feed. (After all, if you want compressed files, it makes no sense to mail you an uncompressed PDF or setext version that you don't want. If you still want those, they're still available, on top of the RSS feeds available to all subscribers.)

We'll announce this in MDJ and MWJ also, but since the "StuffIt file in a Binhex wrapper" format of MDJ and MWJ PDF delivery hasn't really changed in over a decade (except to move to the "newer" StuffIt 5 archives in the late 1990s), we thought we'd give some of you a heads-up in case you have mail filters, automatic processing, or anything similar. Most of you won't notice any difference except that PDF issues won't need to be decompressed before viewing. In most modern mail applications, you'll see an enclosure icon that opens the issue with a single click - and in iPhone, you can tap the enclosure to read it. The coolness factor there isn't going to wear off for a while around here.


# - Posted to Administrivia, MDJ, MWJ on 7/4/07; 3:15:14 PM

How to convert Webstar 4 SSL private key to Apache format

This isn't super-topical, but when you need to find this information, it's almost impossible to come up with it in a search unless you hit exactly the right information. So, for posterity:

Private SSL keys generated with WebSTAR 4 are stored in a binary ASN.1 sequence format. To use them with Apache, you have to convert them to PEM format. Michael Fischer provides the following command-line sequence to do it:

openssl asn1parse -inform DER -in <private-key> -i -offset 21 -out newkey

This should generate a DER-encoded PKCS#8 key using your old keyfile.

Now:

openssl pkcs8 -in newkey -inform DER -outform PEM -out pemkey

This should convert the DER-encoded key to a PEM-encoded private key.

Then, try having Apache use the resultant "pemkey"...

Just so you can find it if you need it.


# - Posted to Administrivia on 6/12/07; 1:07:08 AM

Slight downtime today

Just by way of warning: we need to replace a failing hard drive in one of our server machines this afternoon. The actual installation, plus final duplication from the old drive to the new drive, will require taking the server offline for the duration of those steps of the process. During that time, E-mail to our domains may be interrupted, as will some of the smaller weblog hosting we provide for staff and friends. This server (macjournals.com or www.macjournals.com) is unaffected, which is why you're reading it here.

Nothing to worry about - just an old hard drive that's an accident waiting to happen, getting replaced with a new part that should make us spend a lot less time baby-sitting it for the life of the server. Thanks!

Update (9:00 PM CDT): Connectivity is back, but mail and weblogs are still offline while the cloning continues. Don't worry about bounces - your server almost certainly tries to send mail every 30-60 mins for 2-3 days before giving up, and full (faster) service will be back in a few hours. Thanks!

Update #2 (8:00 AM Fri CDT): The server is restored and connectivity is coming back - thanks for your patience!


# - Posted to Administrivia on 6/7/07; 5:41:42 PM

Digitally signed PDF issues

As the MWJ reboot gets ever closer, we're reminded today that when some of you open PDF issues of MDJ or MWJ in Adobe Acrobat or Adobe Reader (or Adobe Acrobat Reader - perhaps the next version will be called "Acrobat Acrobat Adobe Reader Adobe Adobe Read Reader"), you see dialog boxes like this one:

Although the Language is stilted, and Nouns are strangely capitalized, the first Sentence is the one you want:

The Digital Signature that was used to Certify this document shows that the document has not been tampered with, though the author's Certificate could not be verified.

Our setext issues are digitally signed using industry-standard and mostly-open PGP technology. Long long ago, we used PGP to digitally sign issues of MDJ and MWJ as well, so readers that care could verify that their copy of the issue had not been modified, not by a single bit, since we signed and sent it.

However, we stopped using PGP to sign PDF files in the early 21st century when Adobe built digital signatures into Acrobat, because that way the file can contain its own digital signature. With PGP, we had to sign the issue as a file, producing a separate ".pdf.sig" file that has to accompany the PDF file in distribution.

Digital signatures use public key encryption, so verifying them requires comparing part of the signature to the signer's public key. The PGP keys we use to sign issues are on PGP's keyservers as well as here on our own Web site. For Acrobat, we created a signing key in July 2002, and we make that available to you here. Download, unstuff, and open the ".fdf" file; Acrobat should launch and ask you to import the certificate, as well as whether you trust it or not. You can find more information about sharing Acrobat certificates and security settings here.

We're updating this information today for two reasons:

  1. Due to the traditional problem with relative and absolute URLs, the link to the FDF file on our keys page was incorrect until today.

  2. The MD5 and SHA1 fingerprints for the FDF file, as posted, weren't easy to verify with current versions of Mac OS X (we don't even quite remember how they were generated back in 2002). To solve the identity problem, we've moved the file to our secure server. As long as our SSL certificate is current, your browser verifies that it's getting the FDF file from our secure server as part of the transaction, so you know it came from us.

Sorry for any inconvenience the missing FDF file may have caused. Let us know if you have further questions on MDJ or MWJ digital signatures.


# - Posted to Administrivia on 4/30/07; 1:32:23 PM

OK...

...despite this week's attempts to undo the news work, we now think we have the giant collection of Stoopid that is the past few weeks in the tech industry sorted into the appropriate piles for the requisite demolition.

(See this for an example of why we've had to use large quantities of duct tape to keep our heads from exploding. That's just one we decided we couldn't put in one of the piles lest it devour the whole thing.)

Give us a couple of days without phone and with slow E-mail response and we'll have something. And we don't even need to patent the piles.

(By the way, per the previous item: we did lose power right during a writing/editorial session - but only 4 days after the storm had passed. Go figure.)

Update: Aw, crap - the auto-generated invoices have been jammed up longer than I thought, too. We've sent the most urgent ones and will flush a status message for everyone after MWJ publishes in the next 48 hours. When something derails, getting those boxcars back on the track takes more effort than we remember.


# - Posted to Administrivia on 2/10/07; 3:29:40 AM

Local winter storm warning

When the weather gets as severe as it does around here, we learn to take warnings seriously even if the actual weather probably will not be as bad as they anticipate. After a summer of record-breaking heat, the winter has so far been mild.

They're saying that ends tomorrow:

THE NATIONAL WEATHER SERVICE IN NORMAN HAS ISSUED AN ICE STORM WARNING...FROM 6 PM FRIDAY TO 6 PM CST SUNDAY.

.PERIODS OF FREEZING RAIN WILL OCCUR BEGINNING FRIDAY NIGHT AND CONTINUING THROUGH SUNDAY. HEAVIER PRECIPITATION WILL LIKELY OCCUR FRIDAY NIGHT WITH ANOTHER ROUND EXPECTED SATURDAY EVENING INTO SUNDAY MORNING. SLEET MAY MIX WITH FREEZING RAIN ON SUNDAY BEFORE THE PRECIPITATION ENDS.

ICE ACCUMULATIONS OF ONE HALF INCH TO ONE AND A HALF INCHES ARE EXPECTED WITH THE MOST SIGNIFICANT ACCUMULATIONS EXPECTED TO EXTEND ALONG THE INTERSTATE 44 CORRIDOR. ICE ACCUMULATIONS AND WINDS WILL LIKELY LEAD TO SNAPPED POWER LINES AND FALLING TREE BRANCHES. THIS MAY RESULT IN EXTENDED POWER OUTAGES IN SOME AREAS.

Some of you may recall that in February 2003, GCSF World Headquarters and its entire neighborhood lost power for over three days. Roads were passable even as power lines were covered with ice, so we decamped to another part of the state for a couple of days. That was up the "Interstate 44 corridor," though, so we're not sure we could do that this weekend.

We have plenty of options if the power goes out for more than a few hours; we learned lessons in 2003. We have food and can find local shelter if necessary. This server, providing news and issue distribution, is co-located in Oklahoma City with its own power; mail and local internet services at GCSF World Headquarters are completely separate. Even if we lose local power or Internet, this server won't go down, though mail may not reach us for a few days.

(If you've been with MDJ for a long time, you may know that we didn't have a secure subscription page until 1998. If you subscribed before then, you probably called us; if you called, you probably talked to Marie. She says "hi," but she's even more prepared than we are - in 2003, her house was without power for sixteen days. The backup generator and power cut-off switchover are all tested in the past few weeks and ready to go, you can believe it.)

We've rebuilt the RSS feeds today so MDJ 2006.01.09 (with pre-keynote Macworld Expo coverage) is now in the MWJ RSS feed. We're working on more - yesterday we managed to get NetNewsWire (a primary news collection source) down to 2 unread articles from a peak of, I believe, 3785. (It was at zero on Tuesday morning.)

If weather interferes there may be more delays, but we're prepared to work around them as best we can. Theoretically, the more prepared we are, the less likely we are to need any of those preparations, so there we go.


# - Posted to Administrivia, In The News on 1/11/07; 5:31:01 PM

Happy New Year!

We hope you all were out partying to ring in the new year. We were here, in the office, but we don't mind - in some ways, a lot of the second half of this year felt like an enforced vacation. You party, we'll write!

We will be out of the office for most of New Year's Day due to individual obligations, which is also why we can't stay here until 6AM to get the current issues finished, but we're optimistic for the next 36 hours. This security stuff has kicked our asses for the entire second half of 2006, but it's not 2006 anymore, and we're rolling right along. We're finishing answers, reaching new conclusions, dissecting conventional wisdom - it's all great fun, mitigated by the fact that it's not yet in your hands. (Who decided to put two holidays in seven days at the end of the year? Annoying!) But it's good stuff - see if we're wrong when you get it.

And then - well, I'm not allowed to say, but it should be great fun. (We're not talking about the stock options thing, either. That's got unexplored level of weirdness in it, and we want to look at a few more things about it first.)

Happy new year!


# - Posted to Administrivia on 1/1/07; 2:41:49 AM

Personal and work updates from Matt

Hi, folks:

As hinted in the previous entry, , I was hospitalized last week with what I thought was a breathing problem, but turned out to be congestive heart failure. My heart was pumping less than half the blood on each beat as it should have. This is why I've had no energy all summer, why walking from one session to another at WWDC exhausted me, and why even finding and eliminating allergens in the GCSF studio didn't really solve anything. As it turns out, I did have pneumonia as well, but it was largely a complication of the heart problem, not the root cause itself.

I'd love to be a spokesperson and say "have such-and-such checked so this doesn't happen to you," but there's really no one key to having avoided it - the doctors' best guess, at present, is that I picked up a virus earlier this year that damaged my heart muscle and led to cascading problems. I'll undergo tests early next month to confirm I don't have blocked arteries, but none of the doctors believe I do, and there are no family signs for CHF at my age (I turn 40 later this year). I had been under medical care for the respiratory problems this summer, too, so it's not like I was ignoring them. Apparently lots of CHF cases are mistaken for asthma or other breathing problems at first.

I've been home for close to a week, but it's still an adventure adjusting to the new routine, the new diet, and so on. The doctors haven't restricted my activities because moving around helps me heal, but sitting in an office chair tires me out as much as standing after a while. I'm getting stronger, but I'm not there yet. I'm going to be out being non-sedentary for much of the next few days, if I can handle it.

We hope to resume MDJ and MWJ production early next week. I've spent the energy I've had keeping up with what's going on, and now I just have to find the energy to get it into print for everyone (not optimal, but you have to admit, it pretty much can't work the other way around). We are still paying attention to what we see, like these things:

  • We were a bit surprised that StuffIt Deluxe 11 is an upgrade, since Smith Micro promised last year that the Mac version with StuffIt JPEG photo compression would be a free update. Then again, that's probably why it is a free update to anyone who purchased StuffIt Deluxe after mid-August 2005. It remains to be seen if the new version fixes the annoyances and bugs of version 10 (we don't have a review copy yet), but StuffIt is fighting an uphill battle.

  • There's no reason to avoid Apple's latest AirPort security updates, but you should rely on Software Update to tell you which of the four updates to install, since it's a bit confusing to do it by hand. (AirPort Update 2006-001 applies only to the latest Intel-based systems with Mac OS X builds later than those generally available; everyone else needs to use Security Update 2006-005 for either PowerPC Panther, PowerPC Tiger, or Intel-based Tiger systems that aren't so new that they need the AirPort Update.)

    We've uploaded the list of changes in all four updates in OPML format for your perusal, and if you look, you'll see that they're all small updates that are tightly focused on AirPort driver issues.

    According to Apple, as told to Macworld, the fixes in the updates came from Apple's own internal code review, prompted by the public claims that MacBook computers could be attacked and controlled without requiring the user to do anything other than have AirPort turned on (MDJ 2006.08.30). Apple insists that researchers David Maynor and Jon Ellch did not provide any specific information allowing Apple to find any specific vulnerabilities, but the focus on the issue and on similar flaws in the BSD stack led Apple to find several places where such attacks might occur, even though the company has seen absolutely no evidence that any of them did occur.

    We'll have more on this in the next issues of each journal, but for now, let's point out that far from claiming victory, Maynor and Ellch have now lost any shot at credibility they had left. Now that descriptions and fixes of these potential exploits are in the wild, Maynor and Ellch could easily look at the fixes (even in the Darwin source tree) and say, "Oh, yeah, that's what we found." But when the researchers had two months to prove it, even to Apple Computer itself, they refused. They provided only vague, non-specific hints about "something bad," refusing to let any Macintosh security experts see the exploit, and in fact refusing to demonstrate it live before anyone who might be expert enough to catch Mac hanky-panky in progress.

    They said both that they did and didn't find a way to exploit a standard MacBook, and every time anyone asked for details, they either said they were withholding them for the common good, or made mysterious and completely unsupportable accusations that "Apple legal" told them not to talk about it. (Apple Computer, like any other entity, has no legal right to tell people who aren't under non-disclosure agreements what they can talk about at any time, and any court order enforcing silence would be on the public record.) It's possible that Maynor's employer, SecureWorks, threatened him if he didn't remain silent, but that's neither Apple's fault nor a credit to Maynor. It was he and Ellch who sought publicity by going to the press before showing video of their alleged hack at the public conferences Black Hat and Defcon, so if there's fallout from making public allegations that they can't back up, it's on them, not on anyone else.

    If Maynor and Ellch had demonstrated it or shown code to just one Mac expert who could have verified their claims, they'd rightly be lionized for their work. Instead, they took credit for "hacking a MacBook" at security shows and in the international press while refusing to provide even the barest proof that they'd actually accomplished what they said they had, or at least what they wanted you to believe they'd said. Now that bugs and fixes are in the real world, there's no way of ever knowing if what they say they found matches those bugs or not - when they had the chance to prove it, they refused. It's like saying after the fact that you knew the answer to Final Jeopardy - you have to say it before it's revealed to get credit for knowing it.

    Thus, for now, ends the tale of two hackers who sought the limelight and couldn't handle it when they got it. There is no reason to believe that Maynor and Ellch found anything that could actually take over an unmodified MacBook computer, and now that Apple has publicly released fixes for unexploited bugs that might fit the bill, they'll never be able to prove they found them first. There is zero evidence that Maynor and Ellch found what they said they found, and any rational observer has to treat their findings that way. If they wanted the credit, they had plenty of opportunity to provide proof instead of vague accusations, and they refused every time. If they get bitter, they can go commiserate with Fleischmann and Pons.

We'll get all this in better order and out the door in the next few days. I've appreciated the cards and letters from those of you who had heard of my heart failure, and I thank you for realizing that replies to E-mail or voicemail may be delayed for several days while I focus my energies on more important stuff. I haven't had a lot of energy for a long time, but it's slowly coming back, and at least now I can breathe, so things are looking up.

You have my eternal thanks for your patience - had it not been for MDJ providing medical insurance, I might not even be here to tell you this. It's all new for me, but I'm getting used to it, and look forward to a more normal schedule just as soon as I can make it happen. Thanks again!

Matt Deatherage
Publisher, MDJ & MWJ


# - Posted to Administrivia on 9/22/06; 9:24:01 AM

An update on the publisher

MDJ and MWJ's publisher, as noted on Tuesday, is in the hospital. As recently as one week ago, the discovery of mold and algae in the GCSF Production Studio's ventilation system gave us a working theory of "sick building syndrome," but treatments were not effective. He went to see his doctor on Tuesday morning, and after an examination of his obvious respiratory distress, his doctor recommended immediate hospitalization and aggressive treatment of the respiratory problems.

In the course of that treatment, they found signs of an underlying cardiac problem that had been masked by the breathing trouble he'd had all summer long. He was transferred today to a leading heart hospital in the area, where aggressive treatment for that has already shown great results. He will still be sidelined for several more days, but he's already trying to catch up - believe it or not, the hospital offers wireless internet access!

This is a serious thing, and it will require serious and ongoing treatment, but our present expectation is that he will be back in the office no later than the middle of next week. Thank you for your patience and support in this time.


# - Posted to Administrivia on 9/14/06; 6:15:11 PM

A significant schedule setback

We were hoping it wouldn't come to this, but after consultation this morning, Matt's physician urges that he check into the hospital for "a few days" to treat what has apparently now devolved into pneumonia or other related conditions.

We'll post more information as it's available. Thank you for your patience.


# - Posted to Administrivia on 9/12/06; 9:25:28 AM

A few answers to current questions

  • I last got MDJ or MWJ on such-and-such a date. Has there been an issue published since then?

    Our status page lists the current issues of both MDJ and MWJ, including issue sizes, and when distribution began - and it's been there (and up-to-date) for more than five years. Unless your or our Internet connection is down, this information is always instantly available to you.

    As of this summer, subscribers can also get the same information in their secure RSS feeds. We sent this information to all current subscribers in June, and it's been part of the "Welcome to MDJ" (or 'MWJ') letter for all subscribers since then. See here for more information about how difficult it's proven to be to tell people about this.

  • Have you published anything since then?

    We published over 30 pages of on-the-spot information from WWDC 2006 right here, available to all MDJ and MWJ subscribers. See here for our attempts to tell people about this and how they seem to not have worked very well. We've also provided a few updates on this news blog, including an article on why E-mail is broken, and why we can't use it to tell you things the way we'd like. It's not a standard "issue," but it's still a significant amount of material that some of you didn't seem to know about.

  • Where's the next issue of MWJ?

    We're sorry if we haven't made this very clear somehow, but due to problems with the ventilation in our office, working here this summer has made staff members seriously ill. We're talking emergency rooms, chest X-rays, heavy-duty prescriptions for weeks on end, significant respiratory distress, inability to sleep due to breathing problems, extensive coughing fits, multiple doctor visits - seriously ill.

    We haven't been trying to emphasize this because, honestly, there's really nothing more boring than stories about how other people are sick, is there? But from the questions we're getting, we apparently need to make clearer that the fungus in our office this summer is not like a day of a hay fever attack - it was a continuous, slow-to-build, undiscovered source of poison in the air we breathe. At this point, we're basically just extremely lucky that more staff members didn't get even more ill than they did.

    The most distressing thing about it is that when it was just getting started in June and July, and we had no idea what was going on or how serious it was, we kept spending more time in the office trying not to fall behind. The symptoms were of allergy attacks (not infections), and it seemed perfectly reasonable to go slow in front of a computer instead of at home on bedrest, so we kept trying to get more work done - and every moment we tried, we were getting even more seriously ill and had no idea.

    This does not heal instantly. We've had the ventilation fixed for nearly a week, but the staffers who work here are still having severe coughing fits and other symptoms of the toxins clearing out. (This is similar to what Matt experienced near the end of WWDC, he says - after a week away from the bad ventilation, he felt like he was getting worse, but now he realizes his lungs were just trying to expel the last of the nastiness.)

    It really has been a nasty episode, and we're still amazed that we managed to get MDJ 2006.08.30 out the door (now available to all MWJ subscribers in their RSS feeds). We're hoping to get on a regular schedule next week, and we're planning to spend time away from the studio Friday and Saturday to help make sure things are on track. (That is, if being outside for a long spell and then coming back to the studio makes us feel worse, it's a good sign something is still wrong. We have felt significantly better this week, but a sanity check seems like an excellent idea. We have follow-up doctor appointments this month as well.

    There's really only one thing we want to do more than get back to a June-style schedule around here - we hope you miss us for the same reasons we miss providing the high-quality information and reality check you expect from MDJ and MWJ. That one thing we want more? Unobstructed, regular, oxygen-rich breathing. Once that happens, the rest should be a cinch.

  • But how come I haven't seen any traffic on the MacJournals-Talk (or, as some still call it, MWJ-Talk) mailing list?

    The discussion list has been unavailable for months due to abuses of the honor system, and with everything else going on, we have not had the time to try to complete the work tying it to the subscription database. If you didn't know this, please let us know how we could have communicated it better other than trying to send E-mail to everyone, which has its own set of problems (again, see here for more information on those problems - basically, even if we put important news in the very front of an issue, a lot of people just don't see it, and then ask us months later what's going on). We'd really like to know how to do this better.


# - Posted to Administrivia, MDJ, MWJ on 9/8/06; 3:17:39 PM

Sick building syndrome

Ugh! Our publisher has been getting sicker and sicker, to the point of nuclear antibiotics and chest X-rays, all without any discernible reason. Despite that, he took some time on Friday to go to a nearby town - and after about 2 hours, felt noticeably better. Based on what he knew, that really didn't make any sense.

So, as usual, it set the gears in motion, and Friday night, we took another look at the production studio's standalone cooling, mentioned before here and here. In fact, that second instance had a rather revealing note:

(And trust us, you haven't lived until you have to interrupt a conference call to drain the results of massive dehumidifying from a heat pump with a turkey baster, and repeat it every 4 hours for several days.)

One reason the studio had been so hot was because the heat pump was not properly draining the water it extracted from the air as part of its air conditioning. When the thing got full, it couldn't dehumidify effectively, and it pumped humid, warmer air into the studio. So we've been draining it by hand where we could (the unit weighs about 200 pounds and can't easily be removed by one person, and calling it "easy" for two is a bit of a stretch).

On Friday night, we took the case off and looked at parts of it we normally don't mess with, and sure enough: mold! Right where we had noticed some water dripping when the thing needed to be drained. This, of course, was exceedingly disgusting. We wiped it all down with disinfectant cloths, sprayed every flat surface we could find (not the computers) with disinfectant, and left the studio alone for a while. Our publisher stayed away all day Saturday, too, and while he wasn't cured, his months-long condition did not worsen.

Today we came back into the studio and looked again, and found: mold! Not as much, but in the same places. This appears to be the explanation we've been seeking. The drainage problem is not just a cooling nuisance; it's causing mold to grow in the heat pump, which then puts it into the air in the studio. The more time you spend in the studio, the sicker you get. That's why our publisher, who usually spends at least 12 hours a day in the studio, has been sick to the point of getting emergency chest X-rays - but other staff members, who have been here this summer for a maximum of 40 hours per week, leave the studio and only feel like there's some kind of small allergy issue.

So with family and friends today, we got serious about it - we took the cursed appliance out of the studio and found enough crud inside it to make you think it hadn't been cleaned in a year - instead of four weeks ago, the last time we did it. (We would have posted pictures of this disgusting thing, but we remembered that we like you to much to make you look at them.)

The unit was new in 2001, replacing a 20-year-old model, and upon very close examination, it appears to have been designed for a slightly different carrier housing: unless it's tilted backward on at least a 20° angle, there's just no way water can drain from it. When the weather didn't make the air conditioner work so hard, it never came up. This summer, with it extracting gallons of water per day from the surrounding air, its collection tray became full and stayed that way - warm, wet, and dark. No wonder critters grew!

Last time we had it out for inspection (and the time before that, with the certified repair person here), we thought we just weren't seeing the holes in the bottom where the water should drain out. Today, we confirmed: there are no drainage holes. So we made some, right at the bottom, right where the water should be draining out anyway. In our test with clean water, they seemed to work fairly well. (No heat pump or A/C drains bone-dry, but neither should it have 3 gallons of standing water in it 24/7.)

We thoroughly cleaned every surface (including a previously-hidden filter), disinfected everything, and have reinstalled it in the studio. Meanwhile, since the temperature today in the Crossroads of America is a balmy 72°, we're taking the opportunity to air out the entire building, front to back and side to side. Since it rained most of yesterday, there's not a lot of crud in the air to replace the mold, so it's a good day for fumigation.

Our publisher is cautiously optimistic, especially since he thinks he was about two days away from hospitalization. We'll be checking carefully for drainage and recurrences of mold for the next few days, and he's constantly monitoring to see if he's feeling any better. But this theory, at least, fits all of the available facts as we know them. The only one that seemed odd was that a week in San Francisco didn't seem to help the publisher much, but he says that his hotel room kept sneaking in "luxurious down" pillows and comforters even after he asked them to be removed. When you have 15 pillows in a room, figuring out that one of them is making you sneeze is non-obvious. (He says, "15 pillows in a room is not luxury. It's just annoying.")

So, we may have our building fixed. Cross your fingers for us.


# - Posted to Administrivia on 9/3/06; 12:55:23 PM

RSS feed now auto-discoverable

This little news blog has always had an RSS feed, but due to issues with the blog software, it hasn't been auto-discoverable. That is, Safari and similar programs did not automatically detect the RSS feed, so we added a link in the left sidebar that almost everyone seems to have missed.

Thanks to bug fixes today, the feed now has auto-discovery, so Safari shows the little RSS icon for it just as you'd hope. If you never saw the RSS feed for this blog before, well, there it is.


# - Posted to Administrivia on 8/31/06; 6:21:03 PM

Some notes on E-mail delivery

While the cat's away and the Attitudinal mice are playing, the sysadmins took the time to write up some notes about how E-mail gets delivered. This covers everything from incoming spam filtering to the horrible thing several of you have seen: E-mails that get "spam filtered" so aggressively that they get discarded without a bounce message. You don't get your issue, we don't get an error, so nobody even knows there's a problem.

That's why we added RSS feeds.

E-mail isn't completely broken yet, but our admins have a bit of advice on what you can do to help make sure you can send and receive the mail you want to send and receive. Hope This Helps™.


# - Posted to Administrivia on 8/25/06; 3:15:52 AM

Out-of-band communications

In "Top of the Day" in MDJ 2006.08.04, right at the very front of the issue, we said this:

WWDC 2006!

Apple's annual Worldwide Developers' Conference begins Monday in San Francisco, and MDJ will be there. We have more to say in preparation for the show, but we don't want to delay this issue any further to include it. We're therefore pleased to announce a WWDC 2006 Weblog for MDJ subscribers, where you'll find up-dates from staff members on the conference and its revelations. It will debut this weekend, so check our Web site this weekend for the URL. You'll need your user ID and password for the MDJ RSS feeds to gain access, so apologies, trial subscribers, but this benefit isn't for you. We do not intend to publish regular issues during the conference, so check out the Weblog for all the latest and greatest from the show.

Similarly, MWJ 2006.08.05 said, in the same place:

WWDC 2006!

In the many, many days since we last spoke, our own production studio has fought us repeatedly - first with calamitously timed building repairs that made the normal schedule impossible, then with 19 out of 24 days of triple-digit temperatures and a heat pump that couldn't keep up (giving new meaning to "sweating over every word"). We got a lot done, such as a massive database overhaul, and an updated Web site with a news blog for announcements that many of you have already followed. Now we've cleared the deck of Apple's pressing finan-cial matters as we head to WWDC 2006 in San Francisco next week - and we'll have a subscribers-only WWDC blog available soon so you can catch dispatches from the MWJ Staff at the show. The blog isn't there yet, so check our Web site this weekend for the URL and opening date. You'll need your RSS feed user ID and password to get in (sorry, trial subscribers).

During and after WWDC, it's become clear to us that plenty of readers had no idea we had live Web coverage of WWDC. Several others wrote in to ask about their RSS user IDs and passwords, saying they'd never heard of it before.

These are problems for us.

First, on 2006.07.02, we sent individual E-mail to every MDJ and MWJ subscriber with info on the new RSS feeds. The subject line was "Info: RSS for your MDJ Subscription!" (or MWJ, if that's your journal). They all went to the same addresses to which we send MDJ or MWJ, and while one or two of them came back with "click here to get past my spam filter" replies, most of them went right on through.

Since the only E-mails we ever send to your subscription address are either issues of the journals or information about your subscription, we didn't really think these messages would get lost - and for most of you, they didn't get lost. But clearly, some people either never got them or didn't read them. We also put information about the feeds in the journals themselves, but obviously, we couldn't put user IDs and passwords into published issues. (MDJ and MWJ are not customized for each subscriber - they're more like magazines.)

Similarly, when we went to press just before WWDC, we weren't 100% sure about the exact URL and timing of the live coverage, but we knew it would be there. We put an announcement about it right at the top of the issue, in bold print (for PDF readers), and the URL was live on the front page of our Web site all that week - in fact, it's still there.

These are really the only ways we know to tell you about things - if we send you E-mail and put it on the front page of the Web site, what else can we do to bring it to your attention? Mind you, we don't care if you choose not to use or read something (we believe you to be informed consumers), but it's distressing for us to realize how many people did not even know we had added these features.

It's a bit of a letdown to publish some 30-plus pages of live WWDC coverage, advertised both in the last issue and on our Web site, only to be getting E-mail two weeks later asking if we've published anything since August 4.

We don't want to be one of those companies that sends out meaningless press releases every week trying desperately to get your attention about nothing at all, but when we add some value to your subscription, we really do want you to know about it sooner rather than later. But if individual E-mails and prominent journal placement don't do the trick, we really don't have any further ideas what will. If you do, please contact us and share your ideas!

(We are aware of one failing - we did not announce the WWDC live coverage in this news feed, an oversight due to all the scrambling and travel and such. We'll make sure to get those things posted here in the future.)


# - Posted to Administrivia on 8/22/06; 7:39:50 PM

Where things stand

We've gotten some E-mail asking about what's going on since WWDC, including some today that mentioned this news feed but that didn't know about our WWDC coverage, so let's hit a few bases.

Everyone rested a bit after WWDC, including our publisher, who noted during live coverage that he had been tired:

But honestly, folks, it's been a long summer all around North America. The international scene is more than "tense," the pressure on Apple to either meet rumors or blow past them is astounding, and it just hasn't been a good time. I've been slow and without much energy, though I think in my case it has to do with the effects of weeks of triple-digit temps and high humidity with no breaks, combined with the effects of childhood hay fever and asthma that lasted well into high school. It doesn't mean I'm ready to retire, though an actual vacation by the end of the year wouldn't hurt me a bit.

He found out this week it was more than that - it's bronchitis (and was suspected as pneumonia for a while, but an X-ray cleared that today). So he's moving kind of slow and is on what he calls the "really good cough syrup."

Meanwhile, upon returning to GCSF World Headquarters, The Weekly Attitudinal had barricaded itself into the production studio. The Attitudinal, though popular, does not get onto MDJ's schedule much anymore because it can't meet a deadline to save its pseudonymous butt, so when everyone was away, it got militant. The Attitudinal is apparently ranting on something about "changing the frame" and "command-line jerks," but the publisher doesn't have the energy to mount the invasive rescue operation it will take to dislodge the right-by-definition opinion feature from the keyboard.

We're working on little things as everyone has the energy to do so, and will have an RSS update in the next post.


# - Posted to Administrivia on 8/22/06; 7:18:55 PM

Huzzah!

It appears that the air conditioning at GCSF World Headquarters is finally fixed! Our part of the world has seen triple-digit (Fahrenheit) temperatures 18 times in the past month, and now we're extremely optimistic that the pump can keep our studio cooler than 20°F below outside temperature. We've been doing our best amid other concerns (like prepping for WWDC), but trust us, "sweating over every word" is far better interpreted metaphorically than with actual moisture.

(And trust us, you haven't lived until you have to interrupt a conference call to drain the results of massive dehumidifying from a heat pump with a turkey baster, and repeat it every 4 hours for several days.)

Our thanks to everyone for their patience, which undoubtedly exceeded ours. There should be a big publishing surge in the rest of the week.


# - Posted to Administrivia on 8/2/06; 1:40:28 AM

Maintenance update

We know we've had a lot of radio silence lately, but it doesn't mean we haven't been as busy as ever. It just means we've been busy doing stuff that, if all done perfectly, means that absolutely nothing will look different. Such is life on sysadmin day.

We had several days of building repairs that made it all but impossible to work, and then we had the heat wave that made it too hot to work as much as we wanted. Then, surprisingly, some things we expected to do in October when equipment and software became available instead happened now: we massively upgraded our subscriptions database with an eye on allowing customers to access their own records in the future (before this past week, that wasn't even an option).

We had to rearchitect several key parts to make it work in a network environment, as opposed to letting other machines enter data but only one machine processing it, an entirely different kettle of fish. That wasn't as easy as expected, but I'm told they'll have more on that in upcoming issues of MDJ.

We also got a lot of the static Web site updated today for the first time in years - believe it or not, we'd never been able to get Adobe GoLive CS2 to open our site. It crashed every time, even when trying to rebuild it from scratch. We finally found the troublesome old files today in a ".settings" folder, eliminated them, and got a good portion of the visible pages of the site validating and presenting correctly with HTML 4.01 Transitional. (Yeah, I know, welcome to 2001 - but it means that we can now update and add new pages without having to write all the HTML from scratch or just copy/paste from existing files, and that's a big win for us.)

We actually got about three weeks' worth of infrastructure changes done in the past six days, but we though the three weeks would be spread in September and October. There was no question about going for it now, though, because it will make remotes from WWDC 2006 easier and more supportable. More on that in print next week. Starting tomorrow, us sysadmin types vacate the studio for the editorial staff until called upon in an emergency. We could use the rest, though.


# - Posted to Administrivia on 7/29/06; 1:22:40 AM

RSS access restored

We upgraded database hardware and software today in a long-anticipated (at least around here) move. To our surprise, that broke our new RSS feature - the code that exports the authentication information to the server had some pathname dependencies that we didn't realize.

We've restored the RSS access, but we're not sure we've fixed the systemic problem. We know how to fix it by hand, though, so you may see intermittent RSS access problems for the next day or so. They shouldn't last more than a few minutes when they pop up, and we should have it fixed by the end of the week.

Update: Fixed for good now. For future reference, don't store characters like linefeeds as literal strings - construct them from constants so that when you cut and paste things, helpful applications don't convert them to returns for you.

Also, just because this ought to be easier to find on the Web than it is: You can't use AppleScript's "read" and "write" commands in Standard Additions inside an AppleScript in a FileMaker Pro script step. It's because "read" and "write" conflict with FileMaker's own terminology, so trying to use them gives you a syntax error and FileMaker won't compile the script.

The workaround? Use the raw event codes. Instead of "read", use <<event rdwrread>>, and instead of "write" use <<event readwrit>>, where << and >> are chevrons (option-\ and option-| on US keyboards). Note, however, that if you try this in Script Editor or Script Debugger, it will tokenize and decompile it to the terms "read" and "write" again, so pasting it back into FileMaker will break the script again. We keep the event codes in a nearby comment so we can paste them back into the right place when updating FileMaker's version of the script. Now you know.

Update 2: Although no one outside the office should have seen it, the system to export subscription charges into our bookkeeping program didn't survive the upgrade - it relied on several global fields in FileMaker Pro, and as we found out the hard way, clients can change global fields but their changes are never ever saved to disk. FileMaker's own documentation tries really hard to say this but never quite manages to make it understandable. We had to rewrite some 10-year-old logic for keeping track of authorization batches, but with just a tiny bit of help, we got it done in a few hours.

(Note #1: I only ate two of the five. They don't taste very good, but they work.)

(Note #2: Today is apparently SysAdmin Appreciation Day. As the SysAdmin, I appreciate that most of this is now done.)

(Note #3: We're going to get a couple of really good articles out of all this, I'm told, but not about most of these details.)


# - Posted to Administrivia on 7/25/06; 9:54:30 PM

Holy cow, it's hot around here

If we're a bit slow to respond to E-mail or anything else this week, please accept our advance apologies. The picture on the right shows the current (as of this posting) appearance of the Yahoo! Weather widget for GCSF World Headquarters. It had reached 100°F before noon, and it's not expected to get anywhere below 75°F through the end of the week.

Of course, GCSF World Headquarters is air-conditioned, but our production studio has separate heating and cooling. That works out great in winter, but in the summer, it really can't keep the temperature in the studio much more than 20°F below the outside temperature - particularly if the humidity is as high as it is right now. When it starts to get above 95°F outside, we start to sweat. Right now, our internal studio temperature is registering a balmy 84°F, and it's quite close in here. You can think of many adjectives to describe your ideal production environment, but "moist" is rarely among them.

The upshot is that it's quite difficult to work in the studio for a good 8-hour stretch each day, and it's the 8-hour stretch that you'd expect we'd want to be in here working. We're managing the best we can - we don't quite expect Yahoo's predictions to come true - it's predicted 107°F and 108°F for days earlier this summer, and on those days it barely got above 104, ha!). The second picture shows Apple's weather widget for the same location at the same time, and it thinks things are a little bit cooler. We're still sweating. And wondering why those digits aren't kerned more closely in either widget. Gads.

(Until 1999, the production studio was a much smaller room that now hosts the library and servers, and with body heat and more machines in there, it could easily get above 95°F at this time of day. With just a couple of servers and no people in there, the building's A/C keeps everything in line there now.)

Send us some cool thoughts - with Apple's Q2 results coming this week and a lot of stuff already in progress, it's going to be a hot one!


# - Posted to Administrivia on 7/17/06; 5:58:24 PM

No MWJ this weekend

After coming back from the 2005-2006 MDJ Power 25 and the US 4th of July holiday, we had only two days left in the work week - and several complicated stories (10.4.7, Apple's iPod factory investigation being "complete," the stock options story, and now the whole "phoning home" thing) that just didn't get sorted out before Friday.

Our calendar had shown Apple announcing Q3 results on Wednesday (2006.07.12), but it turns out that's wrong - the announcement comes in 10 days, on 2006.07.19. We expected it a week earlier last quarter, too, and when we discovered our error, we checked Apple's investor calendar and adjusted the dates. Now the adjusted dates for July and October are still a week too early, so we're wondering if it's a Gaslight kind of deal.

Anyway, we build a week off into every month that has five weekends. We'll have to do something similar the second week of August, though, because staffers will be attending WWDC 2006. We've tried to publish MDJ and MWJ while attending the conference before, but it winds up being so exhausting that doctors get involved. We're not sure what the schedule will be that week, but we're considering a few unusual options to keep subscribers up-to-date. Stay tuned!


# - Posted to Administrivia, MWJ on 7/9/06; 12:27:30 AM

A few July notes

Just a couple of things that you might want to know:

A few subscribers have asked about seeing nothing lately on the MacJournals-Talk mailing list, a high signal-to-noise mailing list we've offered for years to paying subscribers of MDJ and MWJ. Your mail client hasn't gone haywire - the list has been down for a few weeks. We mentioned about a year ago that we intended to tie membership on the mailing list to subscription status, because, sadly, the honor system was not working: people who were not subscribers (or who dropped their subscriptions) were continuing to use the mailing list for free support, sneak peeks at issues, and all kinds of other goodies - with resources paid for by subscribers, of course.

That had to change. Alas, the code work isn't going as quickly as we'd hoped - the list server doesn't get its addresses from the subscription database. We can modify the database to send commands to the list server to add and drop subscribers, but if the two get out of sync, it will confuse everyone. (Also, that would mean we'd have to disable manual unsubscribing to avoid sync problems, and we're wary of making you get list mail until one of us gets around to changing it for you.) We plan to bring it back when we figure out how to make it work for subscribers the way we'd always intended. (We really were surprised at the amount of freeloading going on.)

Also, as you may have noticed, with all the new systems and publishing finally in place (including this blog), we've raised the price on MDJ and MWJ for the first time in seven years. Oddly enough, MWJ's new price of US$14.95 per month is the same price that MDJ cost nearly a decade ago - for about the same number of pages per week. We still think both are a bargain at twice the price, and we'll continue to try to prove that to everyone.

Coming on Wednesday: public release of the 2005-2006 MDJ Power 25!


# - Posted to Administrivia, MDJ, MWJ on 7/3/06; 5:41:58 PM

Welcome to the news blog.

We spend so much time getting MDJ and MWJ out the door each month that we rarely update our own Web site. It's a bit odd, in this day of ad-driven blogs, for a subscription-only journal to have been successfully publishing, ad-free, for nearly a decade, but here we are!

This little blog isn't much, but it makes it easier for us to post small updates about what's new with our publications, or to pass along little bits of information that everyone might like to know. There should be an RSS feed and everything. Enjoy!


# - Posted to Administrivia on 7/3/06; 3:26:44 PM

This Page was last updated: Friday, September 28, 2007 at 5:37:22 PM
Copyright 2007, GCSF, Incorporated. All rights reserved.