This doesn’t affect this particular site at all, but in case Dad or Kirsten check in, there seem to be issues with the djwudi.com/hanscomfamily.com/geekmuffin.com/interalia.org server at the moment. Phil and I are trying to work on it and find out what the issue is — whatever it is, it popped up sometime during the night, when neither he nor I were doing anything with the machine. No current ETA on when we’ll be back up and running, though we’re trying to bring it back up as soon as possible.

Update: Still down, still no ETA, and I need to get out of the house for a bit. I’ll get more info up when I can.

This also means that any e-mail sent to my @michaelhanscom.com address will not get to me at the moment. Please use one of my other e-mail addresses if you have one of them and need to get ahold of me.

Update: Everything’s back up again. Phil tracked it down to a cron job that had piled a few thousand e-mail messages into his inbox, which caused everything to choke. Inbox is empty, cron is smacked back into submission, and all’s well that ends well.

iTunes: “Three Wishes” by Waters, Roger from the album Amused to Death (1992, 6:50). Just so I don’t lose track or forget about what we tried so far, a quick copy-and-paste of the work so far:

Phil: el problemo, senor

Michael: what’s up? Phil: i think something has run awry in my crontab Phil: and is chewing up way too many resources Phil: on the webserver. Michael: aah, is that what’s going on Michael: i just had to restart it - it wasn’t responding to _anything_ when I woke up, except pings Phil: i suspect you’ll find tons of sa-learn processes Michael: no ssh or httpd calls would answer Michael: is it down again now? Phil: i got ssh to work verrrrrry slowly. Phil: No responses again now. Michael: hrm Michael: well, yeah, something’s borked - the UI is frozen Michael: gonna restart it again Phil: gah. Phil: let me know when you restart it, i’ll ssh in and nuke my crontab before anything launches. Michael: k Michael: okay, login window is up Michael: give it a shot Phil: i’m in Phil: gonna watch top and see what launches. Michael: :nods Phil: other than a lot of sendmail and procmail processes, nothing seems to be wigging out. Michael: took out whatever it was that was having an issue? or it just doesn’t seem to be “issuing” right now? Phil: i took out what i -think- was causing it. no idea if that was it or not. Phil: but i recall something similar happening last time i tried to make a cron job for sa-learn. Michael: okay Michael: what’s sa-learn? Phil: SpamAssassin’s Bayesian filter learning tool. Phil: It’s the “learn this now, dammit” tool. Michael: ah, okay Phil: i’m still getting some false negatives that it should’ve caught. Phil: um. Phil: i don’t think that was it. i’m not getting any responses on ssh again. Michael: yeah - the UI’s frozen again Michael: wierd Phil: goddamnit. what is going on? Phil: i know i haven’t actually changed anything… Michael: maybe it’s the machine? (though I _hate_ to suggest that) Michael: restarting again Phil: well, we’ll see; i sincerely hope that’s not it. Michael: you and me both Michael: (and kirsten and dad too, I’d bet) Phil: is it acting funny? fans any louder than usual? Michael: doesn’t seem to be Michael: i just sits in the corner and hums happily to itself (shrugs) Michael: as opposed to Marvin, who sits in the corner and hums dismally to himself Phil: no, i’d wager something is getting overloaded. Phil: i don’t know what, though. hmmmm. Michael: could I be getting hit by an attack of some sort, I wonder? Phil: hmmm. Michael: god_damn_ that thing takes a while to start up…lol Phil: there’s a way to test that. Michael: it’d have to be a pretty good attack to completely lock it up, though Phil: during the brief time i can get on i’ll try to install snort. Michael: it’s up Phil: can’t get it to respond to ssh even now. Michael: huh Phil: n/m Phil: there we go Michael: okay Phil: sshd probably wasn’t up yet Michael: i’m logged in, have top -u 15 running in a term window Phil: should see a wget process. Michael: if it locks up, hopefully it’ll give me an idea of what the issue might be (maybe) Phil: hopefully. Michael: right now it’s mostly just top at the top of top (whee) Michael: i’ve seen ssh, gzip, and tar pop up occasionally, but the drop right back down again Phil: those were me. Michael: i’ve got my eye on you, young man…lol Michael: imapd just hit thetop - bouncing between 10 and 25% Phil: probably mail trying to connect. Michael: and…freeze Michael: dammit Michael: the last thing at the top was just imapd at only 10% Phil: yeah, i noticed. hmm. Phil: i’m going to nuke my entire crontab. nothing should be launching, but that doesn’t mean nothing is. Michael: i’m pretty sure mine’s empty…very sure, in fact Michael: okies, restarting Phil: and /etc/crontab just has system stuff in it. Michael: back up Phil: yep. i’m in. Phil: ah HA Phil: apache2 is starting. Michael: ? Michael: i was about to run a permissions check/repair, just for grins and giggles… Michael: oh, wait Phil: if you still want to, go for it Michael: apache2? Michael: didn’t you kill that? Phil: yes. Phil: precisely my point. i thought i did too. Michael: hmm Michael: i wonder why it’s starting…and why it hasn’t been a problem until now Michael: wow Phil: OK, I need to go find the startup item for apache2 and slay it. Michael: it’s fixing a _lot_ of permissions Phil: oh man, wait Phil: that was NOT a good idea. Michael: uhoh Phil: any custom permissions I had set on my website are gonna be borked Michael: oh shit Phil: well, we’ll find out soon enough. Michael: it seemed to be doing a lot in the /System/Library/Perl/ directory Michael: trying to stop it now Michael: spinning rainbow cursor at the moment Phil: frozen again Phil: from CL too Phil: so apparently apache2 is not the problem, but it is a problem. Michael: grrr Michael: hmmm - the UI isn’t _completely_ frozen Michael: if I move the mouse, the cursor will jump to another point on the screen after a few seconds Michael: still, it’s quite unuseable Phil: probably the permissions check is taking a lot of cpu time. Michael: gonna reboot again Michael: christ, this is annoying…lol Phil: tell me about it. Phil: i want my email! Michael: lol Michael: if all else fails, there’s always the possibility of a reinstall Michael: which, admittedly, could suck Michael: system install, getting MT up and running again, making sure the websites don’t break, etc. etc., yadda yadda Phil: Do an archive & install, then. Phil: (If you have the disk space.) Michael: :nods definitely Michael: i do, there’s still gigs free Michael: not the way I was planning on spending my saturday, though Phil: nor I. Michael: if I do have to do that…think it’s worth putting panther on, rather than jaguar? Michael: i’d have to figure out the new mail system (postfix instead of sendmail, right?) Michael: hm Phil: if only for the fact that you’d have to configure postfix, i’m gonna say no Michael: ah Michael: okay Michael: yeah - “starting apache 2 web server” is part of the startup window messages Michael: and, we’re up again Michael: holy shit - we’re locked up again Michael: okay, the fact that that’s getting faster does not bode well Phil: i noticed. Michael: goddammit Michael: y’know, if it was cloudy, i might not mind as much, but it’s a gorgeous day out there…lol Michael: up Michael: according to the apache2 control panel, it’s not running Michael: even though the startup message was in the startup routine Phil: ridiculous. Michael: fuck - locked up Michael: well, locking up Michael: in the process Michael: tried to load the sharing control panel Michael: spinning rainbow cursor of death Phil: this is bloody odd. Phil: something has gone horribly wrong between yesterday afternoon and today. Phil: i haven’t been around to do anything. Michael: and i haven’t touched it Phil: when it starts up, go to the apple menu, check about this mac, and see if anything on the amount of RAM or CPU speed looks unusual. Michael: up Michael: 640Mb ram…trying to remember if that’s right Phil: yes. Michael: yeah, that’s right Phil: maxed out for this architecture Michael: lol Michael: brainless Michael: hm Michael: console log has some conflicts listed Phil: do tell. Michael: login window could not find image named ‘bang’ Phil: ….? Michael: :shrugs Michael: some startup items failed to launch due to conflicts Michael: dammit…do I have ichat on that machine? lol Phil: probably. Michael: fuck, not that it matters Michael: i’m losing the UI again Michael: it just keeps slogging down, slower and slower Michael: until it doesn’t respond Phil: i noticed…. so weird. Phil: incidentally, the apache2 item should now start apache Michael: lol okay Michael: it’s working, it’s just responding like XP installed on a 286 Michael: making glaciers look positively snappy, in other words Michael: i don’t get it Michael: okay - restarting it, but i’ve unplugged it from the network Michael: going to see if that makes a difference at all (though I’m not sure why it would) Phil: we’ll find out. Phil: if that does it, you’re getting DOS’ed. Michael: not thinking it’s a DOS or a hack or anything, simply because it’s behind my firewall, so huge amounts of traffic on my line should affect my connectivity/speed on this ‘puter, too Phil: not if they specifically attacked the webserver by IP address or name. Phil: well, some slowness, maybe. Phil: but the webserver would see the majority of it. Phil: …your windows box isn’t on the network, is it? Michael: nope Michael: well, yes, but it’s off Michael: and on the few times I start it up, the first thing I do is apply any recent security updates Michael: it’s as up to date as it can be without doing a daily security update check Michael: okay, this isn’t good Phil: eh, if it’s off, it doesn’t concern us. Phil: what? Michael: the box doesn’t seem to be restarting Phil: … Michael: i’m going to unplug it and let it sit for a few minutes Phil: well, that’s not good. Michael: given that this started happening when neither of us were doing anything on the box, and the fact that it seems to be a gradual thing - works fine for a bit, then gradually gets slower - and that it seems to be happening sooner on each reboot, and that it doesn’t want to reboot now… Michael: i’m really starting to worry that it’s a hardware issue Phil: that’s not good. Michael: which, realistically, is a possibility - this little G3 was used when I got it, and it’s been running as a server 24/7 for, oh, probably around four or five years now Michael: the three years since i’ve been in seattle, plus a year or two in anchorage Michael: it would _suck_ if that’s the case, though Michael: i don’t have the funds for a replacement Michael: and i could move things onto the G5, but i’ve liked having a server separate from my work machine Phil: Yeah, I can definitely understand that… I was gonna ask about that. Michael: it’s an option, though, should the g3 not come back Phil: Do you still have the hardware diagnostic CD somewhere for the g3? Michael: i don’t think I ever had one Phil: must be a new thing. Michael: :nods Michael: either that, or I just didn’t get it with the machine Phil: well, we have an hour and twelve minutes to figure something out. Michael: 1:12? lol Phil: I’m going out to the Irish festival at that point. Michael: aah Michael: yeah, I want to get out of the house at some point Michael: shame to waste a perfectly good kilt day Phil: amen. if I had a utilikilt i’d wear it today. Michael: you should do it, man Michael: not that you’d have it today, but still Phil: yeah, i know. well, hey, my birthday’s coming up in a few months… we’ll see Michael: excellent Michael: okay, booting Phil: OK, it freezes up, you need to restart in single-user mode and run fsck to see if there are disk errors. Phil: http://www.macosxhints.com/article.php?story=2004011205473937&query=fsck Michael: and I do that…how, again? lol Phil: click the link Michael: I did Michael: The specific sequence of how to do this varies depending on what version of OS X you’re running. Go to Apple Support for specific instuctions. Phil: it’s still detached from the network, right? Michael: yeah Phil: OK, let me find something that doesn’t suck i just noticed “check/fix the filesystem” Michael: lol Phil: i think you need to hold down S while booting. Michael: gaaa…doesn’t seem to be booting Michael: got the startup chime, nothing else Phil: oh, maaaaan. that’s not good. Phil: actually, though. Phil: that might be a sign of a disk error caused by the forced reboots. Phil: we had the exact same thing happen with a G4 at work. Michael: :nods it got in last time, and I did a manual (correct) restart Phil: iMovie froze, the finder crashed and wouldn’t restart, and everything rainbowed. Phil: I’m going to suggest an archive and install. Michael: :nods that’s what i’m thinking (grr) Michael: i’ll deal with that later, though Phil: just remember to hold down C as you turn it on with the jaguar disk in the driver (or panther; if you want, i can look up postfix guides while you wait) Phil: k Michael: right now…i’m a little annoyed, hungry, and want outside lol Michael: i may do panther…it’s got enough under-the-hood upgrades that it might be a good idea Michael: just take a little poking around to get postfix up Phil: and really, from what i recall, setting up postfix wasn’t hard at all. Phil: i did it at WWDC the day i installed panther. Michael: as long as the websites survive, i’ll be satisfied Phil: i think the config files made more sense to human beings Michael: that’s my big worry, really Michael: i’d hate to end up nuking everyone’s sites (again) Phil: i do too. y’know, if all else fails, stick the HD into the G5. I assume it has an expansion slot for a second HD. Michael: yeah, but I can’t just drop the drive in Michael: different drive specs Michael: serial ATA on the G5 Phil: oh, fuck, that’s right. dammit. Michael: i’d need to drop the old drive(s) into ATA cases Phil: if nothing else, even if the machine won’t boot, our data is still there. Michael: yeah Phil: you’re good, webmaster. go get food and sunshine; i’ll be doing the same myself shortly. Michael: fuck - and i’m using michaelhanscom.com for most of my e-mail right now, too…lol Michael: argh Michael: ah, well Michael: it’ll wait for a day Michael: or a few hours, or whatever Michael: anyway…i’m wandering - enjoy the irishfest Phil: will do. later man

<

p>

[See also: We all have our priorities | We’re Not Gonna Take It | Bugfixes are good | The MovableType/Mac conspiracy… | 2 hours in the life of a ‘puter geek ]

« KMFDM at the Fenix Underground | Vote »