Plan B? There is no plan B!

Our server at work went down... It went down hard... So hard that it's out of comission till tomorrow, when a new mobo and cpu get here. Sigh...

Why on earth did we cheep out on the super-duper HP care package warranty thingy? I really wish I we had it now - I could have a tech at my location in 4 hours flat. When you deal with a critical server, next business day service just doesn't cut it.

I hope to God that this is just a bad mobo. If the drives died I will fucking shoot someone... But since we had a RAID setup we should be ok. Both drives would have to die... Of course, with my luck that is certainly not ouf of question. So I keep my fingers crossed and pray for the best.

Of course we do have a a tape backup - but that's only data. Setting up this system from scratch will drive me up the wall.

We are pretty much dead till the new mobo is sitting in the case. This server was our domain controller and hosted our DNS server, DHCP server, SMTP server and HTTP server. This was a really bad idea to have all that shit on one machine. And we have no backup strategy, no contingency plan, no nothing. If the server goes, we are royally fucked. And it just happened.

I turned our sonicwall into a temporary DHCP for the office so at least people can use internet in the morning. That's as much as I could do today. I can't just plug in my RAID SCISI disks or the SCISI tape drive to our low end Dell office boxen... Sigh...

Once we get up and running I'm thinking about distributing the workload to more than one machine. I would really prefer to have a separate box for the webserver and another one for mailserver. We will probably need to keep the file and printer sharing, DNS and DHCP on the windows box... But there is just no reason why we should not have a dedicated webserver and mailserver in the office...

It doesn't have to be a beefed up server beast either, as we are only hosting web apps user internally by the employees. This is all low traffic stuff. Our official webpage is hosted offsite (and is crappy as hell, but it's not my pair of shoes so screw that). We can probably put in some low end Dell server to accommodate our mail and web needs - as long as we have a backup strategy for it it should be ok.

I need to run it by the boss. As it is right now our biggest issue is 50+ employees in the field trying to push their outgoing mail through our dead server, and trying to access out web apps. If we had that on a separate machine, we still would be fucked but at least it would be an internal thing - leaving all the people in the field unaffected...

I never thought I'd say that, but thank god that our POP3 server is hosted and maintained offsite by a consulting company. It is usually annoying as hell to go through them to set up new accounts and do tweaking... But at least our employees can still receive email now.

Update Wed, March 15 2006, 09:30 PM

Server is still down :( HP dispatch did not have my parts. The motherboard will be shipped tomorrow morning, but the CPU is on back order and they don't know when they will have one. This is absolutely ridiculus. If this is not fixed by tomorrow my boss will explode... HP tech support blows!

Update Thu, March 16 2006, 15:17 PM

Replacing mobo on the server did not help. The technician should be back tomorrow with a new CPU and a power supply backplane. This is day 2 of downtime, going into day 3... We have never been down so long... Fuck!


Update Fri, March 17 2006, 08:59 PM

Server is finally up! Yaaay!

Tags: server, server crash, server problems, it, it problems, downtime, contingency, backups, this sux