Where I work, I have to be on call two weeks out of four. This means that when I leave the office, I have to make sure that I’m carrying both phones (personal and work) and have the pager (yes, you remember those don’t you?) and that everything has a full charge, just in case.
Most of the time it’s quite a routine thing and things are relatively quiet (apart from the Great Big F*** Ups) with the occasional page in the middle of the night as one of our systems decides to throw a bit of a hissy fit. These hissy fits are usually short-lived (a page to say something’s broken, followed immediately by another page to say it’s fixed). I’m sure the pager has a sense of humour as it’s usually 02:00 or 03:00 which is when this happens:
03:00 “xxxxx has failed”
03:01 “xxxx is restored”
The subtext of this is:
“Just checking that you’re awake and paying attention”
We have tried everything we can to prevent these non-critical pages as there is a danger of becoming complacent. Then this happens:
annoying (and loud) pager tones
Phhhh! Bloody pager!
Hit the button, squint at the display in an attempt to read the tiny writing (I have to squint as I don’t wear my glasses when I go to sleep, funnily enough) then you swear, roll over and go back to sleep
Whereas the “proper” response should be:
annoying (and loud) pager tones
Bloody hell – what’s broken?
Panic slightly until you’ve woken up properly, which actually only takes a matter of seconds when this sort of thing doesn’t happen several times a week. I’m very lucky in this regard. There seems to be a switch in my brain which clicks as soon as the pager sounds or the relevant alert tone comes in on the mobile phone and I’m 100% awake straight away.
Notice the careful use of the word “awake”. This is not to imply that I am, in any way, fully compos mentis immediately. It would be totally unrealistic to expect that but I’ve fooled many a person in the past 🙂
So, when the pager went off at 03:07 this morning, I only half woke up – it’s been giving a lot of bogus alerts lately. So I waited for all of 2 mins before realising that only the “fail” message had arrived.
I put on my glasses, just as another page came in, indicating the problem may be more serious than the usual. By the time I’d got on my dressing gown (it’s cold at 3am you know and I’m not as hardy as I once was) another couple of pages had arrived and I was hotfooting it downstairs to my laptop.
So I was in the office at 04:00 today. By 04:30 my boss was in the office too and we were trying to get our UPS working again which had decided it had enough and blown out a circuit. So we lost all our routers, servers and everyfink. Nice.
At least the coffee machine doesn’t take long to warm up …
 Great Big F*** Ups (or GBFU) refers to multiple incidents beyond our control which have seriously screwed up our service, resulting in the pager having an apopletic fit while it tried to send out the multiple failure messages and the phone almost melted from the amount of alert SMS it was receiving. Not interesting enough to write a blog about. Unfortunately. Unless you’re a geek. Which I sort of am, but realise that most folks aren’t and I don’t want to send anyone to sleep with boredom 🙂