View topic - Website outage

Page 1 of 1

Website outage

PostPosted: Mon Feb 17, 2014 1:54 pm
by GregS
I would like to apologize to everyone who was affected by last week's outage of tuc.org. The hosting company is moving to a new, improved data centre, but in the process of moving this server, it seems that something went wrong, because when it was powered up at the new location the hard drive was slow to respond. It took some time to convince them that it was a real problem, then it took longer than it should to make a copy of the hard drive, then the new drive wouldn't boot. They started a second copy of the drive, and that was taking even longer, so at my request they cancelled that and instead set up a fresh copy of the operating system, which they then restored the off-site backup onto. But the guy doing that followed the wrong procedure the first time around, adding more downtime, and then when it was finally done, it turns out that they neglected to refresh the off-site backup from the old drive as I had requested, so some data was lost. And along the way there were various delays where I'd finally have to call them to see what the status was and remind them to get going on the next thing.

I'm surprised that they had the problems that they did, as their support to date has always been first-rate and prompt. They are, as I said, in the midst of physically moving all of their servers, which is going to take a lot of their time, and their staff is physically split between the two facilities right now, which caused other issues, and there was a major snow storm there which prevented the guy that best knew how to solve the final problem from just going directly to the server and doing what needed to be done (he was in the old facility). But when a client has been down for a couple of days, you really need to prioritize them and pay close attention to get them back up ASAP, and that didn't happen here.

I have now gained access to the old hard drive, and restored some of the key lost data, specifically emails, registrations and BBS posts that happened during the window between the last backup and when the site was finally taken down. I can also look at score submissions and schedules, but it seems likely that the easiest way to "recover" those is just for captains to re-submit the scores and conveners to recreate the schedules. If there are other things you find missing, please let me know (email webmaster@tuc.org is probably best), and I'll see what I can do.

Thank you for your patience and understanding through this time!

PostPosted: Mon Feb 17, 2014 2:43 pm
by Edk001
I read the second word in your subject title as "outrage". Glad nobody was outraged at the outage.

PostPosted: Mon Feb 17, 2014 5:08 pm
by GregS
Edk001 wrote:I read the second word in your subject title as "outrage". Glad nobody was outraged at the outage.

I'm sure some were, but it's not like they could post on the BBS about it at the time...

PostPosted: Tue Feb 18, 2014 2:47 pm
by DjDATZ
That sounds like an ordeal and a half! Glad you got it sorted out and the site's back up!

PostPosted: Wed Feb 19, 2014 9:13 pm
by larrypmac
Is everything up? Because my game tomorrow night are not yet scheduled, so I wondered if my volunteer commissioner was unable to get in and set it. This is the Thursday Monarch hat league.

PostPosted: Thu Feb 20, 2014 12:20 am
by larrypmac
larrypmac wrote:Is everything up? Because my game tomorrow night are not yet scheduled, so I wondered if my volunteer commissioner was unable to get in and set it. This is the Thursday Monarch hat league.


Must be up now, schedule finally posted.