2hr Outage YesterdayJune 27, 2012
Hi folks. Yesterday we had an outage lasting a couple of hours and wanted to report back on the cause, and why it can take a bit of time to get the site running, tested, and restored. We first verified it wasn't a hacker, by locking everyone out but the admin and inspecting system files. Then we verified it wasn't a plugin (one of the special features that gets routinely updated and is the most common cause of site problems) by disabling all plugins. Then we verified it wasn't a database issue. Lastly we looked at non-plugin features that employ 3rd party servers (e.g. facebook, lulu, amazon), and discovered that was the problem. In short, yesterday servers of some major companies were affected globally, and when you feature components of their sites on your own site (a facebook gallery, a lulu reader, an amazon store), there is always the risk of your own site being hampered when their servers have major problems, as seldom as that occurs.
As of right now, we've restored most functionality, but are still remaining conservative about restoring 3rd party server feeds in case there's a 2nd wave of issues with those servers. Other than the aforementioned widgets, the site should be fully operational from a user's standpoint. We restored the main site video and sidebar video feeds within about 2hrs of the original outage. Thanks for your patience. This is not to reflect in any way on other 3rd party services like amazon, which normally has bullet proof operations. Best wishes.