Hi everyone - long time no speak.
As I'm sure you've all noticed, we've run into a bout of server issues over the last week that has lead to downtime of our beloved forum.
First, I want to thank you for your patience during this annoyance - nobody likes downtime.
Next, and more importantly, I'd like to fill you in on what has happened over the last week and what will be happening in the short term.
Each day when it'd go offline again, we'd troubleshoot it more and rule out more and more possible causes of the errors. As you saw, the system would stay online for about 24hrs and seem to work perfectly before halting again.
The cause has now been diagnosed: the motherboard that hosted our beefy server ran into hardware issues. Specifically the northbridge was affected by heat which lead to sporadic errors (MachineCheckException Bus Interconnect errors)
VMWare's ESXi system is very resilient to these types of errors in small amounts, but as time ticked on, more and more of these errors piled up. After about 24hrs when ESXi logged a certain amount of these errors over its threshold, it halted operation as it was clear something was wrong with the hardware.
The good news is - the server has been replaced with a temporary server and all of our VMs have been migrated over. This temporary server is slower (so you'll notice the forum will behave more slowly than you're used to) but it works. A new beefy server has already been ordered and is on its way. When it arrives we will be taking this temporary server down for maintenance to replace it which should restore the forum to its normal operating speeds - if not a little faster.
Again - I want to thank you for your patience while we troubleshooted the issue and finally identified the cause. We hope to have the new server online within the next couple of weeks to bring everything back to normal operating conditions.
Have fun and keep Allegiancing,
--TE
Server Woes Resolved
Need cash? THROW UP A DONATION BUTTON. I'm in to help TE build a badder system for 50 vs 50 Games. I'll comm first one if the community hits $2000 in one day.
PS... I'm recruiting for a new Squad. @Element.
Thallium@Element.
Whos next?
PS... I'm recruiting for a new Squad. @Element.
Thallium@Element.
Whos next?
Tigereye wrote:QUOTE (Tigereye @ May 23 2014, 05:07 PM) Hi everyone - long time no speak.
As I'm sure you've all noticed, we've run into a bout of server issues over the last week that has lead to downtime of our beloved forum.
First, I want to thank you for your patience during this annoyance - nobody likes downtime.
Next, and more importantly, I'd like to fill you in on what has happened over the last week and what will be happening in the short term.
Each day when it'd go offline again, we'd troubleshoot it more and rule out more and more possible causes of the errors. As you saw, the system would stay online for about 24hrs and seem to work perfectly before halting again.
The cause has now been diagnosed: the motherboard that hosted our beefy server ran into hardware issues. Specifically the northbridge was affected by heat which lead to sporadic errors (MachineCheckException Bus Interconnect errors)
VMWare's ESXi system is very resilient to these types of errors in small amounts, but as time ticked on, more and more of these errors piled up. After about 24hrs when ESXi logged a certain amount of these errors over its threshold, it halted operation as it was clear something was wrong with the hardware.
The good news is - the server has been replaced with a temporary server and all of our VMs have been migrated over. This temporary server is slower (so you'll notice the forum will behave more slowly than you're used to) but it works. A new beefy server has already been ordered and is on its way. When it arrives we will be taking this temporary server down for maintenance to replace it which should restore the forum to its normal operating speeds - if not a little faster.
Again - I want to thank you for your patience while we troubleshooted the issue and finally identified the cause. We hope to have the new server online within the next couple of weeks to bring everything back to normal operating conditions.
Have fun and keep Allegiancing,
--TE
Last edited by Thallium on Fri May 23, 2014 9:21 pm, edited 1 time in total.
I don't understand any of this Thal...Thallium wrote:QUOTE (Thallium @ May 23 2014, 02:20 PM) Need cash? THROW UP A DONATION BUTTON. I'm in to help TE build a badder system for 50 vs 50 Games. I'll comm first one if the community hits $2000 in one day.
PS... I'm recruiting for a new Squad. @Element.
Thallium@Element.
Whos next?
There is a donation button on the main community page. http://www.freeallegiance.org/forums/index.php?act=home
And I don't understand how giving more money to TE will help build a system for 50v50 games.. TE isnt involved in growing the game, just paying for the server.
And you are making a new squad!????
Also why do you only have 13 posts?
Literally everything he posted in the last day was based on false precedents and therefor nonsensical and wrong.
I think he deleted all his other posts.
Go home Thallium, you're drunk.
I think he deleted all his other posts.
Go home Thallium, you're drunk.
Last edited by Mastametz on Sat May 24, 2014 10:02 am, edited 1 time in total.
There's a new sheriff in town.
