Important Official Server Reboots & Maintenances Thread

Seems the network event is causing some problems again, be patient if you get disconnected from your favourite gameserver


Answer from Senior Administrator

Seems a very large ddos of about 40Gb/s needed some extra secureport rules to be added manually and filtered by our hosting provider.
Problem fixed

 

You may have noticed a disconnect about 1 hour ago caused by a cisco router bug fixing.

We will perform a planned maintenance of about 10 minutes in few hours and most of our dedicated servers will be offline during this time.
This is needed as we haven't completly rebooted our dedicated servers for nearly a year.

 

The planned maintenance has come to an end on all servers. Thank you for being patient, all servers are online.

 

We are having an unplanned maintenance on our US dedicated servers today due to a power failure inside of our datacenter which caused issues to the backup generator on our node.

Please be patient


EDIT: There might be a hardware issue as well so we are waiting for the hosting company answer on this. Be patient

LATER EDIT: The servers are back online for about 15 minutes already and we will monitor them. Seems the power failure might have caused some memory faults on two of our dedicated servers. If the server will start acting strange again we will let the engineers do some memory fault tests and replace the bad memory if needed.

 

Dear Community Members,


A planned maintenance will take place in few hours in order to apply some OS updates and remove a 3rd party backup driver that could be the issue of the crashes we had today. The maintenance shouldn't take long, once it starts.


In case there will still be issues once we're done, our server senior admin will run a full scan of our systems such as S.M.A.R.T scan and memory scan, but hopefully it won't be needed.


Sorry for the inconvenience and once we are done, we will reward everyone for the trouble.


Full updates can be read in this thread so please do not open new ones.

 

Maintenance currently ongoing

 

Planned Maintenance has been completed.

OS Updates have been installed (fixing a critical exploit bug), R1Soft 3rd party backup agent was upgraded to latest version.

Servers are online.

We are constantly monitoring the server in case there will be any further issues.



Thank you for your patience

 

We had a small downtime of about two minutes today due to some hosting issues.
Servers are back online



Official mail from hosting provider:

 

Planned Maintenance announced by the hosting company that will affect several of our servers will take place on 10th June 12am PDT - 1am PDT (GMT-8).

The servers will be unavailable for about 5-10 minutes at max during this time.

The purpose of this maintenance is to upgrade routers IOS and prevent the short downtime of about 2 minutes we had yesterday due to some cisco routers bugs during large attacks, which seems they were launched on the market with this error.

 

The planned maintenance did not caused any downtime as the hosting company re-routed traffic before starting so we are happy.
They successfully announced the IOS upgrade and 0 downtime during the maintenance.

 

Seems there are some issues on the hosting end so you may experience some lag during this time.
It should be fixed as fast as possible so be patient.

Fixed two minutes later.

 

The gameserver you might play on, may have suffered a 2 min downtime due to this

 

You may have noticed a mass disconnect a hour ago on some of our servers.
It was because our hosting company was updating an arbor patch for the ddos mitigation device.

The new 3 MILLION USD ddos protection equipment has been installed by the hosting company. This is a great investment they made and music to our ears when it comes to ddos protection so we can offer you the best gameplay ever!

PS: Remember that first time when connecting to the server, you may get an error or a loading screen so make sure you try it again so your IP gets whitelisted on your 2nd attempt. This is a so called TCP authentication and provides more security for the server and the players.

Let us know if you feel any issues in the next coming days.

 

Our hosting company has updated their internal network today and brought in the new L7 protection.

While rerouting the traffic to the new network, some mass disconnects have occured for some of our players. The problem was fixed few seconds later and everything is stable.

 

Our hosting company replaced some faulty core routers, network cards and is is almost done upgrading the internal network.

Until they are completly done, this may lead to random disconnects so please be patient as we want this to be done as fast as possible too.


UPDATE: Seems the issue camed from the new Arbor update so we removed it from the server and everything is fine now.
We will wait for news about this situation.

 

We had a loop issue on one of the ddos mitigation arbors today which caused some issues during an attack we had.
We hardened the rules and let the hosting company upgrade the software, therefore the reason why some of you could not connect for a couple of minutes.

The whitelist is currently reloading so in case you cannot connect, restart your modem and should be fine.

 

Some of you could had some issues today when connecting to our servers today due to a false-positive wrong interogation.
Runned couple of tests with the hosting company and it's finally fixed. If you still have any issues accessing our website or servers, send me a PM.

 

Dear Players,


Either today or tomorrow we are going to have a planned maintenance in order to optimize our servers, upgrade parts, etc. It will be done during the night and a general message will be shown on our website once we have started.


The servers will be unavailable during this maintenance period and it will take few hours.



Thank you

 

The maintenance will be held today ( Friday 29 November ).
Start hour is now known yet but should be around 2:00 PM Server Time ( US night time)

 

Maintenance Started