Registry Incident Report

Before we get into this blog we would like to sincerely apologize for the recent troubles that all of you encountered. This is on us and there's no excuse for this and for that we apologize. We have identified the issue and implemented a permanent fix.

What Happened

A few weeks ago we started having performance  issues with our registry server. The current solution we had worked great until recently when we started to run into performance issues. Our server simply could not handle the amount of data flowing from the registry server -> every server, development instance, etc. Our registry is accessed  every-time a server is turned on. We also use registry for development and other projects we are working on so this means it's mission critical for us.

Resolution  

After recent incidents with performance and the registry server going offline at random times. We have migrated it to a brand new cluster This is a high available cluster, with storage attached using s3. It's performant, fast, and finally resilient. It can withstand downtime on a node, and continue running just like our game panel. Again We want to issue an apology for the recent issues. This will not occur again. We thank you your patience regarding these high priority issue. If you have questions or concerns feel free to reach out to us via ticket or Discord.