On September 2, 2023 at 7:00 UTC one of the primary servers of our database processing cluster failed. The system has failed to properly engage a failover server, resulting in EMX service interruption for our customers. Our primary, distributed data collection systems were not affected and no data loss has been detected.
We have since performed an audit of the systems in question and identified the root cause of the failure as a misconfiguration of log space allocation by a third party responsible for server cluster maintenance. We have replaced the affected system with a significantly more robust, powerful and reliable AWS Aurora cluster and are in the process of migrating all data to the new system. We expect the process to continue over the next 48-96 hours, with additional data becoming available on-line during that time period, until all data is restored.
We sincerely apologize for the inconvenience - our team has been working around the clock through the holiday weekend to restore full access as soon as possible.
