DNS changes have been successfully propagated to core DNS servers. No additional issues have been found.
Issue is being closed, but we will continue to monitor server health in detail. Feel free to reach out to support@bluefirestudios.com or join our discord discord.gg/PRrqwe7dpD if you have any questions!
Posted Sep 27, 2022 - 05:47 UTC
Update
A few API requests from customers were getting blocked by the new traffic filtering rules. We have made some modifications to prevent authorized clients with valid API keys from get caught in the filters.
We continue to monitor server health as the DNS changes propagate.
Posted Sep 27, 2022 - 01:07 UTC
Monitoring
We implemented a few additional abnormal traffic protections, and added an additional web cluster of servers to deal with the excessive abnormal load.
We have seen two more abnormal spikes in traffic that have not resulted in additional 50x errors.
Currently we are waiting for full DNS propagation and monitoring server health.
In total there have been about 11 minutes where portions of incoming requests were being dropped by overloaded servers.
Posted Sep 27, 2022 - 00:16 UTC
Identified
We are continuing to see increased web request error rates due to extremely high abnormal traffic which is circumventing our page caching.
We are working on implementing an emergency mitigation.
Posted Sep 26, 2022 - 23:08 UTC
This incident affected: Power Outage Systems (Power Outage Site, Power Outage API).