All Systems Operational
Uptime over the past 90 days. View historical uptime.
Power Outage Systems Operational
90 days ago
99.96 % uptime
Today
Power Outage Site ? Operational
90 days ago
99.93 % uptime
Today
Power Outage API Operational
90 days ago
99.98 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.

No incidents or maintenance related to this downtime.

System Metrics Month Week Day
Utilities Updating ?
Fetching
Total Electric Customers Out
Fetching
Past Incidents
Sep 28, 2022

No incidents reported today.

Sep 27, 2022
Increased web request error rates
Resolved - DNS changes have been successfully propagated to core DNS servers.
No additional issues have been found.

Issue is being closed, but we will continue to monitor server health in detail.
Feel free to reach out to support@bluefirestudios.com or join our discord discord.gg/PRrqwe7dpD if you have any questions!
Sep 27, 05:47 UTC
Update - A few API requests from customers were getting blocked by the new traffic filtering rules.
We have made some modifications to prevent authorized clients with valid API keys from get caught in the filters.

We continue to monitor server health as the DNS changes propagate.
Sep 27, 01:07 UTC
Monitoring - We implemented a few additional abnormal traffic protections, and added an additional web cluster of servers to deal with the excessive abnormal load.

We have seen two more abnormal spikes in traffic that have not resulted in additional 50x errors.

Currently we are waiting for full DNS propagation and monitoring server health.

In total there have been about 11 minutes where portions of incoming requests were being dropped by overloaded servers.
Sep 27, 00:16 UTC
Identified - We are continuing to see increased web request error rates due to extremely high abnormal traffic which is circumventing our page caching.

We are working on implementing an emergency mitigation.
Sep 26, 23:08 UTC
Sep 26, 2022
Investigating increased web request error rates
Resolved - We experienced a brief increase in web request errors for about two minutes.
This was caused by a sudden increase in traffic which was seemingly designed to circumvent our first line of page caching.

In total our servers served back 165k 503 errors, and successfully handled 302k requests, all in the two-minute span. Before this surge we were serving about 61k web requests an hour, about 1k a minute.

Our server auto-scaling for handling increase loads cannot respond that fast.

In this case there is not much we can do in the short term to prevent this type of abuse, how they circumvented the cache forcing our web servers to have to build pages for each request can't be fixed quickly, but it is now on our radar for future development.

In the meantime, we have left our web cluster at a higher than normal scaling to help handle other similar request storms if that occur.
Sep 26, 22:16 UTC
Investigating - We are investigating increased web request error rates.
Sep 26, 21:25 UTC
Sep 25, 2022

No incidents reported.

Sep 24, 2022

No incidents reported.

Sep 23, 2022

No incidents reported.

Sep 22, 2022

No incidents reported.

Sep 21, 2022

No incidents reported.

Sep 20, 2022

No incidents reported.

Sep 19, 2022

No incidents reported.

Sep 18, 2022

No incidents reported.

Sep 17, 2022

No incidents reported.

Sep 16, 2022

No incidents reported.

Sep 15, 2022

No incidents reported.

Sep 14, 2022

No incidents reported.

← Incident History Powered by Atlassian Statuspage