Issues with metered billing report generation
Mar 31, 15:10 UTC Resolved - This incident has been resolved. Thank you for your patience and understanding as we addressed this issue. A detailed root cause analysis will be shared as soon as it is available.
Mar 31, 15:01 UTC Monitoring - The degradation has been mitigated. We are monitoring to ensure stability.
Mar 31, 14:59 UTC Update - We have applied mitigations to a data store related to billing reports, and are seeing partial recovery to billing report generation. We continue to monitor for full recovery.
Mar 31, 14:56 UTC Update - We are seeing a high number of 500s due to timeouts across GitHub services. We are redeploying some of our core services and we expect that this allow us to recover.
Mar 31, 14:39 UTC Update - We're continuing to see high failure rates on billing report generation, and are working on mitigations for a data store related to billing reports.
Mar 31, 13:56 UTC Update - We're seeing issues related to metered billing reports, intermittently affecting metered usage graphs and reports on the billing page. We have identified an issue with a data store, and are working on mitigations.
Mar 31, 13:47 UTC Investigating - We are investigating reports of impacted performance for some GitHub services.
Discussion in the ATmosphere