Post-mortem: Service Degradation in OBS

Between February 15th and 18th, the Open Build Service (OBS) experienced intermittent service degradations. Users experienced sporadic latency across various workflows (with delayed jobs involved), leading to periods of total service unavailability for most users. Those were the consequences of a huge traffic spike. We want to give you some insight into what happened and what we plan to do to handle similar circumstances in the future more efficiently. Detection The dashboards on BuildOps monitoring...