Post-mortem: Stuck Critical Jobs Queue

Between March 4th and 5th, the Open Build Service (OBS) experienced a service degradation. Impact: Users weren’t able to retrieve the diff changes of submit requests. Detection The issue was first identified by team members who noticed that diffs for new submit requests were not loading. Minutes later, it was confirmed that this issue was affecting all submit requests across the production instance. Root Cause Due to multiple factors, the latest code changes increased the...