2022-05-19 15:31 UTC - 2022-05-19 15:34 UTC
Our backend authentication service experienced a configuration error resulting in customers experiencing HTTP 403 responses to impacted services.
At 15:34 UTC our engineering team was alerted to a problem related to an increase in customer API failures. Further investigation revealed an increase in customer HTTP 403 responses, and we immediately attributed the cause to a recent software update which had been deployed at 15:28 UTC. The software update was completed at 15:34 UTC and therefore no further impact was experienced after this time.
On May 19th at 15:28 UTC we deployed a routine software update on our platform; no customer impact was expected as part of this update. At the start of the deployment process, a manual step was missed and the upshot of this remained undetected during the deployment. Upon completion, the update appeared to be successful. However, due to missing an initial step in the deployment process, the authentication service experienced instability issues. More specifically, requests requiring authentication were incorrectly routed to services undergoing restarts (to commit the update). Ultimately, the services undergoing a restart were unable to process requests resulting in the inability to authenticate some customer API requests temporarily.
No direct restorative actions were required; the software update was completed and therefore customer impact halted.