Date/Time Impacted

2022-05-19 15:31 UTC - 2022-05-19 15:34 UTC

Services Impacted

SMS API

Verify API

Messages API

Summary

Our backend authentication service experienced a configuration error resulting in customers experiencing HTTP 403 responses to impacted services. 

At 15:34 UTC our engineering team was alerted to a problem related to an increase in customer API failures. Further investigation revealed an increase in customer HTTP 403 responses, and we immediately attributed the cause to a recent software update which had been deployed at 15:28 UTC. The software update was completed at 15:34 UTC and therefore no further impact was experienced after this time.

‌

Root Cause

On May 19th at 15:28 UTC we deployed a routine software update on our platform; no customer impact was expected as part of this update. At the start of the deployment process, a manual step was missed and the upshot of this remained undetected during the deployment. Upon completion, the update appeared to be successful. However, due to missing an initial step in the deployment process, the authentication service experienced instability issues. More specifically, requests requiring authentication were incorrectly routed to services undergoing restarts (to commit the update). Ultimately, the services undergoing a restart were unable to process requests resulting in the inability to authenticate some customer API requests temporarily. 

Restoration Actions

No direct restorative actions were required; the software update was completed and therefore customer impact halted.

Next Steps

• Implement additional automation in our deployment process to reduce manual intervention
• Implement configuration to gracefully shutdown internal services processing authentication requests
• Expand our monitoring scope by alerting on elevated 4xx errors, per region
• Review the frequency of our automated testing
• Configure services to respond with a specific status code during a period of unavailability