During this time, some SMS delivered to the United Kingdom experienced increased latency.
2022/06/17 10:12:58 UTC to 2022/06/20 16:08:00 UTC
SMS MT to United Kingdom
On 2022/06/17 a routine update was made to our Carrier Lookup Service (CLS) (used to identify the operator of a phone number) that is used for the United Kingdom routing. Everything tested fine after the change. Subsequently, an issue developed with this supplier that caused lookups to timeout after 20 seconds of no response from our supplier on a subset of UK mobile numbers. This resulted in delays of ~20-23 seconds of extra delivery delay on these MT to UK mobile numbers.
The issue was detected on 2022/06/20 13:19:00 UTC via customer notification of delay of SMS being delivered and we discovered our alerting mechanisms did not pick up the increased latency and flag to our technical teams as it was designed to. This was raised as an internal incident and investigated further.
At 16:08:00 UTC a fix was put in place to resolve it and mitigate all customer impact.
There was unexpected behaviour in our CLS supplier. This resulted in a number of requests that timed out before failing over to our internal numbering plan. As a result, our system was delayed for 20 seconds before sending out the SMS as expected.
Upon further investigation we did not have the proper alerting in place for reasonable increased latency alerts.
At 16:08:00 UTC we removed the CLS provider that was giving us increased latency and moved back to a different provider to restore service.