Service Issue in Platform

Incident Report for Vonage API

Postmortem

Incident Statement - Global CloudFlare Issues

Multiple API services were impacted due to a global Cloudflare outage which occurred on 21-Jun-2022

Date/Time Impacted

Jun 21, 2022 06:27:00 GMT to Jun 21, 2022 07:42 GMT

‌

Services Impacted

Customer Dashboard

Voice API

‌

Summary

On 21-Jun-2022, Cloudflare reported a critical incident that caused multiple Vonage API services to be impacted at 6:27 AM GMT. Customers were unable to log in to the customer dashboard during the period of the incident. Already logged-in users were affected depending on their location, many were blocked but some users were able to continue with their sessions. Additionally, some sign-ups and payments during the time of the incident were also being affected.

Also during the Cloudflare outage some Voice API requests to the affected API domains would have been lost during the incident. The incident also affected us and our customers who rely on Cloudflare for callbacks.

‌

Root Cause

Cloudflare suffered an outage that affected traffic in various data centers

‌

Restoration Actions

Cloudflare proxying is disabled on affected endpoints

‌

Next Steps

Review proxy rules for all API endpoints

Posted Jun 24, 2022 - 13:26 UTC

Resolved

The problem has been resolved.

The following services were impacted from 21-June-2021 06:27 UTC until 21-June-2021 07:42 UTC.

Customer dashboard
Support Ticketing System (support@api.vonage.com)
Voice API

All services have been restored.
Please let support@api.vonage.com know if you continue to experience problems with this.
A post-mortem will be published in the next few days.

Posted Jun 21, 2022 - 14:58 UTC

Monitoring

We are seeing all services coming back to normal.

The list of services that were impacted has been confirmed as:
Customer dashboard
Support Tickting System (support@api.vonage.com)
Voice API

We will continue to monitor the service during the next few hours and post updates should anything change.

Posted Jun 21, 2022 - 07:46 UTC

Identified

We have identified the root cause of the incident as a network issue impacting multiple services, we are currently working on getting the issue fixed and the services are recovering, we will post further updates here.

Posted Jun 21, 2022 - 07:40 UTC

Investigating

Our monitoring has alerted us to a potential service issue with our platform which is currently unavailable. We are working hard to resolve the incident and restore service)

If there is a customer impact, it is likely to be affecting the following products:
Customer dashboard
Support Tickting System (support@api.vonage.com)
Voice API

We aim to provide an update on our investigation within 1 hour.

Please see this article - https://help.nexmo.com/hc/en-us/articles/360015693092-Nexmo-Incident-Handling - for an explanation of our approach to publishing incidents.

Posted Jun 21, 2022 - 06:58 UTC

This incident affected: Voice APIs (See https://goo.gl/2WoAtx for Current known problems) (Voice API) and Nexmo Dashboard.