Degraded Service
Incident
Report for Vimeo
Postmortem
On June 12, from approximately 2:00 PM to 3:15 PM EDT, a widespread outage at our cloud provider caused most Vimeo services to become temporarily unavailable.
During this time, Vimeo users received generic error messages when attempting to access core aspects of the service, including upload, playback, transcoding, API, analytics, and live streaming.
Vimeo service was restored when our cloud provider resolved the issue, although residual effects continued until 4:30 PM EDT while our systems worked through large backlogs in the transcoding pipeline.
As a result of this outage, our engineering leadership is reviewing our plan for multilevel cloud failure to improve recovery time in the future.
Resolved
The upload and video processing backlog has been handled. There is still a very small percentage of jobs that need some intervention to retry which will be progressing over the coming hours. Aside from that, all systems are performing within normal parameters.
Update
New live streams from this point forward should be fully operational while archives for previous streams may require extended recovery time. Upload and video processing continues to make progress though there is still a backlog of work causing some delays.
Monitoring
Our systems continue to recover with most being completely operational. Live Streaming may still encounter issues with archiving once the stream concludes. Additionally, uploads and video processing may be delayed as the system processes the backlog.
Identified
We're starting to see recovery across the platform.
Update
We can confirm that vimeo.com, mobile, and playback are broadly affected.
Update
We are continuing to investigate this issue.
Update
We are continuing to investigate this issue.
Update
We are continuing to investigate this issue.
Investigating
We are currently investigating an issue with degraded service across various parts of Vimeo.
This incident affected: Website, On-Site Player, Embedded Player, Upload, API, and Live streaming features.