Telefonica Spain undertook planned work as part of a migration to new hardware. A Planned Outage Notification was not provided by Telefonica to Virgin Media O2 in the UK in error, and as a result no notification was provided to DataSIM.
The Planned Outage was undertaken and completed but failed, and comprehensive end to end testing was not undertaken and so the failure was not detected and service remained in a down state.
Telefonica fixed the issue by implementing a platform reset.
Virgin Media O2 and Telefonica are reviewing this failure cascade.
A Root Cause Analysis has been requested.