This incident has now been resolved, the network has been stable for more than 2 hours and the data ingestion backlog has been replayed.
Apologies for the interruption.
Posted Sep 15, 2016 - 10:09 UTC
Update
The network has now stabilised and the impact to our rendering and data ingestion services has subsided.
However, we will still be monitoring the situation for some time until we are confident that these issues will not re-occur.
We will also be replaying any data dropped by our data ingestion services before resolving this incident.
Posted Sep 15, 2016 - 08:27 UTC
Monitoring
We have detected internal network connectivity issues that has caused high render response times and impacted our data ingestion services. Our provider carried out emergency maintenance on some network hardware at 06:00 UTC resulting in intermittent network disruption to some of our services. Impact to data ingestion and website access has been identified between 07:05 UTC to 07:40 UTC and is ongoing. Impact has lessened however and we are currently working to mitigate the effect of this issue. We will continue to monitor the situation until we're confident that it is resolved.