Delayed Metric Processing

Incident Report for Hosted Graphite

Resolved

At 00:30 UTC, we have worked through the backlog of metrics from the affected period and all services are fully operational again.

Posted Jun 09, 2019 - 00:38 UTC

Update

As of 23:41 UTC, the network latency has subsided and we're working through the backlog of metrics from the affected period. We will update again once the backlog has been processed.

Posted Jun 08, 2019 - 23:52 UTC

Identified

From 23:02 UTC, an increase in latency between our hosts has lead to a delay in processing metrics. This may cause missing metric alerts, graph rendering and PNG generation timeouts as well as an increase in response time for the website and any renders.

Posted Jun 08, 2019 - 23:33 UTC

This incident affected: Website, Graph rendering, Ingestion, and Alerting.