As of 13:58 UTC all backlogs have been processed and data from the period 12:14 UTC to 13:10 UTC is available for query as normal. All data ingested after 13:10 UTC has been processed normally. This incident is now resolved.
Posted Jul 29, 2019 - 13:59 UTC
Monitoring
As of 13:10 UTC we have resolved the source of the delays, which was a capacity issue in our aggregation layer and new data is now being processed as expected. We are now processing the backlog of datapoints from the period of 12:14 UTC to 13:10 UTC. Until the backlogs are finished processing, querying data in this time period may result in partial graphs. We will provide an update in one hour or when backlogs are finished processing.
Posted Jul 29, 2019 - 13:27 UTC
Identified
As of 12:14 UTC we are experiencing delays processing metrics at our aggregation layer. This will result in the mis-firing of alerts as well as partial renders for metrics ingested in this time period. We are currently expanding our aggregation layer to resolve these delays. No data has been lost and metrics are being ingested normally.
Posted Jul 29, 2019 - 13:03 UTC
This incident affected: Graph rendering and Alerting.