Alerting Service Down

Incident Report for Hosted Graphite

Resolved

We have restored the alerting service and normal operation has resumed. However, some alerts triggered between 17:50 UTC and 18:25 UTC will not have been processed.

Posted Jun 14, 2017 - 18:49 UTC

Monitoring

We have performed the rollback as of 18:25 UTC and alerts are beginning to be processed and sent again. We are monitoring the situation as recovery continues.

Posted Jun 14, 2017 - 18:30 UTC

Identified

We've identified a problem with our alerting caused by a change rolled out to fix issues experienced last night and are currently rolling back this change - as of 17:50 UTC some alerts will not be sent or processed until this change is fully rolled back.

Posted Jun 14, 2017 - 18:15 UTC