False Positives for some Alerts

Incident Report for Hosted Graphite

Resolved

We have successfully failed over the database. We will be reviewing our failover process to ensure that the impact from this is reduced in future.

Posted Oct 17, 2017 - 17:57 UTC

Update

From 17:00UTC until 17:02UTC we performed another failover attempt which may have resulted in alerts triggered as malformed queries. Alerts triggered in this fashion will have recovered immediately and normal operation has been resumed.

Posted Oct 17, 2017 - 17:24 UTC

Monitoring

While performing a database failover as part of essential maintenance, alerts triggered between 13:50UTC and 13:55UTC may have been triggered as false positives - this is due to the alert queries being interpreted as a 'bad query' when the database is unavailable. Alerts triggered in this fashion will have recovered immediately after, and normal operation has been resumed.

Posted Oct 17, 2017 - 14:14 UTC