False positives for traffic alerts

Incident Report for Hosted Graphite

Resolved

We have identified and resolved an issue that caused some traffic alerts to trigger as false positives. This only affected traffic alerts (alerts of the form "HG-Alert: Concurrent metrics above 80% of limit") and not regular alerts.

A new version of one of our services was deployed at around 16:00 UTC yesterday that introduced this issue, which was fixed at 23:10 UTC. Unfortunately the alerts triggered by this issue had entered an invalid state that needed to be fixed before the alerts could be resolved. This change has been made as of 10:45 UTC, so all traffic alerts are back to normal.

If you have received a traffic alert during this time interval it's likely that it's just a false positive.

Posted Sep 21, 2017 - 10:45 UTC