Invalid Loadbalancer Configuration Causing 301 Redirects on hostedgraphite.com
Incident Report for Hosted Graphite
Resolved
Between 10:23 UTC and 10:45 UTC a 301 redirect caused up to 33% of HTTP requests to hostedgraphite.com to be redirected to metricfire.com.

This resulted in up to 33% of datapoints sent over HTTP being dropped during this period, with corresponding gaps in graphs.

Users using the HostedGraphite Agent will need to restart the Agent due to a caching issue which persists the 301 redirect. Restarting the Agent will allow backfilling of any data not sent, and no data will be lost
Posted Jul 31, 2019 - 11:26 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jul 31, 2019 - 11:09 UTC
Identified
Between 10:23 UTC and 10:45 UTC a configuration change was made on our frontend load balancers causing 301 redirects from hostedgraphite.com to metricfire.com. A 301 redirect is a permanent redirect from one URL to another.

This impacted HTTP ingestion, alerts that use graphite functions and anyone accessing hostedgraphite.com would have been redirected. Your web browser caches 301 redirects and so its cache will need to be cleared in order for you to access hostedgraphite.com successfully.

Only your cache needs to be cleared. To do that, follow these instructions for your browser:
Firefox: https://support.mozilla.org/en-US/kb/how-clear-firefox-cache
Chrome: https://support.google.com/accounts/answer/32050
Posted Jul 31, 2019 - 11:09 UTC
This incident affected: Website, Ingestion, and Alerting.