All Systems Operational
Website Operational
Graph rendering Operational
Ingestion Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
System Metrics Month Week Day
www.hostedgraphite.com uptime ?
Fetching
Interface health: TCP ?
Fetching
Interface health: UDP ?
Fetching
Interface health: StatsD ?
Fetching
Interface health: HTTP API ?
Fetching
Interface health: carbon relay (pickle) ?
Fetching
Past Incidents
Dec 10, 2016

No incidents reported today.

Dec 9, 2016

No incidents reported.

Dec 8, 2016

No incidents reported.

Dec 7, 2016
Resolved - We have replayed the missing data and everything should be back to normal now.
Dec 7, 18:31 UTC
Monitoring - Due to a misconfiguration in a change rolled out to the Cloudwatch Integration metric ingestion was temporarily stopped from 16:48 UTC to 17:45 UTC. We have rolled back this change and are replaying the data which will be backfilled shortly.
Dec 7, 18:08 UTC
Dec 6, 2016

No incidents reported.

Dec 5, 2016

No incidents reported.

Dec 4, 2016

No incidents reported.

Dec 3, 2016

No incidents reported.

Dec 2, 2016
Resolved - We have reverted a previous configuration change and things are now back to normal.

Our ingestion pipeline was approximately dropping 16% of incoming traffic from 18:13 to 18:28 (UTC). The elevated render times proved to be a false alarm as the 99th percentile remain unchanged during the whole incident.

We have traced the root cause of the issue back to a configuration change made while testing improvements to our DNS automation.
Our DNS automation worked as expected but the configuration change had side effects we weren't expecting, that caused traffic from one of our load balancers to be rejected by our ingestion pipeline.
We now understand the side effects of this kind of change and we will work on adding extra safeguards around it in the future.
Dec 2, 18:44 UTC
Investigating - We're currently experiencing elevated response times and seeing reduced levels of traffic across our ingestion pipeline.
Dec 2, 18:30 UTC
Dec 1, 2016

No incidents reported.

Nov 30, 2016

No incidents reported.

Nov 29, 2016

No incidents reported.

Nov 28, 2016

No incidents reported.

Nov 27, 2016

No incidents reported.

Nov 26, 2016

No incidents reported.