Partial renders across all resolutions

Incident Report for Hosted Graphite

Resolved

As of 16:08 UTC we are no longer experiencing partial renders. During routine upgrades to our short term storage & aggregation layer layer, we experienced an issue with the newly added hosts in which our web servers timed out attempting to read from the new hosts' short term storage. This resulted in partial renders for up to 40% of reads across all resolutions. We have removed the affected hosts and the aggregation layer is stable. This incident is now resolved.

Posted May 22, 2019 - 16:14 UTC

Investigating

As of 15:40 UTC we are experiencing partial renders across all resolutions due to an issue reading from our short term storage layer for up to 40% of reads.

Posted May 22, 2019 - 16:01 UTC

This incident affected: Graph rendering.