We have not seen any regression or recurrence of the issue, Grafana datapoints retrieval are no longer sluggish. The system has returned to full health.
Posted Apr 05, 2020 - 09:48 UTC
Monitoring
We're closely monitoring it for any signs of regression or any new issues
Posted Apr 05, 2020 - 08:54 UTC
Update
The database layer for 5s resolution have been stabilized. We're closely monitoring it for any signs of issue.
Posted Apr 05, 2020 - 08:54 UTC
Identified
We have identified the issue and are working towards stabilizing the database layer for 5s resolution.
Posted Apr 05, 2020 - 08:42 UTC
Investigating
Two our our distributed database nodes for 5s resolutions were unhealthy, and this increased the load on the rest of the database nodes resulting is slower than usual datapoints retrieval. We are working towards stabilizing the database layer for 5s resolutions. Only customers that have 5s resolution views are affected but only when viewing using a time period of 0-1 hour.