Backend Outage
Incident Report for hull
Postmortem

On Tuesday January 15, Hull data pipeline stopped processing messages and the segment builder became unavailable for 58 minutes between 22:11 pm and 23:09pm UTC. Data ingestion was not affected by this outage and no data loss should have occured.

This incident was caused by the failure of an Elasticsearch cluster used for segmentation in the Hull data pipeline. The cluster itself was brought down by a StackOverflowException caused by a problematic query. This is a known issue in the version of Lucene used by that ElasticSearch cluster.

Our engineering team is working on a mitigation plan to make sure this ElasticSearch cluster is protected against that particular issue.

Posted Jan 15, 2020 - 08:42 EST

Resolved
This incident has been resolved, all systems operational
Posted Jan 14, 2020 - 19:53 EST
Monitoring
A fix has been implemented and we're monitoring the results.
Posted Jan 14, 2020 - 18:29 EST
Investigating
We're currently experiencing issues with our backend database that may affect latency on the platform as well as accessibility of the data through the UI. We are investigating the issue now
Posted Jan 14, 2020 - 17:38 EST
This incident affected: Hull Core Platform (Hull API).