Root Cause Analysis: 09/05/18
Customer Business Impact:
Event metadata written from 8:40 AM PST to 10:00 AM PST were not recorded for the following services:
This incident only affects metadata; content updates were not impacted. For example, if a user edited content on a page during the incident, the changed they made were saved properly. The record of this change, however, would not show up in the revision history of that page / content. Any content updates made before or after the incident were not affected and were recorded properly.
Event processing was disrupted for approximately 80 minutes while the event pipeline was not writing metadata to the MindTouch servers for cataloging.
The MindTouch event pipeline stream and associated APIs were reset as a debugging step in order to normalize service.
Chronology of Events (all times PDT):
09/05/18 [8:51 AM]
MindTouch Engineering is alerted about a change in the event pipeline’s status
09/05/18 [8:59 AM]
MindTouch Engineering is made aware that that the event pipeline was not writing metadata for events.
09/05/18 [9:31 AM]
MindTouch Engineering restarts an afflicted API service as a means of triaging the issue with positive results.
09/05/18 [9:40 AM]
MindTouch Engineering confirms that they have restarted all afflicted API services and event metadata is confirmed as being sent over.
Sep 6, 01:47 UTC
MindTouch Engineering is investigating an issue with reporting data not being available in real time. We will update this page as information becomes available.
Sep 5, 18:46 UTC