Resolved - Root Cause Analysis: 09/05/18

Customer Business Impact:
Event metadata written from 8:40 AM PST to 10:00 AM PST were not recorded for the following services:
Page History
Draft History
Site History
Site Activity
Help Requests
Search Insights
Import/Export
Page views
Indexing

This incident only affects metadata; content updates were not impacted. For example, if a user edited content on a page during the incident, the changed they made were saved properly. The record of this change, however, would not show up in the revision history of that page / content. Any content updates made before or after the incident were not affected and were recorded properly.

Problem Summary:
Event processing was disrupted for approximately 80 minutes while the event pipeline was not writing metadata to the MindTouch servers for cataloging.

Recovery:
The MindTouch event pipeline stream and associated APIs were reset as a debugging step in order to normalize service.

Chronology of Events (all times PDT):

09/05/18 [8:51 AM]
MindTouch Engineering is alerted about a change in the event pipeline’s status

09/05/18 [8:59 AM]
MindTouch Engineering is made aware that that the event pipeline was not writing metadata for events.

09/05/18 [9:31 AM]
MindTouch Engineering restarts an afflicted API service as a means of triaging the issue with positive results.

09/05/18 [9:40 AM]
MindTouch Engineering confirms that they have restarted all afflicted API services and event metadata is confirmed as being sent over.
Sep 6, 01:47 UTC