Search Degradation
Incident Report for MindTouch
Postmortem

Root Cause Analysis:  

Cloud Avg Response Time error.   

Customer Business Impact: 

Customers experienced degraded performance for several hours while we wrestled with malicious agents. 06:19 UTC –18:00 UTC. 

Customer Communication: 

CXone Expert systems were attacked, targeting various customer sites, with DDOS, fuzzing and data mining activities.  CXone Expert Engineering blocked many IPs and J3 fingerprints to stabilize the system and engaged with our AWS Support representatives to find ways to better secure our systems from these volumetric attacks.  

The attackers used several systems such as Akamai, Digital Ocean and Linode to increase their attack vector.  Systems were disrupted while we filtered out the malicious traffic and tightening our Web Application Firewall (WAF) rules.

Posted Mar 29, 2024 - 13:29 UTC

Resolved
This incident has been resolved.
Posted Mar 19, 2024 - 22:22 UTC
Update
We are continuing to monitor for any further issues.
Posted Mar 19, 2024 - 22:22 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Mar 19, 2024 - 17:44 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Mar 19, 2024 - 16:51 UTC
Investigating
Investigating
The MindTouch Engineering team is investigating reports of slow search and failed search requests.
Posted Mar 19, 2024 - 16:21 UTC
This incident affected: Search.