MindTouch degraded performance: Slow load time
Incident Report for MindTouch
Postmortem

An orchestrated DDoS attack affected our systems on Feb 29th 2024, starting at 19:41UTC and ending 22:17 UTC.  We responded to the event and identified several IP addresses accessing data on one of our customers' public facing sites at extreme volume and rate.  Each series of requests per IP address were just under our WAF filtering volume / minute threshold so we started manually mitigating the situation. 

We were able to identify a single J3 fingerprint which allowed us to block all the inbound traffic from the suspected botnet allowing site performance to return to normal.  After monitoring the situation for another hour or so and verifying with impacted customers we closed the incident as resolved.  Further investigation showed that the J3 fingerprint continues to block traffic and is remaining in place.

Posted Mar 07, 2024 - 20:32 UTC

Resolved
This incident has been resolved.
Posted Mar 01, 2024 - 00:50 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Feb 29, 2024 - 22:34 UTC
Investigating
MindTouch Engineering is looking into issues related to slow load on MindTouch sites.
Posted Feb 29, 2024 - 21:24 UTC
This incident affected: Application (General Service), Search, In-Product Contextual Help, Email Services, MindTouch Success Center, and Analytics.