Root Cause Analysis: 

A spike in PDF requests caused server latency. 

‌

Customer Business Impact: 

On 20231207 between 02:10 – 13:40 UTC all sites experienced increased latency and 50x error 

On 20231206 between 16:40 - 16:47 UTC and 16:55 – 17:01 UTC all sites experienced increased latency and 50x errors 

Recovery: 

CXone Expert Engineering banned all the IPs making more than 200 PDF requests per hour. Once the IP restriction ACLs were put in place traffic returned to normal. 

Corrective Actions: 

CXone Expert added additional metrics and monitoring around this type of behavior. Additionally, we have an item on our engineering maintenance board to implement more advanced rate limiting on a per-api-endpoint basis. This will help maintain consistent performance for various API actions.