| Date of Event: 2025/11/18 6:30 am EST | Date Resolved: 2025/11/18 9:30 am EST | |
| Event: Blue Triangle was impacted by a Cloudflare CDN outage occurring on November 18th which impacted a large portion of the Internet. This was the most impactful outage experienced by Cloudflare in 6 years. |
| Impact: The service experienced a partial degradation affecting Portal access and data collection services, beginning sporadically at 6:30am EST. This degradation escalated into a total service outage at 8:10am EST. The Blue Triangle Disaster Recovery plan was successfully initiated at 9:00am EST, which restored primary data collection and reporting functions. Full service restoration, including a complete recovery of Cloudflare services was achieved at 9:30am EST. |
| Summary: The incident began at 6:30am EST with sporadic failures impacting Portal access and core data collection services. This instability rapidly worsened at 8:10am EST resulting in a complete block of all incoming traffic. To mitigate impact, the Disaster Recovery plan was activated at 9:00am EST, serving as the primary recovery action which immediately restored most data collection and reporting functionality. Full remediation was confirmed at 9:30am EST following the restoration of the Cloudflare CDN, which was the ultimate component which led to the traffic block. The Disaster Recovery components were reverted at 12:30PM EST after it was deemed that Cloudflare had stabilized with no further errors. |
| Resolution: The Blue Triangle Engineering team initiated an established DR plan to re-route traffic to Fastly CDN services until the Cloudflare issue was mitigated. |
| Prevention Strategy: While the established Disaster Recovery plan proved effective in restoring service, proactive traffic re-routing automation will be implemented. This automation will expedite the failover process to alternate content delivery services to reduce mean time to recovery for future incidents. |
Comments
0 comments
Please sign in to leave a comment.