Skip to content
MONTEGRE

Microsoft says testing flaw and traffic surge caused global 365 outage

Microsoft's preliminary report blames a latent defect, mismatched test environment, and unrelated traffic spike for last week's widespread service disruption.

Sources: Computerworld Australia, Network World, TheStreet3 sources ↓|· 1 min read
Microsoft says testing flaw and traffic surge caused global 365 outage
Photo: Computerworld Australia

KEY POINTS

  • Microsoft issued a preliminary post-incident report on Monday
  • The test environment did not match actual production infrastructure
  • A fix for Microsoft 365 networking gateway servers contained a latent defect
  • The change was deployed worldwide for more than three months without impact
  • Nearly 14 percent of networking gateway server capacity was out of service before the outage

Microsoft released a preliminary post-incident report on Monday explaining last week's roughly day-long global outage of many Microsoft 365 services. The company admitted that the environment used to test the problematic software update did not match the actual infrastructure where it would run.

A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions. The defect resulted from a fix prepared specifically for Microsoft 365 networking gateway servers, and its deployment began in July.

“A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions.”

— Microsoft's preliminary post-incident report

For more than three months, the change was deployed worldwide without impact. Microsoft said its normal load testing did not detect the latent issue because the traffic and SKU mix used in its test labs differed from production.

The contention issue worsened on machines with more cores and newer hardware, while load testing tended to focus on weaker hardware SKUs with fewer cores. A surge in traffic caused by unrelated issues also helped trigger the problem and delayed its resolution.

Approximately fifteen hours before the outage, Teams failed over from the alternative networking gateway to the primary gateway that also supports Exchange Online. This increased the load on the networking gateway and changed its traffic patterns.

Nearly 14 percent of server capacity for the networking gateway was out of service due to an unrelated issue from the previous day. Microsoft did not provide details of that unrelated issue.

COMMENTS (0)

0/2000

No comments yet. Be the first.

RELATED STORIES

This page was compiled with AI assistance from the outlets named above and passed an automated language check before publication. Montegre has no reporters of its own; the byline names the outlets the story was compiled from. Method and editorial standards