Newshot

Microsoft exec explains why Microsoft 365 went down worldwide: a threefold failure set off a chain reaction that took out 80% of servers

AI summary

On November 12, a large-scale outage hit multiple core Microsoft 365 cloud services, leaving millions of users unable to access email for over 12 hours. Microsoft's preliminary report attributes the incident to a software defect from July that combined with two other infrastructure issues to trigger a cascading failure, at one point leaving 80% of Microsoft's network servers unresponsive. Executive Charles Lamanna detailed on X how a latent lock contention bug, reduced gateway capacity, and a Teams traffic shift interacted to cause the outage.

Translation(model translation, not the source's own text)

Microsoft Executive Explains Why Microsoft 365 Services Went Down Worldwide: Triple Failure Triggered a Cascading Reaction, Knocking Out 80% of Servers

IT之家 November 16 news, on November 12 local time, a large-scale failure occurred in multiple core cloud services under Microsoft's Microsoft 365, causing Outlook, Exchange Online, Teams, SharePoint and others to go down worldwide, leaving millions of users unable to access email for over 12 hours.

Today Microsoft released a preliminary report on the incident. According to the preliminary assessment, the root cause of the failure was a software defect from July, which then acted in concert with two other infrastructure issues to trigger a cascading failure, at one point causing 80% of Microsoft's web servers to become unresponsive.

Microsoft executive Charles Lamanna (Charles Lamanna) has now shared a detailed explanation of the cause of the failure on Platform X. IT之家 summarizes the key content as follows:

  • In July of this year, Microsoft introduced a software defect while fixing a Microsoft 365 network gateway issue. The flaw caused lock contention in server processes, preventing them from responding to specific workloads. However, the software containing the defect passed Microsoft's routine testing and was deployed worldwide within weeks without any issues arising during that period.
  • Microsoft found that the testing at the time failed to detect the software defect because the test environment mainly used older, lower-performance CPUs and hardware, whereas on newer servers with more cores, the flaw was more likely to be triggered.
  • A few hours before this large-scale failure, Microsoft shifted Teams traffic from a backup network gateway to the primary network gateway. That gateway also handles Exchange Online traffic, and the adjustment was made to ensure Teams could operate normally. Although this was a routine failover operation in Microsoft's infrastructure, it increased the load on the Exchange Online primary network gateway and changed its traffic distribution.
  • Due to another unrelated issue that occurred the previous day, about 14% of the network gateway's server capacity was already unusable. At that point, the remaining 86% of capacity was still sufficient to handle normal workloads.
  • When the above three issues were triggered simultaneously, the service outage formally began. The increased Teams traffic combined with the reduced server capacity jointly triggered the previously undetected software flaw.
  • When servers became unresponsive, Microsoft's infrastructure automatically forwarded traffic to servers that were operating normally. But this mechanism instead made the situation worse. After receiving large amounts of forwarded traffic, server load increased, which in turn triggered the same software flaw again, causing more and more servers to fall like a cascading reaction. Within less than two hours, the failure had spread to 80% of Microsoft's network gateway servers.
  • Normally, Microsoft limits the impact of such failures through independent capacity partitions and regions. However, another issue prevented these protection mechanisms from stopping the cascading failure, ultimately allowing it to spread worldwide.

Microsoft plans to release a detailed report later this week, at which time more technical details will be disclosed.

Related Articles

  • Microsoft tests remote lock feature for Windows 11 enterprise devices, allowing lost PCs to be frozen instantly with no local unlock
  • Users discover that Microsoft Windows 11's initial setup process OOBE does not support pure IPv6 networks
  • Microsoft Windows 11 reportedly affected by SignDrop zero-day vulnerability: ordinary users can escalate privileges to SYSTEM, with no dedicated fix patch currently available
  • Developer launches WinPopBit project to enable Microsoft Windows 11 to run on older processors such as AMD K8 and Intel Pentium 4
  • Microsoft's new WinUI3 "Chinese Input Method" quietly listed: supports importing personal dictionaries, focuses on being ad-free and supports Bing search
  • Microsoft pushes original XBOX / XBOX 360 startup animations to XBOX Series X|S and XBOX One consoles

The translation is machine-generated and individual terms may be off. Nothing is rewritten or cut; the original is authoritative.

Published by IT之家
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗