Newshot
Account

Microsoft says attempts to fix global outage not going 'as expected'

Original text(via Hacker News)

Live Reporting

Edited by Imogen James and Jenna Moon

  • Not much at the moment, political correspondent Iain Watson says.He says he's been touch with the Cabinet Office, which has a coordinating role across government to find out how departments might be affected. They've said they're aware of the ongoing technical issues and are working "closely" with Microsoft to assess any impact on government services, Watson says. But they also say they have "robust" contingency plans in place, but haven't said what those contingency plans are, he adds.The government is very keen to stress that the intelligence and security services are unaffected, adding that there's a "separate system for anything secret", Watson says. Share pageAbout sharing

Not much at the moment, political correspondent Iain Watson says.

He says he's been touch with the Cabinet Office, which has a coordinating role across government to find out how departments might be affected.

They've said they're aware of the ongoing technical issues and are working "closely" with Microsoft to assess any impact on government services, Watson says.

But they also say they have "robust" contingency plans in place, but haven't said what those contingency plans are, he adds.

The government is very keen to stress that the intelligence and security services are unaffected, adding that there's a "separate system for anything secret", Watson says.

A purple banner which says Your Voice
A purple banner which says Your Voice
  • Jake Moore, a cyber-security adviser at internet security company ESET, says the scale of this outage is "absolutely massive".Moore says Microsoft "sits at the heart of many businesses" meaning disruption is "huge". Companies need to make sure they have resilience built in for situations such as this, he adds. "This elevated load that they talk about is just one of those examples that we're going to see more and more in the future," he says.There needs to be resilience, he argues, which means extra server space and protections. Share pageAbout sharing

Jake Moore, a cyber-security adviser at internet security company ESET, says the scale of this outage is "absolutely massive".

Moore says Microsoft "sits at the heart of many businesses" meaning disruption is "huge". Companies need to make sure they have resilience built in for situations such as this, he adds.

"This elevated load that they talk about is just one of those examples that we're going to see more and more in the future," he says.

There needs to be resilience, he argues, which means extra server space and protections.

Purple banner showing the words: "Your Voice"
Purple banner showing the words: "Your Voice"
  • Asked if a cyber attack has been ruled out at this stage, senior technology reporter Chris Vallance says that Microsoft has stated that there's nothing malicious about this incident. "When you look at previous outages of this kind, that affected a lot of different services, it's hard to think of one that was due to a cyber attack," he says. "Most of them have been due to things going wrong internally rather than malicious action," he says, so there's "certainly no reason to doubt" what Microsoft are saying at this stage. Share pageAbout sharing

Asked if a cyber attack has been ruled out at this stage, senior technology reporter Chris Vallance says that Microsoft has stated that there's nothing malicious about this incident.

"When you look at previous outages of this kind, that affected a lot of different services, it's hard to think of one that was due to a cyber attack," he says.

"Most of them have been due to things going wrong internally rather than malicious action," he says, so there's "certainly no reason to doubt" what Microsoft are saying at this stage.

A purple banner with white writing saying "Your Voice" on it
A purple banner with white writing saying "Your Voice" on it
  • "What we're lacking from Microsoft is a clear, plain English post mortem on what's going on", technology report Chris Vallance says.They may be a bit busy at the moment, to be fair, he adds. Vallance explains that the company has been posting on the Microsoft 365 status page a series of updates in quite technical language. They're saying they are having some impact in their mitigating measures, Vallance says, "but clearly we're hours on from something which started at 18:00 yesterday". "For something so critical that has affected key business software, that really is a long time for them not to be able to get things back on track." Share pageAbout sharing

"What we're lacking from Microsoft is a clear, plain English post mortem on what's going on", technology report Chris Vallance says.

They may be a bit busy at the moment, to be fair, he adds.

Vallance explains that the company has been posting on the Microsoft 365 status page a series of updates in quite technical language.

They're saying they are having some impact in their mitigating measures, Vallance says, "but clearly we're hours on from something which started at 18:00 yesterday".

"For something so critical that has affected key business software, that really is a long time for them not to be able to get things back on track."

A purple banner which says YOUR VOICE
A purple banner which says YOUR VOICE
  • Our experts will be answering your questions on the Microsoft global outage from 10:30 GMT, which you can watch live at the top of the page.On the panel, there's senior technology reporter Chris Vallance, political correspondent Iain Watson, senior health reporter Jim Reed and Jake Moore, a global cybersecurity advisor. Share pageAbout sharing

Our experts will be answering your questions on the Microsoft global outage from 10:30 GMT, which you can watch live at the top of the page.

On the panel, there's senior technology reporter Chris Vallance, political correspondent Iain Watson, senior health reporter Jim Reed and Jake Moore, a global cybersecurity advisor.

A purple banner which says YOUR VOICE
A purple banner which says YOUR VOICE
  • Image source, PA MediaThe impact of Microsoft's outage is far-reaching, with institutions across the world reporting issues and workarounds.Multiple police forces in the UK say they have been affected by the outage, and are asking people to call if they need to get in touch.MPs are reporting their access via parliamentary accounts to Microsoft products is currently down - alternative arrangements are in place, a spokesperon says.The outage is also affecting access to the NHS email system, limiting pharmacies’ ability to communicate with GP practices and some patients. The UK National Pharmacy Association warns waits may be longer for prescriptions.The GMB union has said the Microsoft outage is having a "serious impact" across ambulance and NHS trusts.Many schools are reporting that they too are affected by the Microsoft outage this morning.Some have taken to social media to encourage parents to get in touch with them by phone, rather than email, for things like reporting pupil absence.Earlier, we heard how the Japanese government and the Australian spy agency were affected, though Japan says operations are continuing through "alternative means of communication".The Philippines Supreme Court has warned that email responses may be delayed, while the UK's Parliament says "alternative arrangements are in place" after the outage disrupted email use on the parliamentary network.We'll be keeping track of who and what else is affected, stick with us. Share pageAbout sharing
A smartphone displaying the outlook email app, which says: "Downloading your messages, this may take a few moments depending on the folder size and the network connection"
A smartphone displaying the outlook email app, which says: "Downloading your messages, this may take a few moments depending on the folder size and the network connection"

The impact of Microsoft's outage is far-reaching, with institutions across the world reporting issues and workarounds.

Multiple police forces in the UK say they have been affected by the outage, and are asking people to call if they need to get in touch.

MPs are reporting their access via parliamentary accounts to Microsoft products is currently down - alternative arrangements are in place, a spokesperon says.

The outage is also affecting access to the NHS email system, limiting pharmacies’ ability to communicate with GP practices and some patients. The UK National Pharmacy Association warns waits may be longer for prescriptions.

The GMB union has said the Microsoft outage is having a "serious impact" across ambulance and NHS trusts.

Many schools are reporting that they too are affected by the Microsoft outage this morning.

Some have taken to social media to encourage parents to get in touch with them by phone, rather than email, for things like reporting pupil absence.

Earlier, we heard how the Japanese government and the Australian spy agency were affected, though Japan says operations are continuing through "alternative means of communication".

The Philippines Supreme Court has warned that email responses may be delayed, while the UK's Parliament says "alternative arrangements are in place" after the outage disrupted email use on the parliamentary network.

We'll be keeping track of who and what else is affected, stick with us.

  • Jaltson Akkanath ChummarAsia business producer Image source, Getty ImagesImage caption, Single tech outage can impact multiple business functionsToday’s disruption of Microsoft services highlights the risks faced by businesses that rely heavily on a single technology provider for major operation needs. Companies generally go for one particular tech company like Microsoft or Google for their email, meeting and software needs as it's easier to manage a single provider and is more cost effective. But that makes tech outages a more severe challenge for those businesses. "When several critical services depend on the same provider, one disruption can affect multiple business functions simultaneously. This is known as concentration risk," says Bernard Tan from global technology firm Protiviti.He says the issue isn't relying on one major provider, but whether "companies understand these dependencies and have practical alternatives when services become unavailable," Bernard told the BBC. "A more practical approach is to invest in alternative arrangements for the services that are most important to their operations," he added. Share pageAbout sharing

Jaltson Akkanath ChummarAsia business producer

Image of a cloud networking logo
Image of a cloud networking logo

Single tech outage can impact multiple business functions

Today’s disruption of Microsoft services highlights the risks faced by businesses that rely heavily on a single technology provider for major operation needs.

Companies generally go for one particular tech company like Microsoft or Google for their email, meeting and software needs as it's easier to manage a single provider and is more cost effective.

But that makes tech outages a more severe challenge for those businesses.

"When several critical services depend on the same provider, one disruption can affect multiple business functions simultaneously. This is known as concentration risk," says Bernard Tan from global technology firm Protiviti.

He says the issue isn't relying on one major provider, but whether "companies understand these dependencies and have practical alternatives when services become unavailable," Bernard told the BBC.

"A more practical approach is to invest in alternative arrangements for the services that are most important to their operations," he added.

  • Laura CressTechnology reporter While we seem to know the cause of the outage - a technical problem with Microsoft’s server infrastructure - the length of time it has taken to fix has left many experts scratching their heads. Gergely Orosz, author of The Pragmatic Engineer newsletter and a former leading software engineer at several major tech companies including Skype and Uber, described the disruption as "very unusual" for a company with Microsoft’s level of engineering expertise and calibre. "That mitigation is taking 12 plus hours for a global outage for a service with 100 million plus users - this is very rare," he said. He added it was still difficult to know just how many people have been affected by the disruption, but said he "cannot recall" an outage over business email ever taking this long to return to normal. "This could well be the most disruptive email-related business outage of all time," he said. Share pageAbout sharing

Laura CressTechnology reporter

While we seem to know the cause of the outage - a technical problem with Microsoft’s server infrastructure - the length of time it has taken to fix has left many experts scratching their heads.

Gergely Orosz, author of The Pragmatic Engineer newsletter and a former leading software engineer at several major tech companies including Skype and Uber, described the disruption as "very unusual" for a company with Microsoft’s level of engineering expertise and calibre.

"That mitigation is taking 12 plus hours for a global outage for a service with 100 million plus users - this is very rare," he said.

He added it was still difficult to know just how many people have been affected by the disruption, but said he "cannot recall" an outage over business email ever taking this long to return to normal.

"This could well be the most disruptive email-related business outage of all time," he said.

  • Clinical and operational systems in the NHS still appear to be working effectively during the Microsoft outage, the chief executive of the NHS Alliance has said.Ciarán Devane says: "NHS leaders and their teams are working hard to minimise disruption, maintain essential services and support staff and patients who may be affected."He adds that the priority of leaders is always to ensure patient safety and continuity of care. "NHS organisations have workarounds in place when email is unavailable, so contingencies are in place to manage situations like this," he says. Share pageAbout sharing

Clinical and operational systems in the NHS still appear to be working effectively during the Microsoft outage, the chief executive of the NHS Alliance has said.

Ciarán Devane says: "NHS leaders and their teams are working hard to minimise disruption, maintain essential services and support staff and patients who may be affected."

He adds that the priority of leaders is always to ensure patient safety and continuity of care.

"NHS organisations have workarounds in place when email is unavailable, so contingencies are in place to manage situations like this," he says.

  • Image source, Getty ImagesThe GMB union has said the Microsoft outage is having a "serious impact" across ambulance and NHS trusts.Rachel Harrison, the union's national secretary, says: “Those who work in 111 and 999 services are unable to refer patient details in the normal way which is creating a massive backlog and has the potential for chaos."She says emergency and life-threatening call outs appear to be unaffected, with the union working to support members during the outage.Staff working in the NHS have spoken about how the incident is affecting them today, with one GP describing the outage as "very disruptive". Share pageAbout sharing
A lines of yellow and green ambulances parked up
A lines of yellow and green ambulances parked up

The GMB union has said the Microsoft outage is having a "serious impact" across ambulance and NHS trusts.

Rachel Harrison, the union's national secretary, says: “Those who work in 111 and 999 services are unable to refer patient details in the normal way which is creating a massive backlog and has the potential for chaos."

She says emergency and life-threatening call outs appear to be unaffected, with the union working to support members during the outage.

Staff working in the NHS have spoken about how the incident is affecting them today, with one GP describing the outage as "very disruptive".

  • We've been hearing from more of you as the Microsoft outage continues.Ester from Bamber Bridge in Lancashire says her team can't get any work done with this outage, as they're unable to contact colleagues abroad. "People are asking, does this mean we can go home early? But so far no news from management saying we can go home. When does this end," she tells BBC News. Kellie in Shropshire says she can't download her ticket for the BBC Children In Need filming this evening. "Stressed doesn’t cover it, having to drive for nearly two hours and hope the situation has been resolved by the time we get there," she says. Joshua in West Yorkshire says "scammers are looking to take advantage already". "We have had a phone call from a lady claiming to be from Microsoft needing our login details to restore access at work. The only problem is we don't use Microsoft 365!" he says. Share pageAbout sharing

We've been hearing from more of you as the Microsoft outage continues.

Ester from Bamber Bridge in Lancashire says her team can't get any work done with this outage, as they're unable to contact colleagues abroad.

"People are asking, does this mean we can go home early? But so far no news from management saying we can go home. When does this end," she tells BBC News.

Kellie in Shropshire says she can't download her ticket for the BBC Children In Need filming this evening. "Stressed doesn’t cover it, having to drive for nearly two hours and hope the situation has been resolved by the time we get there," she says.

Joshua in West Yorkshire says "scammers are looking to take advantage already".

"We have had a phone call from a lady claiming to be from Microsoft needing our login details to restore access at work. The only problem is we don't use Microsoft 365!" he says.

Purple banner which says your voice
Purple banner which says your voice
  • Cath BurnsHealth correspondent We heard earlier from the UK National Pharmacy Association, which said the outage is limiting pharmacies’ ability to communicate with GP practices and some patients.“There will be major problems NHS-wise." That’s a message from one of my GP contacts. He says Outlook is the “main mode of comms” and that he relies heavily on Teams meetings too. Several say they can’t use their emails at all. One says: “Morning, it’s affecting us quite a bit, we have no access to NHS mail so we can't see any prescription requests or emails from patients.” Another tells me they’re “unable to send any referrals as all go through NHS encrypted email (Microsoft). Just working out a plan for urgent issues for the day - I suspect it will involve a lot of telephony…” The main issue does seem to be figuring out how to contact other parts of the system - nursing homes, district nurses, community mental health teams, social workers. A few GPs are telling me they’re not seeing any major issues - generally those who don’t tend to use Teams that much. One says: “Overall it’s business as usual but might be more telephone communications where we can’t access emails.” Share pageAbout sharing

Cath BurnsHealth correspondent

We heard earlier from the UK National Pharmacy Association, which said the outage is limiting pharmacies’ ability to communicate with GP practices and some patients.

“There will be major problems NHS-wise."

That’s a message from one of my GP contacts. He says Outlook is the “main mode of comms” and that he relies heavily on Teams meetings too.

Several say they can’t use their emails at all. One says: “Morning, it’s affecting us quite a bit, we have no access to NHS mail so we can't see any prescription requests or emails from patients.”

Another tells me they’re “unable to send any referrals as all go through NHS encrypted email (Microsoft). Just working out a plan for urgent issues for the day - I suspect it will involve a lot of telephony…”

The main issue does seem to be figuring out how to contact other parts of the system - nursing homes, district nurses, community mental health teams, social workers.

A few GPs are telling me they’re not seeing any major issues - generally those who don’t tend to use Teams that much. One says: “Overall it’s business as usual but might be more telephone communications where we can’t access emails.”

  • Microsoft has given another update as it works to fix the outage. It says it is seeing "early signs of stabilisation", but service capacity has "not yet returned to normal levels". System health "remains variable" it says, as traffic is "carefully reintroduced", it says. "As part of this effort, we are extending the email timeout window to 48 hours to help protect delayed messages while service recovery continues," the statement says. The email timeout window is, effectively, the amount of time a client like Outlook will spend waiting to receive an email. Microsoft is doing this to avoid a situation where emails which have been sent - such as from a Gmail account or something else unaffected - do not ultimately arrive when things are fixed. Microsoft says it is continuing to "treat this with urgency", and will share a full report after the incident is resolved. Share pageAbout sharing

Microsoft has given another update as it works to fix the outage.

It says it is seeing "early signs of stabilisation", but service capacity has "not yet returned to normal levels".

System health "remains variable" it says, as traffic is "carefully reintroduced", it says.

"As part of this effort, we are extending the email timeout window to 48 hours to help protect delayed messages while service recovery continues," the statement says.

The email timeout window is, effectively, the amount of time a client like Outlook will spend waiting to receive an email.

Microsoft is doing this to avoid a situation where emails which have been sent - such as from a Gmail account or something else unaffected - do not ultimately arrive when things are fixed.

Microsoft says it is continuing to "treat this with urgency", and will share a full report after the incident is resolved.

  • Nathan StandleyEducation reporter Image source, Getty ImagesMany schools are reporting that they too are affected by the Microsoft outage this morning.Some have taken to social media to encourage parents to get in touch with them by phone, rather than email, for things like reporting pupil absence. Schools from Bishop Fox's Community School in Taunton, Somerset, to the Queen Elizabeth Academy in Atherstone, Warwickshire, and the Stockport School have all said on X that their emails are currently down.We have been in touch with the Department for Education about how the education system is being affected more widely and will report back what they tell us. Share pageAbout sharing

Nathan StandleyEducation reporter

A line of school children wearing jumpers and backpacks
A line of school children wearing jumpers and backpacks

Many schools are reporting that they too are affected by the Microsoft outage this morning.

Some have taken to social media to encourage parents to get in touch with them by phone, rather than email, for things like reporting pupil absence.

Schools from Bishop Fox's Community School in Taunton, Somerset, to the Queen Elizabeth Academy in Atherstone, Warwickshire, and the Stockport School have all said on X that their emails are currently down.

We have been in touch with the Department for Education about how the education system is being affected more widely and will report back what they tell us.

  • Tom GerkenTech reporter I’ve been writing about outages for years, and one problem we always come up against is a question of scale.Perhaps unsurprisingly, companies are rarely eager to tell us just how many people are affected when their products become inaccessible.So, we often rely on the website Downdetector, which tracks online outages. It said it received 43,000 reports of issues with Microsoft Outlook at around 07:00 GMT, but that number is typically a low estimate. For one thing, it’s only helpful for the countries where folk are awake - at its peak, there were more than 130,000 reports in the US, a number far lower now it's the middle of the night.And the other glaring issue is it only really tells us how many people use the website. Think of this - are you affected by the outage and did you tell Downdetector about it?I think for a lot of people, the answer to that question would be a “yes” followed by a “no”. With 430 million people using Office365 services according to Microsoft, you have to wonder just how many aren’t able to access their emails this morning. It all points to this being a massive, global outage. Share pageAbout sharing

Tom GerkenTech reporter

I’ve been writing about outages for years, and one problem we always come up against is a question of scale.

Perhaps unsurprisingly, companies are rarely eager to tell us just how many people are affected when their products become inaccessible.

So, we often rely on the website Downdetector, which tracks online outages. It said it received 43,000 reports of issues with Microsoft Outlook at around 07:00 GMT, but that number is typically a low estimate.

For one thing, it’s only helpful for the countries where folk are awake - at its peak, there were more than 130,000 reports in the US, a number far lower now it's the middle of the night.

And the other glaring issue is it only really tells us how many people use the website. Think of this - are you affected by the outage and did you tell Downdetector about it?

I think for a lot of people, the answer to that question would be a “yes” followed by a “no”.

With 430 million people using Office365 services according to Microsoft, you have to wonder just how many aren’t able to access their emails this morning.

It all points to this being a massive, global outage.

  • Chris MasonPolitical editor Image source, Getty ImagesMPs are reporting their access via parliamentary accounts to Microsoft products is currently down.A parliamentary spokesperson said: “We are aware of the global issue affecting Microsoft email services, which is impacting users of the parliamentary network. We are working with Microsoft to resolve the matter as quickly as possible. "Alternative communication arrangements are in place, allowing today’s business to continue.” The House of Commons will sit as normal from 09.30 this morning. Share pageAbout sharing

Chris MasonPolitical editor

View of the palace of Westminster with the sun shining on the side of the building.
View of the palace of Westminster with the sun shining on the side of the building.

MPs are reporting their access via parliamentary accounts to Microsoft products is currently down.

A parliamentary spokesperson said: “We are aware of the global issue affecting Microsoft email services, which is impacting users of the parliamentary network. We are working with Microsoft to resolve the matter as quickly as possible.

"Alternative communication arrangements are in place, allowing today’s business to continue.”

The House of Commons will sit as normal from 09.30 this morning.

  • Tom EspinerBusiness reporter Journalists here at BBC News are picking up phones and putting in calls to a number of organisations such as major supermarkets, banks, transport hubs, and government departments, to see if they’ve been affected, and if there's no immediate answer we’re waiting to hear back.These efforts are being a little hampered by two things. Firstly, our corporate emails are not working, so we’re phone bashing.Secondly, some organisations have done away with giving out a phone number for their press offices - instead providing an email. It’s easy to guess the difficulty with this: if you’re contacting an organisation to see if their email is working, it’s probably best not to email them to do so - it might not be getting through. Share pageAbout sharing

Tom EspinerBusiness reporter

Journalists here at BBC News are picking up phones and putting in calls to a number of organisations such as major supermarkets, banks, transport hubs, and government departments, to see if they’ve been affected, and if there's no immediate answer we’re waiting to hear back.

These efforts are being a little hampered by two things. Firstly, our corporate emails are not working, so we’re phone bashing.

Secondly, some organisations have done away with giving out a phone number for their press offices - instead providing an email.

It’s easy to guess the difficulty with this: if you’re contacting an organisation to see if their email is working, it’s probably best not to email them to do so - it might not be getting through.

  • Dr Tim Frank, an NHS GP in Penzance, Cornwall, tells BBC Radio 4's Today programme that Microsoft products are "central" to the operation of his area's primary care and community services.He adds that he has been unable to receive any emails overnight or again this morning."This is going to be very disruptive today and Fridays are always a heavily laden day due to the upcoming weekend," he says.The doctor says alternative communication numbers are already being distributed among teams so no patients suffer. Share pageAbout sharing

Dr Tim Frank, an NHS GP in Penzance, Cornwall, tells BBC Radio 4's Today programme that Microsoft products are "central" to the operation of his area's primary care and community services.

He adds that he has been unable to receive any emails overnight or again this morning.

"This is going to be very disruptive today and Fridays are always a heavily laden day due to the upcoming weekend," he says.

The doctor says alternative communication numbers are already being distributed among teams so no patients suffer.

  • Tinshui YeungLive reporter The UK National Pharmacy Association has told me that the Microsoft outage is also affecting pharmacies across the country.A spokesperson says the outage is affecting access to the NHS email system, limiting pharmacies’ ability to communicate with GP practices and some patients.“Please bear with your pharmacy teams today, where waits may be longer than normal and there may be delays to some prescriptions,” he says. Share pageAbout sharing

Tinshui YeungLive reporter

The UK National Pharmacy Association has told me that the Microsoft outage is also affecting pharmacies across the country.

A spokesperson says the outage is affecting access to the NHS email system, limiting pharmacies’ ability to communicate with GP practices and some patients.

“Please bear with your pharmacy teams today, where waits may be longer than normal and there may be delays to some prescriptions,” he says.

  • Readers in the UK have been getting in touch with the BBC to say how the Microsoft outage is impacting their days.Teachers Jack and Matt are affected - Jack says his emails are currently unavailable but lesson plans are still accessible, while Matt is unable to get on to One Drive to use his school's resources.Dave says he has lost his calendar entries and has been unable to access emails since just after 21:00 on Thursday.He says: "This is having a terrible effect on my business and I’m so stressed."Meanwhile, Martin says he will have to rely on phone calls outside of Teams throughout his workday."Have woken up to no new emails... absolute bliss, or it was before I realised today is a crucial day to conclude a business deal," he says. Share pageAbout sharing

Readers in the UK have been getting in touch with the BBC to say how the Microsoft outage is impacting their days.

Teachers Jack and Matt are affected - Jack says his emails are currently unavailable but lesson plans are still accessible, while Matt is unable to get on to One Drive to use his school's resources.

Dave says he has lost his calendar entries and has been unable to access emails since just after 21:00 on Thursday.

He says: "This is having a terrible effect on my business and I’m so stressed."

Meanwhile, Martin says he will have to rely on phone calls outside of Teams throughout his workday.

"Have woken up to no new emails... absolute bliss, or it was before I realised today is a crucial day to conclude a business deal," he says.

A purple graphic with Your Voice written on it in white
A purple graphic with Your Voice written on it in white

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by Hacker News
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗