Skip to content
NOWCAST WDSU News at 10pm
Watch on Demand
Advertisement

Recovering from the global tech outage could be a long, arduous process

Recovering from the global tech outage could be a long, arduous process
YOU. SAM CARTER WLKY NEWS. SAM THANK YOU. THIS IS AFFECTING SO MANY ASPECTS OF LIFE, RIGHT? THE CYBER SECURITY FIRM AT THE HEART OF THE OUTAGE, CROWDSTRIKE, TELLS CBS NEWS THE OUTAGE IS NOT A SECURITY INCIDENT OR A CYBER ATTACK. INSTEAD, THE COMPANY’S CEO SAYS IT WAS CAUSED BY A FAULTY UPDATE FOR WINDOWS HOST. SO THIS MORNING, WE’RE JOINED BY LOCAL CYBERSECURITY EXPERT RICHARD CONNOR. RICHARD, WE TALKED TO YOU EARLIER IN THE HOUR AND YOU EXPLAINED. TELL US AGAIN EXACTLY HOW AN UPDATE CAUSED ALL OF THIS DISRUPTION. RIGHT. SO AN UPDATE WAS PUSHED OUT TO MACHINES. THAT UPDATE CAUSED THOSE MACHINES TO NOT WORK AS INTENDED. AND SO YOU HAVE TO MANUALLY RESET YOUR MACHINES AND DO SOME MANUAL MANIPULATION. PARDON ME, TO GET THINGS BACK UP AND RUNNING. YEAH. SO WE’RE SO RELIANT ON WHAT CELL PHONES, INTERNET, THE THINGS DEBIT CARDS, CREDIT CARDS, ALL OF THOSE THINGS. MOBILE ORDERS THAT WE JUST THINK WE CAN QUICKLY PUT IN, TALK ABOUT WHAT A BIG DEAL IT IS THAT WE’RE RELYING ON THOSE THINGS. AND THEN WHEN THIS HAPPENS, WE CAN’T USE IT. THAT’S RIGHT. I MEAN, EVEN A CUP OF COFFEE, YOU CAN MAKE YOUR OWN CUP OF COFFEE IN THE MORNING, OR YOU CAN GO TO YOUR LOCAL STORE AND GET YOUR CUP OF COFFEE. BUT TODAY THIS TYPE OF OUTAGE SHOWS US THAT THINGS ARE SO INTERCONNECTED THAT WHEN ONE THING FAILS, OTHER THINGS FAIL. SO YOU COULD GO TO YOUR LOCAL COFFEE SHOP. THEY MIGHT BE OPEN, BUT YOU CAN’T SWIPE YOUR CREDIT CARD. THEIR TOP EMPLOYEE COULDN’T SHOW UP TODAY BECAUSE THEY COULDN’T CATCH A FLIGHT. SO THESE CASCADING EFFECTS, WE’RE LEARNING MORE AND MORE ABOUT, ALL BECAUSE OF AN UPDATE, A SYSTEM UPDATE. THAT’S RIGHT. WE’RE NOT EVEN TALKING CYBER ATTACKS. THAT HAPPENS EVERY SINGLE DAY. AND JUST THIS ONE IN PARTICULAR. IT’S JUST A FLUKE. DIDN’T GO AS INTENDED. RIGHT. IS THERE ANYTHING THAT WE CAN DO? IS THERE ANYTHING THAT THE GENERAL PUBLIC CAN DO TO PROTECT THEMSELVES FROM THIS TYPE OF THING, FROM HAPPENING? WELL, IN YOUR OWN MACHINE, YOUR OWN LAPTOP, YOU CAN HAVE AUTOMATED UPDATES AS WELL. YOU MIGHT WANT TO NOT HAVE THAT SET ON ANYMORE. AND YOU MIGHT WANT TO JUST DO MANUAL UPDATES, KIND OF WAIT UNTIL THE UPDATE HAS BEEN PUSHED. THINK ABOUT WHETHER OR NOT YOU WANT TO UPDATE, DO SOME RESEARCH, AND THEN PUSH THAT UPDATE LATER. BUT FOR THE MOST PART, THIS DOESN’T HAPPEN. I MEAN, THESE THESE UPDATES GO OUT EVERY SINGLE DAY. JUST JUST THIS ONE IN PARTICULAR HAD A BIG IMPACT. IT DIDN’T GO AS INTENDED. SO FOR EXAMPLE, IF YOU USE CROWDSTRIKE, IF THIS YOUR COMPANY USED THIS TECHNOLOGY AND YOU DID NOT HAVE THOSE AUTOMATIC UPDATES ON THIS WOULDN’T HAVE AFFECTED YOU, IT WOULD NOT HAVE, YOU WOULD BE ABLE TO EVALUATE THAT BEFORE YOU UPDATED YOUR SYSTEMS AND YOU’D SEE THE NEWS. THIS MORNING AND SAY, MAYBE WE NEED TO RETHINK OUR POSITION HERE. OH, THERE YOU GO. THERE YOU GO. WELL, THANK YOU SO MUCH. IS THERE ANY LAST ADVICE? MAYBE WHEN IT COMES TO JUST PRIVACY PROTECTION, THAT KIND OF THING? SURE. PLAN AND BACKUP PLAN. THINK ABOUT HOW YOU’RE BACKING UP YOUR DATA, EVEN YOUR PERSONAL DATA. IF YOUR PHONE IF YOUR SOCIAL MEDIA DIDN’T WORK TODAY, DO YOU HAVE THOSE PHOTOS BACKED UP? DO YOU HAVE YOUR OTHER INFORMATION BACKED UP? IF YOU WERE TO LOSE EVERYTHING THAT’S ON YOUR LAPTOP TODAY, WHAT IMPACT WOULD THAT HAVE? START MAKING SOME BACKUP PLANS FOR THAT. WELL, RICHARD, THANK YOU SO MUCH FOR YOUR EXPERTISE. THINGS ARE GOING TO SLOWLY BUT SURELY GET BACK ONLINE. AT LEAST THAT’S HOW THINGS HAVE HAPPENED HERE AT OUR STATION THIS MORNING AS WE DEALT WITH THE EXACT SAME ISSUES.
Advertisement
Recovering from the global tech outage could be a long, arduous process
The company that caused a massive worldwide computer outage said a flawed update had been rolled back, but that doesn’t necessarily help the thousands of businesses affected by the glitch.Video above: How to protect yourself after global outageThe CrowdStrike software issue at the heart of the outage runs at such a deep level in affected computers and systems that getting them up and running just to be fixed will be, in many cases, an enormous challenge.That’s compounded by the fact that many of the servers that may contain information needed to get these systems working again are themselves caught in a cycle of crashing and rebooting."I don’t think it's too early to call it: this will be the largest IT outage in history," said security expert Troy Hunt in a post on X.The CrowdStrike software at fault operates at a computer’s kernel level, a much deeper level than what more ordinary applications such as browsers or video games do. This portion of a device has much greater visibility and control over a computer and its components, making it critical for the operation of all other systems and far more sensitive.Running at the kernel level means CrowdStrike's software can do more to detect cyberattacks, but it also means the current bug is causing Windows computers to crash to a Blue Screen of Death before users can take any actions to correct it.The issue appears to be recoverable, CrowdStrike has said, but in many cases it requires painstaking work: Each affected device must be accessed by an administrator and manually rebooted into safe mode. Then, the offending CrowdStrike file must be deleted by hand.For businesses with hundreds or thousands of laptops, desktops and servers running CrowdStrike’s security software, an individual human may have to perform that process over and over and over again."You can’t automate that," said Kevin Beaumont, a security researcher and former Microsoft threat analyst, in a post on X. "So this is going to be incredibly painful for CrowdStrike customers."Video above: CrowdStrike CEO says company has 'deployed a fix' to tech issueIt gets worse.Organizations that take security seriously will have likely encrypted their computers’ hard drives, making it even more challenging to access the file that needs to be deleted.For those organizations, "you need to manually decrypt the disk with a BitLocker Recovery Key, which is probably — for most companies — stored digitally on one of the servers that is currently booting over and over," said Ira Bailey, a security researcher, in a post on BlueSky.Every affected computer that is BitLocker-encrypted will need to be unlocked with a recovery key before organizations can begin the process of deleting the bad CrowdStrike file and restoring normal operation, said the cybersecurity expert who goes by the pseudonymous handle SwiftOnSecurity in a post on X.Recovery will be enormously expensive for Fortune 500 companies with large teams of IT staff and likely even more challenging for smaller firms, Kenn White, an independent security researcher who specializes in network security, told CNN."If you don’t have physical staff that can actually touch it, this is going to take many, many days for much of corporate America to recover from," White said. "It's just a ton of labor-intensive manual work.""It’s a fairly complicated procedure for non-technical people," White added, "and even a lot of skilled IT professionals will find it difficult to do this at the scale that’s going to be required given the number of machines that are affected."How did the CrowdStrike bug lead to such widespread effects?Because CrowdStrike’s security software is running on countless individual computers all around the globe, the update that got pushed to those devices caused them all to shut down, virtually simultaneously.And in today’s networked economy, an outage in one part of a supply chain can cause domino effects up and down the line. When multiple parts of a supply chain go down, it touches off a cascade of problems.Imagine a person trying to buy a coffee, said Andrew Peck, a cybersecurity expert at Loughborough University in the UK. What may seem like a simple transaction relies on multiple computers working in tandem, from the coffee shop’s point of sale to the payment processor’s own back-end systems.Video above: Starbucks, hotels, businesses affected by CrowdStrike outage"There are a lot of computers in this chain, and usually the larger the business, the larger the chain," Peck said. "If any one of the computers are down in the chain, the transaction will not complete."Now scale that up to something like the massive aviation industry, the critical financial services sector or the life-or-death operations of a health care provider, and the scope of the disaster becomes starkly clear.What is Microsoft’s role in all this?A separate issue earlier, on Thursday, did lead to significant impacts on many of Microsoft’s own cloud customers, but it was resolved overnight and was unrelated to the CrowdStrike issue, multiple cybersecurity experts said.The CrowdStrike bug may have initially been conflated with the Microsoft issue because CrowdStrike’s error affected only Windows machines.“Both are Microsoft-related, but Microsoft had nothing to do with the second incident,” White told CNN.That appears to be supported by Microsoft’s own status account on X, which on Thursday announced an issue affecting "Microsoft 365 apps and services" and a separate announcement Friday addressing the CrowdStrike outage. The two issues are being tracked using different reference numbers.As of Friday morning, Microsoft said the issue with Microsoft 365 had been resolved and that the situation was improving. Microsoft didn’t immediately respond to a request for comment.Since the update to CrowdStrike’s software was delivered by the company’s own systems, it appears unlikely that Microsoft bears direct responsibility for Friday’s outages, said Beaumont, who said he reviewed a copy of CrowdStrike’s flawed update.The problem with CrowdStrike’s update was that it wasn’t formatted correctly “and causes Windows to crash every time,” Beaumont posted on X.CNN’s Olesya Dmitracova contributed reporting.

The company that caused a massive worldwide computer outage said a flawed update had been rolled back, but that doesn’t necessarily help the thousands of businesses affected by the glitch.

Video above: How to protect yourself after global outage

Advertisement

The CrowdStrike software issue at the heart of the outage runs at such a deep level in affected computers and systems that getting them up and running just to be fixed will be, in many cases, an enormous challenge.

That’s compounded by the fact that many of the servers that may contain information needed to get these systems working again are themselves caught in a cycle of crashing and rebooting.

"I don’t think it's too early to call it: this will be the largest IT outage in history," said security expert Troy Hunt in a post on X.

The CrowdStrike software at fault operates at a computer’s kernel level, a much deeper level than what more ordinary applications such as browsers or video games do. This portion of a device has much greater visibility and control over a computer and its components, making it critical for the operation of all other systems and far more sensitive.

Running at the kernel level means CrowdStrike's software can do more to detect cyberattacks, but it also means the current bug is causing Windows computers to crash to a Blue Screen of Death before users can take any actions to correct it.

The issue appears to be recoverable, CrowdStrike has said, but in many cases it requires painstaking work: Each affected device must be accessed by an administrator and manually rebooted into safe mode. Then, the offending CrowdStrike file must be deleted by hand.

For businesses with hundreds or thousands of laptops, desktops and servers running CrowdStrike’s security software, an individual human may have to perform that process over and over and over again.

"You can’t automate that," said Kevin Beaumont, a security researcher and former Microsoft threat analyst, in a post on X. "So this is going to be incredibly painful for CrowdStrike customers."

Video above: CrowdStrike CEO says company has 'deployed a fix' to tech issue

It gets worse.

Organizations that take security seriously will have likely encrypted their computers’ hard drives, making it even more challenging to access the file that needs to be deleted.

For those organizations, "you need to manually decrypt the disk with a BitLocker Recovery Key, which is probably — for most companies — stored digitally on one of the servers that is currently booting over and over," said Ira Bailey, a security researcher, in a post on BlueSky.

Every affected computer that is BitLocker-encrypted will need to be unlocked with a recovery key before organizations can begin the process of deleting the bad CrowdStrike file and restoring normal operation, said the cybersecurity expert who goes by the pseudonymous handle SwiftOnSecurity in a post on X.

Recovery will be enormously expensive for Fortune 500 companies with large teams of IT staff and likely even more challenging for smaller firms, Kenn White, an independent security researcher who specializes in network security, told CNN.

"If you don’t have physical staff that can actually touch it, this is going to take many, many days for much of corporate America to recover from," White said. "It's just a ton of labor-intensive manual work."

"It’s a fairly complicated procedure for non-technical people," White added, "and even a lot of skilled IT professionals will find it difficult to do this at the scale that’s going to be required given the number of machines that are affected."

How did the CrowdStrike bug lead to such widespread effects?

Because CrowdStrike’s security software is running on countless individual computers all around the globe, the update that got pushed to those devices caused them all to shut down, virtually simultaneously.

And in today’s networked economy, an outage in one part of a supply chain can cause domino effects up and down the line. When multiple parts of a supply chain go down, it touches off a cascade of problems.

Imagine a person trying to buy a coffee, said Andrew Peck, a cybersecurity expert at Loughborough University in the UK. What may seem like a simple transaction relies on multiple computers working in tandem, from the coffee shop’s point of sale to the payment processor’s own back-end systems.

Video above: Starbucks, hotels, businesses affected by CrowdStrike outage

"There are a lot of computers in this chain, and usually the larger the business, the larger the chain," Peck said. "If any one of the computers are down in the chain, the transaction will not complete."

Now scale that up to something like the massive aviation industry, the critical financial services sector or the life-or-death operations of a health care provider, and the scope of the disaster becomes starkly clear.

What is Microsoft’s role in all this?

A separate issue earlier, on Thursday, did lead to significant impacts on many of Microsoft’s own cloud customers, but it was resolved overnight and was unrelated to the CrowdStrike issue, multiple cybersecurity experts said.

The CrowdStrike bug may have initially been conflated with the Microsoft issue because CrowdStrike’s error affected only Windows machines.

“Both are Microsoft-related, but Microsoft had nothing to do with the second incident,” White told CNN.

That appears to be supported by Microsoft’s own status account on X, which on Thursday announced an issue affecting "Microsoft 365 apps and services" and a separate announcement Friday addressing the CrowdStrike outage. The two issues are being tracked using different reference numbers.

As of Friday morning, Microsoft said the issue with Microsoft 365 had been resolved and that the situation was improving. Microsoft didn’t immediately respond to a request for comment.

Since the update to CrowdStrike’s software was delivered by the company’s own systems, it appears unlikely that Microsoft bears direct responsibility for Friday’s outages, said Beaumont, who said he reviewed a copy of CrowdStrike’s flawed update.

The problem with CrowdStrike’s update was that it wasn’t formatted correctly “and causes Windows to crash every time,” Beaumont posted on X.

CNN’s Olesya Dmitracova contributed reporting.