Rantburg

Today's Front Page   View All of Wed 05/28/2025 View Tue 05/27/2025 View Mon 05/26/2025 View Sun 05/25/2025 View Sat 05/24/2025 View Fri 05/23/2025 View Thu 05/22/2025
2021-10-05 Cyber
Understanding How Facebook Disappeared from the Internet
[Cloudflare] “Facebook can't be down, can it?”, we thought, for a second.

Today at 15:51 UTC, we opened an internal incident entitled "Facebook DNS lookup returning SERVFAIL" because we were worried that something was wrong with our DNS resolver 1.1.1.1. But as we were about to post on our public status page we realized something else more serious was going on.

Social media quickly burst into flames, reporting what our engineers rapidly confirmed too. Facebook and its affiliated services WhatsApp and Instagram were, in fact, all down. Their DNS names stopped resolving, and their infrastructure IPs were unreachable. It was as if someone had "pulled the cables" from their data centers all at once and disconnected them from the Internet.

How's that even possible?

Update from Facebook
Facebook has now published a blog post giving some details of what happened internally. Externally, we saw the BGP and DNS problems outlined in this post but the problem actually began with a configuration change that affected the entire internal backbone. That cascaded into Facebook and other properties disappearing and staff internal to Facebook having difficulty getting service going again.

Now on to what we saw from the outside.

Meet BGP
BGP stands for Border Gateway Protocol. It's a mechanism to exchange routing information between autonomous systems (AS) on the Internet. The big routers that make the Internet work have huge, constantly updated lists of the possible routes that can be used to deliver every network packet to their final destinations. Without BGP, the Internet routers wouldn't know what to do, and the Internet wouldn't work.

The Internet is literally a network of networks, and it’s bound together by BGP. BGP allows one network (say Facebook) to advertise its presence to other networks that form the Internet. As we write Facebook is not advertising its presence, ISPs and other networks can’t find Facebook’s network and so it is unavailable.

The individual networks each have an ASN: an Autonomous System Number. An Autonomous System (AS) is an individual network with a unified internal routing policy. An AS can originate prefixes (say that they control a group of IP addresses), as well as transit prefixes (say they know how to reach specific groups of IP addresses).

Cloudflare's ASN is AS13335. Every ASN needs to announce its prefix routes to the Internet using BGP; otherwise, no one will know how to connect and where to find us.

Our learning center has a good overview of what BGP and ASNs are and how they work.

In this simplified diagram, you can see six autonomous systems on the Internet and two possible routes that one packet can use to go from Start to End. AS1 → AS2 → AS3 being the fastest, and AS1 → AS6 → AS5 → AS4 → AS3 being the slowest, but that can be used if the first fails.

Read the rest at the link
Posted by badanov 2021-10-05 05:13|| || Front Page|| [11129 views ]  Top

#1 Best related Facebook Meme so far.
Posted by NN2N1 2021-10-05 05:59||   2021-10-05 05:59|| Front Page Top

#2 heh
Posted by The Walking Unvaxed 2021-10-05 06:19||   2021-10-05 06:19|| Front Page Top

#3 MISC Data Bits collected from sources:

Key part of Cover Story:
Problems began with a routine BGP update that went wrong, wiping out the DNS routing information that Facebook needed.

Facebook-Group down 6+/- Hours.

It was reported by a few anon staff Facebook sent a CERT Team to its Cal. data site to address the server issues and had badge access problems, requiring landline comm verification for access.
It was verified the Facebook outage also lockout many employees from communicating with each another using their internal Facebook tools. FB employs a LARGE number of HOME WORKERS.

Outage coincidentally happened around CBS’s 60 Minutes release of the interview with Frances Haugen (Facebook Whistle Blower). She is scheduled to testify in the US Senate Oct. 5th. regarding FB User safety and Political abuses in 2020.

Sources speculate if the 2019-2020 political censorship engagement data trail/files were "lost" during the "crash".

A few dark ch. sources reported big offers of $$$$$$$/ employment to the person or persons responsible for the outage were authorized. Which fueled initial reports of a DNS Hack. IT Security Expert Madory said. “We obviously can’t rule out someone hacking them, but they also could have done this to themselves.”

Its estimated that Facebook and its acquisitions lost about $5.9B in related revenue during that 6+/- hrs.

FB stock price fell roughly $20 a share in 6 hrs

Several different domain name registers listed the domain Facebook.com as up for sale. Yep you could have been the owner of Facebook.com for a few hours ☺☺☺

Either way the X & Millennium generations got to exp. a sample of 1960 retro communications
Posted by NN2N1 2021-10-05 07:21||   2021-10-05 07:21|| Front Page Top

#4 This looks like some powerful player identified (or created) a single point of failure for one of the internet's major infrastructure hubs and then pulled the trigger.

<speculation>
I wonder if this Facebook trouble was Russia's strategic retaliation/shot across the bow for Pandora.
</speculation>
Posted by Elmerert Hupens2660 2021-10-05 07:22||   2021-10-05 07:22|| Front Page Top

#5 Stop speculating it was some sort of attack. It was a screwed up update. Sadly, it happens.
Posted by Rob Crawford 2021-10-05 07:37||   2021-10-05 07:37|| Front Page Top

#6 In other, highly technical, words: Oops!
Posted by CrazyFool 2021-10-05 08:44||   2021-10-05 08:44|| Front Page Top

#7 wish it had stayed down for good.
Posted by Chris 2021-10-05 08:58||   2021-10-05 08:58|| Front Page Top

#8 Facebook whistleblower to appear before Senate, compare company to big tobacco
Posted by Skidmark 2021-10-05 09:31||   2021-10-05 09:31|| Front Page Top

#9 "Blundered Server Update"
Posted by Skidmark 2021-10-05 09:52||   2021-10-05 09:52|| Front Page Top

#10 

OK I'll ask...
Having been in I.T. relate operations for over 35 years.
Our common sense roll out practice was always to BETA TEST ANY UPDATE .....BEFORE..... to going operational.

Are we to believe a F-100 company does not practice common basic operational safety roll out steps?

Hence why I feel there is more to the story.
Posted by NN2N1 2021-10-05 11:03||   2021-10-05 11:03|| Front Page Top

#11 I developed a network for a Paradyne product in the 1970's. And it was impervious to failure because all the nodes continuously updated their routing paths. That was a mickey-mouse operation. Am i to believe that the internet is dependent on a single node?
Posted by irish rage boy  2021-10-05 12:00||   2021-10-05 12:00|| Front Page Top

#12 It's hard to test anything as core as naming and routing -- do you build a separate Internet for testing? -- so mostly you do with layers of reviews. Sometimes mistakes still filter through.

This one sounds like someone didn't do any immediate checking, and let an error propagate for a while.

And, no, the Internet isn't dependent on a single node -- but when all nodes are getting the same (wrong) answer...
Posted by Rob Crawford 2021-10-05 12:48||   2021-10-05 12:48|| Front Page Top

#13 Insightful comment from old timer Internet Engineer Bill Woodcock on KrebsOnSecurity.com:

"In old-school networking, we had literally dozens of routing protocols to choose from, and as we had a wealth of layer-1 and layer-2 protocols to choose from. So the chances that an organization would be running the same IGP (internal gateway protocol) and EGP (external gateway protocol) were essentially nil; one chose best-of-breed for each independently, and they have very different needs.

"By about 1996, the EGP choice had devolved into a monoculture of people running BGP, and that makes sense, because convergence on a single protocol is driven by the Network Effect, just as a common trading language makes it possible to do business across borders… Knowing that you’d be using the same routing protocol with all your external neighbors decreased the time and effort needed to reach the goal. Unfortunately, when we reached an EGP monoculture, competitive pressure also went away. We were using BGPv4 in 1996, and we’re still using the same version now today, twenty-five years later, with literally no significant improvement or development to the protocol. And although it played no part in yesterday’s incident, the thin layer of security that’s added on to the side of BGPv4 to try to remedy its worst deficiencies has actually _lost_ significant ground since its peak in about 2003-2004.

"It took longer for IGPs to die out, and when they did, the monoculture that took hold was in the form of “iBGP,” which meant simply using BGP for internal routing as well as external routing, even though it was a technically much worse choice for the purpose… and I’m afraid I was one of the people who pushed that forward, in the name of efficiency and, frankly, laziness. A lot of the poor Internet engineering choices that we’re now living with the consequences of were done in the name of laziness, in times when businesses were striving for exponential growth, money was plentiful, but time and attention were scarce and growing scarcer. So today, essentially _every_ network of global scale runs BGP as both their external and internal routing protocol

"And it’s in this environment that a problem like the one Facebook had becomes possible. Pushed to simultaneously grow and cut costs, their engineering departments are hollowed out, losing the entire middle to a bipolar structure of a very small handful of the same old farts that I was working with when they were kids, thirty years ago, who actually know how things work, and a lot of actual kids who are paid a pittance and given no meaningful path to intellectual and career growth; where before they’d have had mentors and a ladder of ever-more-challenging work to make their way up, now their growth path is blocked by automation. “Dev Ops” automation that allows the attention of a team of two or three or four people to be spread across thousands of routers and tens or hundreds of thousands of servers through abstracted fleet management, using the kids for “rack & stack” grunt-work, where they never get to touch anything that they could break, but also never get to learn how it works. Like I said, I’m dealing with exactly the same small set of people now, in my counterpart networks, that I was thirty years ago… but some of them are starting to retire, others are wanting more time with family, and we’re all just getting old; this isn’t sustainable. So, in this environment of too little attention from too few senior people stretched too thin, and using too many abstraction layers of automation that was all-too-often written for unrelated purposes, you get own-goals like yesterday’s, where the senior people have to dig through layers and layers of crufty scripts in half a dozen languages (an area where there has _not_ been useful consolidation) to find and remediate the problem, while the junior folks are _literally_ locked out and unable to help.

"It’s worthy of note that this monoculture/stagnation process has certainly not been limited to routing protocols; where we had many competing and improving layer-2 protocols, since about 2001 they’ve pretty much all been squeezed out by Ethernet in its many, equally mediocre, forms. There used to be dozens of email formats, with a rich market in gateways between; since that competition was eliminated, email has become homogenous and an efficient home for spammers. It’s precisely this trend toward stagnation in open standards and open protocols that has made the predatory centralization of proprietary garbage like Gmail and Office365 and Facebook possible… When there were alternatives, nobody would ever have given such half-assed things a second glance."
Posted by Classical_Liberal 2021-10-05 12:57||   2021-10-05 12:57|| Front Page Top

#14 Fascinating, Classical_Liberal.
Posted by trailing wife 2021-10-05 13:03||   2021-10-05 13:03|| Front Page Top

#15 What of the reports of employees not even able to gain access to the building so as actually work on the situation?
Posted by Rex Mundi 2021-10-05 14:51||   2021-10-05 14:51|| Front Page Top

#16 DevOps is where the adults work. The other software dev teams are full of young H1B drones who know little and have next to no experience to draw upon.

Care to guess which group gets executive attention and lavish resources, and which is starved?
Posted by Sheba tse Tung4690 2021-10-05 15:07||   2021-10-05 15:07|| Front Page Top

#17 Worked with a Dev ops team from Adobe while at Scripps. All - ALL - were H1Bs. I was QA so we were treated even worse.
Posted by Rex Mundi 2021-10-05 15:26||   2021-10-05 15:26|| Front Page Top

#18 In other news ‘American corporate productivity spiked yesterday by 300%’.
Posted by Airandee 2021-10-05 18:07||   2021-10-05 18:07|| Front Page Top

#19 Proof that the entire internet can be shut down for an extended period. Crippling many businesses that do financial transactions on line, etc. during that time.
Posted by Blackbeard Barnsmell6454 2021-10-05 18:16||   2021-10-05 18:16|| Front Page Top

11:07 HeavyG
11:07 Angstrom
11:06 DarthVader
11:05 HeavyG
10:58 alanc
09:43 Mullah Richard
09:27 Warthog
09:11 Mercutio
09:07 AlmostAnonymous5839
08:52 Matt
08:24 Matt
08:20 SteveS
07:43 Procopius2k
07:42 BrerRabbit
07:42 Procopius2k
07:39 Procopius2k
07:36 Procopius2k
07:35 Procopius2k
07:34 trailing wife
07:31 Procopius2k
07:30 NN2N1
07:22 NN2N1
07:18 trailing wife
07:14 Richard Aubrey









Paypal:
Google
Search WWW Search rantburg.com