Article

2021-10-05 Cyber

Understanding How Facebook Disappeared from the Internet

Archived material is restricted to Rantburg regulars and members. If you need access email fred.pruitt=at=gmail.com with your nick to be added to the members list. There is no charge to join Rantburg as a member.

Posted by badanov 2021-10-05 05:13|| || Front Page|| [8 views ] Top

#1 Best related Facebook Meme so far.

Posted by NN2N1 2021-10-05 05:59|| 2021-10-05 05:59|| Front Page Top

#2 heh

Posted by The Walking Unvaxed 2021-10-05 06:19|| 2021-10-05 06:19|| Front Page Top

#3 MISC Data Bits collected from sources:

Key part of Cover Story:
Problems began with a routine BGP update that went wrong, wiping out the DNS routing information that Facebook needed.

Facebook-Group down 6+/- Hours.

It was reported by a few anon staff Facebook sent a CERT Team to its Cal. data site to address the server issues and had badge access problems, requiring landline comm verification for access.
It was verified the Facebook outage also lockout many employees from communicating with each another using their internal Facebook tools. FB employs a LARGE number of HOME WORKERS.

Outage coincidentally happened around CBS’s 60 Minutes release of the interview with Frances Haugen (Facebook Whistle Blower). She is scheduled to testify in the US Senate Oct. 5th. regarding FB User safety and Political abuses in 2020.

Sources speculate if the 2019-2020 political censorship engagement data trail/files were "lost" during the "crash".

A few dark ch. sources reported big offers of $$$$$$$/ employment to the person or persons responsible for the outage were authorized. Which fueled initial reports of a DNS Hack. IT Security Expert Madory said. “We obviously can’t rule out someone hacking them, but they also could have done this to themselves.”

Its estimated that Facebook and its acquisitions lost about $5.9B in related revenue during that 6+/- hrs.

FB stock price fell roughly $20 a share in 6 hrs

Several different domain name registers listed the domain Facebook.com as up for sale. Yep you could have been the owner of Facebook.com for a few hours ☺☺☺

Either way the X & Millennium generations got to exp. a sample of 1960 retro communications

Posted by NN2N1 2021-10-05 07:21|| 2021-10-05 07:21|| Front Page Top

#4 This looks like some powerful player identified (or created) a single point of failure for one of the internet's major infrastructure hubs and then pulled the trigger.

<speculation>
I wonder if this Facebook trouble was Russia's strategic retaliation/shot across the bow for Pandora.
</speculation>

Posted by Elmerert Hupens2660 2021-10-05 07:22|| 2021-10-05 07:22|| Front Page Top

#5 Stop speculating it was some sort of attack. It was a screwed up update. Sadly, it happens.

Posted by Rob Crawford 2021-10-05 07:37|| 2021-10-05 07:37|| Front Page Top

#6 In other, highly technical, words: Oops!

Posted by CrazyFool 2021-10-05 08:44|| 2021-10-05 08:44|| Front Page Top

#7 wish it had stayed down for good.

Posted by Chris 2021-10-05 08:58|| 2021-10-05 08:58|| Front Page Top

#8 Facebook whistleblower to appear before Senate, compare company to big tobacco

Posted by Skidmark 2021-10-05 09:31|| 2021-10-05 09:31|| Front Page Top

#9 "Blundered Server Update"

Posted by Skidmark 2021-10-05 09:52|| 2021-10-05 09:52|| Front Page Top

#10

OK I'll ask...
Having been in I.T. relate operations for over 35 years.
Our common sense roll out practice was always to BETA TEST ANY UPDATE .....BEFORE..... to going operational.

Are we to believe a F-100 company does not practice common basic operational safety roll out steps?

Hence why I feel there is more to the story.

Posted by NN2N1 2021-10-05 11:03|| 2021-10-05 11:03|| Front Page Top

#11 I developed a network for a Paradyne product in the 1970's. And it was impervious to failure because all the nodes continuously updated their routing paths. That was a mickey-mouse operation. Am i to believe that the internet is dependent on a single node?

Posted by irish rage boy 2021-10-05 12:00|| 2021-10-05 12:00|| Front Page Top

#12 It's hard to test anything as core as naming and routing -- do you build a separate Internet for testing? -- so mostly you do with layers of reviews. Sometimes mistakes still filter through.

This one sounds like someone didn't do any immediate checking, and let an error propagate for a while.

And, no, the Internet isn't dependent on a single node -- but when all nodes are getting the same (wrong) answer...

Posted by Rob Crawford 2021-10-05 12:48|| 2021-10-05 12:48|| Front Page Top

#13 Insightful comment from old timer Internet Engineer Bill Woodcock on KrebsOnSecurity.com:

"In old-school networking, we had literally dozens of routing protocols to choose from, and as we had a wealth of layer-1 and layer-2 protocols to choose from. So the chances that an organization would be running the same IGP (internal gateway protocol) and EGP (external gateway protocol) were essentially nil; one chose best-of-breed for each independently, and they have very different needs.

"By about 1996, the EGP choice had devolved into a monoculture of people running BGP, and that makes sense, because convergence on a single protocol is driven by the Network Effect, just as a common trading language makes it possible to do business across borders… Knowing that you’d be using the same routing protocol with all your external neighbors decreased the time and effort needed to reach the goal. Unfortunately, when we reached an EGP monoculture, competitive pressure also went away. We were using BGPv4 in 1996, and we’re still using the same version now today, twenty-five years later, with literally no significant improvement or development to the protocol. And although it played no part in yesterday’s incident, the thin layer of security that’s added on to the side of BGPv4 to try to remedy its worst deficiencies has actually _lost_ significant ground since its peak in about 2003-2004.

"It took longer for IGPs to die out, and when they did, the monoculture that took hold was in the form of “iBGP,” which meant simply using BGP for internal routing as well as external routing, even though it was a technically much worse choice for the purpose… and I’m afraid I was one of the people who pushed that forward, in the name of efficiency and, frankly, laziness. A lot of the poor Internet engineering choices that we’re now living with the consequences of were done in the name of laziness, in times when businesses were striving for exponential growth, money was plentiful, but time and attention were scarce and growing scarcer. So today, essentially _every_ network of global scale runs BGP as both their external and internal routing protocol

"And it’s in this environment that a problem like the one Facebook had becomes possible. Pushed to simultaneously grow and cut costs, their engineering departments are hollowed out, losing the entire middle to a bipolar structure of a very small handful of the same old farts that I was working with when they were kids, thirty years ago, who actually know how things work, and a lot of actual kids who are paid a pittance and given no meaningful path to intellectual and career growth; where before they’d have had mentors and a ladder of ever-more-challenging work to make their way up, now their growth path is blocked by automation. “Dev Ops” automation that allows the attention of a team of two or three or four people to be spread across thousands of routers and tens or hundreds of thousands of servers through abstracted fleet management, using the kids for “rack & stack” grunt-work, where they never get to touch anything that they could break, but also never get to learn how it works. Like I said, I’m dealing with exactly the same small set of people now, in my counterpart networks, that I was thirty years ago… but some of them are starting to retire, others are wanting more time with family, and we’re all just getting old; this isn’t sustainable. So, in this environment of too little attention from too few senior people stretched too thin, and using too many abstraction layers of automation that was all-too-often written for unrelated purposes, you get own-goals like yesterday’s, where the senior people have to dig through layers and layers of crufty scripts in half a dozen languages (an area where there has _not_ been useful consolidation) to find and remediate the problem, while the junior folks are _literally_ locked out and unable to help.

"It’s worthy of note that this monoculture/stagnation process has certainly not been limited to routing protocols; where we had many competing and improving layer-2 protocols, since about 2001 they’ve pretty much all been squeezed out by Ethernet in its many, equally mediocre, forms. There used to be dozens of email formats, with a rich market in gateways between; since that competition was eliminated, email has become homogenous and an efficient home for spammers. It’s precisely this trend toward stagnation in open standards and open protocols that has made the predatory centralization of proprietary garbage like Gmail and Office365 and Facebook possible… When there were alternatives, nobody would ever have given such half-assed things a second glance."

Posted by Classical_Liberal 2021-10-05 12:57|| 2021-10-05 12:57|| Front Page Top

#14 Fascinating, Classical_Liberal.

Posted by trailing wife 2021-10-05 13:03|| 2021-10-05 13:03|| Front Page Top

#15 What of the reports of employees not even able to gain access to the building so as actually work on the situation?

Posted by Rex Mundi 2021-10-05 14:51|| 2021-10-05 14:51|| Front Page Top

#16 DevOps is where the adults work. The other software dev teams are full of young H1B drones who know little and have next to no experience to draw upon.

Care to guess which group gets executive attention and lavish resources, and which is starved?

Posted by Sheba tse Tung4690 2021-10-05 15:07|| 2021-10-05 15:07|| Front Page Top

#17 Worked with a Dev ops team from Adobe while at Scripps. All - ALL - were H1Bs. I was QA so we were treated even worse.

Posted by Rex Mundi 2021-10-05 15:26|| 2021-10-05 15:26|| Front Page Top

#18 In other news ‘American corporate productivity spiked yesterday by 300%’.

Posted by Airandee 2021-10-05 18:07|| 2021-10-05 18:07|| Front Page Top

#19 Proof that the entire internet can be shut down for an extended period. Crippling many businesses that do financial transactions on line, etc. during that time.

Posted by Blackbeard Barnsmell6454 2021-10-05 18:16|| 2021-10-05 18:16|| Front Page Top

22:49 JohnQC
22:42 JohnQC
22:19 rjschwarz
22:15 rjschwarz
22:04 Thing From Snowy Mountain
22:02 Sheba tse Tung4690
21:51 Woodrow
21:33 Rob Crawford
21:32 trailing wife
21:03 SteveS
20:33 Rex Mundi
20:32 746
20:25 The Walking Unvaxed
20:04 Don Vito Elmart2595
19:42 Angstrom
19:04 Sheba tse Tung4690
18:54 Rex Mundi
18:50 swksvolFF
18:34 Fred
18:16 Blackbeard Barnsmell6454
18:07 Airandee
17:54 JohnQC
17:47 Cesare
17:46 JohnQC