Back on Christmas Eve of last year there were some reports that Elon Musk was in the process of shutting down Twitter’s Sacramento data center. In that article, a number of ex-Twitter employees wer…
Mark Beaks from Disney’s 2017 Ducktales remake was ahead of its time with regards to asshole Silicon Valley type CEOs, and obviously has a few riffs on Elon Musk as well. www.youtube.com/watch?v=4iPhrnUtQ5U
Ducktales turned out really well. You’ve got the long-running protagonist Billionaire, Scrooge McDuck and his long-time rival Billionaire Glomgold. However, as a 2017 reimagining, they added the Silicon Valley type Mark Beaks and it was chef’s kiss perfect. Old money vs new money is still a relevant subject a hundred-years after the “Gilded Age” of the 1920s / 1930s, that these characters were originally based on.
Funny thing is this kind of behaviour isn’t unique to Musk. A lot of entrepreneurs and CEOs seem to have similar kind of attitude. They want everything done cheaper faster and there’s no 2 ways about it. It’s their way or highway. If shit goes to hell it’s other people’s heads that roll.
It’s frustrating they don’t even know what to be angry about. Like instead of flying to Sacramento and ripping out 5000 servers why not flip out that the code has 70,000 different hard coded references to a single data center instead of one.
This is exactly why our last governor here in Illinois, didn’t CEO Governor Rauner, was probably the most ineffectual governor of our state ever had. He literally had no capacity for compromise, and was a Republican, in an Illinois where the legislature has long been solidly Democrat. I can’t think of a single thing he actually got done. All because he was a CEO trying to run Illinois like a business.
Yeah, that’s basically why I got out of IT. Too many managers/clients refusing to listen to warnings about what would happen when they did X, then blaming the techies when things went to shit.
Because they are the “boss”, they have 0 accountability. Worst case for them is a golden handshake, and failing upwards where the cycle starts again.
At the risk of sounding like an apologist for this prat, the frustration at being told it will take months to move systems is understandable. Also, the idiot developers who hard-coded the data center location deserve to be fired. Data center floor tiles can be removed easily with a flat blade as a lever instead of using suction cups.
Obviously a coke-fuelled man-child doing it in the middle of the night is ridiculous but if you have unlimited resources you can move any number of servers in a few days, easily. In some ways it’s impressive that he was able to pull this off on a whim without a catastrophe (the DeSantis fiasco notwithstanding). It definitely should not have been suggested by a competent data center engineer that it would take months to move anything if the CEO wants it in weeks. Even though he’s an ass, I don’t blame him for being annoyed about that.
Remember he just gutted Twitter before pulling this stunt, so the estimate of a few months to move the servers might be true if the entire department that handle infra has been gutted with only skeleton crew left.
Well, yes. If nobody left has a clue then it’s going to take a little longer but you could physically move just about anything in a few weeks with the right crew, even if you had to bring them in cold. An open checkbook solves a lot of logistical problems.
The proof is more or less self evident. If this idiot and his cousin were able to pull it off without breaking anything critical, then it stands to reason that a properly managed team would have been able to do it in a more orderly way in a few days.
I get that everyone wants to paint this as completely irresponsible, but apart from the fact that it was done so haphazardly in the dead of night and gate crashing the data center security (nobody is going to refuse access to the CEO), there’s really nothing here that’s completely out of order. Locking the gear in the trucks is pretty standard for intact secure data transport. The real mistake is the infra manager sandbagging the move estimate - or not understanding how to plan and execute it.
Physically move them is one thing. Reassigning each server into the new data center network is a whole other thing. It won’t be as simple as connecting the power and network cables. From the post, the rack density is different so you’ll probably have to change the each server name to match the new rack position. Then the hostname and subnet probably changes in the new data center, so now you’ll have to map everything again (the hard coded references to Sacramento mentioned by Musk). The 100MM contract means they have a lot of servers to account for. This is the real headache of the migration and probably the reason Twitter keep having random outages for months after this stunt. They probably took shortcut and can’t bring all those servers online in time to handle traffic bursts which leads to another Musk’s shenanigans (e.g. forbidding visitors from viewing tweets unless they’re logged in to limit servers load, etc).
Edit: the more I think about this, the more my head hurt. If any infra people reading this, what are you going to do if you suddenly received truckloads of servers yanked from another data center location and told to bring them online again ASAP, while more than half of your team has been laid off? Seriously, what’s the step you’re gonna do to bring all these servers online again? Oh, and those servers probably not gracefully shut down and just have their power cable yoinked off.
Oh by no means am I suggesting it was reasonable to do this. Musk would be a fucking nightmare as an employer. As a customer probably not much better but you know what they say about a fool and his money. This fool would be a great customer as long as you had a good lawyer to write the contracts.
I do suspect that some of the details of this story are somewhat embellished though, if only for the sheer joy of it, which I’m all for. It’s a great story. I don’t believe, for instance, that they could possibly have moved 5000 racks - or even 5000 servers - as I think the story was intimating. It sounds like they filled a few semis, which would be a small fraction of the systems. Maybe this was just the last of it that was too hard to move earlier. As for the rack configs at the other end, they would need power and services and an empty space if they are just rolling the stuff in. That’s only a few weeks of lead time in a properly run facility.
If they had their reservations set up correctly they wouldn’t need to change hostnames or even addresses, just wheel in the racks, brace and connect them. Ideally stuff would be shut down gracefully, but it shouldn’t really matter if they just pulled the plug. The software should be resilient enough to restart ok. Again, no idea if they had anything thought out, probably not, given the way it was done. But I have seen a big tech co move several rows this way when they basically couldn’t be bothered figuring out how to logically migrate them. Of course they weren’t doing it with a coked up CEO at 2am on Christmas Eve, but it wasn’t as difficult as you might imagine. But yeah not 5000 racks at once. Not even close to possible.
They were somewhere over Las Vegas when James made his suggestion that they could move them now. It was the type of impulsive, impractical, surge-into-the-breach idea that Musk loved.
Musk and his renegade team were rolling servers out without putting them in crates or swaddling them in protective material, then using store-bought straps to secure them in the truck. “I’ve never loaded a semi before,” James admitted.
The moving contractors that NTT wanted them to use charged $200 an hour. So James went on Yelp and found a company named Extra Care Movers that would do the work at one-tenth the cost.
The servers had user data on them, and James did not initially realize that, for privacy reasons, they were supposed to be wiped clean before being moved. … So James sent someone to Home Depot to buy big padlocks, and they sent the combination codes on a spreadsheet to Portland so the trucks could be opened there. “I can’t believe it worked,” James says.
LMAO who’s this James guy and why does he understand Musk so thoroughly like his own spouse?
To be fair, that seems to be the best case scenario for nepotism. Imagine if the daughter and her boyfriend were put into positions of real power without the expertise to back it up. The dad seems to understand and put them kids into harmless job role.
On CHRISTMAS FUCKING EVE! He has like 10 kids. He started a fire drill for employees on Christmas Eve, they have families too. What a cartoonishly villainous thing to do.
I’m shocked that the data center required retinal scans but that the employee with access could then just hold the door and let him and others in.
I used to work at a data center with lots of security. To get into the area with the servers you had to go through a man trap. It was a room a little larger than a telephone booth with automatic doors on both sides. To open the first door you needed a physical card key. Once inside the door closed, then to open the inner door you needed to both enter a PIN and have your hand scanned in a biometric scanner. Only after all that could you get inside. The booth also weighed you, and if your weight was off by a certain amount after your last pass through then it wouldn’t let you in. That was to prevent somebody from piggybacking with you.
There was third party security at this particular facility. You had to show your ID, have them confirm you’re authorized to bring equipment in, then they’d bring it through a locked freight door for you.
Separate double bay doors. They have a pair for each floor that opens to an outside wall. You use a forklift to get the pallets up. That or there is a big ass freight elevator, depending on the data center.
I’m sure there were other, larger, entry points that could be opened for moving equipment in and out. They would then be locked down during normal operation.
Even the smaller data center I used to go to would have an alarm go off if the door was open for more than a few seconds. The first door opened with your hand being scanned and the cage to our racks could be opened with a key card.
While this is of course true, to me part of saying him no is not giving him media attention. If his stunts didn’t have the publicity they do, he might not actually do them. If people stopped caring about whatever shit he writes on Twitter then he wouldn’t have a power to destroy Starlink’s government deal or affect stock values.
…so you sold your Tesla stock before it was even a public company or sold a single car? Stock that you held for “years” when the company itself was less than a year old?
lol what bullshit. Elon Musk was literally one of the first and biggest investor in the company before they even began designing their first car. That’s when he “took over” (and thus got to be named as a founder)
I am all for billionaires wanting go to Mars. They will just be on their own, getting back. Oopise daisy, the rocket only had enough fuel go to Mars and not back. If they want to come back, well they all that wealth to pay us and we’ll get them, one day…soon…eventually…maybe…probably not.
Lmao, Musk is the new Gary Young, the supplment king. Behind the Bastards did an episode on him, he really wanted to be a surgeon without that pesky medical license requirement.
Elon Musk is a privileged manchild who never grew out of his teenager phase, throwing around his inherited wealth like the kid from Blank Check and throwing temper tantrums anytime someone calls him out on his bullshit. Any claims to success he may have had been entirely in spite of him, not because of him. He doesn’t have any fucking idea what he’s doing and if any one of you or I failed even a fraction as much as Musk had, we’d have all been fired ten times over.
They’re “successful” if you define success as being born into wealth and/or being willing to ignore the suffering of others. For you and me (and most right thinking people), that’s no real measure of success. Unfortunately, capitalism and the US political systems gives power to the people least suited to wielding it.
Add comment