One ISP's 600,000 Dead Routers

A curious cyberattack (maybe) takes out Windstream customers.

This by Dan Goodin was forward to me by a friend, with their only comment being a large supply of question marks, apparently worrying that I’d run out of them, just as Windstream did with replacement customer routers.

Mystery malware destroys 600,000 routers from a single ISP during 72-hour span

Size Matters:

That’s a big number, but Windstream covers a big area. How many customers do they have in total? Examining AS7029 using Hurricane Electric’s BGP Looking Glass we find that they originate twelve million IPv4 addresses. They have IPv6 prefixes as well, but HE does not deign to offer a total. Keep in mind that those are not mutually exclusive - ISPs often have IPv4 and IPv6 available on the same network.

Every customer has at least one public IP address, but not every IP address has a customer. They are handled in blocks of 256, read The Shape of Cyberspace if you aren’t up on this. The Windstream Holdings Wikipedia says they have 8.1 million customers so a two thirds density in the public IPv4 blocks? As a former operator of a network that competed with Windstream’s predecessor in Iowa, with plenty of experience filling out ARIN paperwork, this makes sense to me.

So someone felt the need to whack 7.5% of Windstream’s customer base, starting from the very bottom, and the big question is … why?

Explanations:

I read an article so I don’t know what happened, but I did a few quick Google searches and found this on Windstream foreign ownership after their 2020 bankruptcy.

Another ownership group is buying into Windstream Holdings Inc., which was acquired by a New York hedge fund three years ago and now is taking on foreign investors to complete ownership reorganization of the Little Rock-based communications provider.

The Federal Communications Commission on Friday approved Windstream's proposal to sell a large stake in the company to Nexus Aggregator, which will own 49.27% of Windstream. FCC approval was required to exceed a 25% ownership benchmark set by federal regulators.

The FCC's approval noted that a total of 66.29% of Windstream will be foreign-owned, including entities tied to Elliott Management Corp., the hedge fund that purchased Windstream's distressed assets in bankruptcy in 2020. That was step one in the reorganization, which allowed Windstream to emerge from bankruptcy as a privately held company with reduced debt and a commitment from the new owners to invest $750 million to expand operations.

"We find that public interest would not be served by prohibiting foreign ownership of Windstream," the FCC said in Friday's order. That approval completed step two of the reorganization, which relied on the foreign investment.

"This order is the last step necessary to approve Windstream's ownership structure in relation to its emergence" from bankruptcy, Kristi Moody, the company's general counsel, said in a statement Monday.

The FCC requires that telecommunications carriers in the U.S. have less than 25% ownership by foreign investors. Windstream's investors have corporations in Canada, Australia, Germany and the Cayman Islands, according to Moody.

Windstream provides voice, data and transport services over a fiber network.

So who are those foreign owners? I haven’t looked and I probably won’t, but this is an area ripe for exploration.

---

The ActionTec T3200 is a two wire DSL termination device. The suspicious ISP plant engineer in me now asks “7.5% of total customers, but what percentage of Windstream’s total DSL customers just got a forced upgrade?”

Here in the Bay Area I’ve seen someone who was set in their ways have a DSL line go down and they spent ten days calling for repairs. I’m not a yeller as a rule but I finally raised my voice, using driving words starting with “dumbfuck” and escalating from there. The dumbfuck, and I say this in the most loving way possible, quickly discovered cable modems were readily available at their location. I suspect there were a handful of DSL customers left among thousands of cable modems, and the carrier was just done stocking parts for gear that hadn’t been built in twenty years.

Another dumbfuck, at a different location, got in a snit about Xfinity and ordered “AT&T fiber” as a replacement. When I arrived the technicians was there, rooting around in the phone wiring box on the side of the build. “Where’s the fiber?” I asked, knowing the answer was “no closer than the wiring pedestal for this neighborhood. One of those rare places in dry ass California that periodically floods, of course. After a week of agonizing uploads akin to ISDN speeds, the snit was resolved in Xfinity’s favor, albeit with much expense.

DSL from the CO is 1990s tech and it’s dying. DSL marketed as something else in a “fiber to shitty copper we don’t maintain” loop is a consolation prize for neighborhoods that can’t actually get cable modems. Overbuilding is common and only dumbfucks (as above) take the dinosaur option when the good stuff is available.

What percentage of Windstream customers have two wire DSL? If the answer is in the vicinity of 7.5%, maybe this is a sneaky corporate cost control measure gone awry. The DSL economics here are legit, but why a company would volunteer itself to replace 600k devices without delivery of service moving to a new medium is a head scratcher.

So … like the ownership, who has what type of service where is a question that makes me want to look deeper.

---

Do you know the name Robert Tappan Morris? I do and I recall precisely the instant he became famous, because I had classwork involving a DEC VAX due and I’d manage to snag a login on VAXE, the one snappy 11/782 among a herd of 11/780s.

The throttle on his code was orders of magnitude too fast for what he intended. If he’d got the timing right he’d be known as the creator of the first botnet, rather than the first worm.

I wonder if some enterprising lad didn’t sprearphish a front line tech support person at Windstream, pivot to get an admin, and work up to sneaking an update into the customer device management system that was meant to corral 600k devices into a botnet. Doing this using the management system would be an amazing coup, enabling persistence, but if they goofed on the payload they wanted to use on the devices …

The article does NOT support this notion. The outage wasn’t a whoops, it was intentional.

A report published Thursday by security firm Lumen Technologies’ Black Lotus Labs may shed new light on the incident, which Windstream has yet to explain. Black Lotus Labs researchers said that over a 72-hour period beginning on October 25, malware took out more than 600,000 routers connected to a single autonomous system number, or ASN, belonging to an unnamed ISP.

While the researchers aren’t identifying the ISP, the particulars they report match almost perfectly with those detailed in the October messages from Windstream subscribers. Specifically, the date the mass bricking started, the router models affected, the description of the ISP, and the displaying of a static red light by the out-of-commission ActionTec routers. Windstream representatives declined to answer questions sent by email.

According to Black Lotus Labs, the routers—conservatively estimated at a minimum of 600,000—were taken out by an unknown threat actor with equally unknown motivations. The actor took deliberate steps to cover their tracks by using commodity malware known as Chalubo, rather than a custom-developed toolkit. A feature built into Chalubo allowed the actor to execute custom Lua scripts on the infected devices. The researchers believe the malware downloaded and ran code that permanently overwrote the router firmware.

“We assess with high confidence that the malicious firmware update was a deliberate act intended to cause an outage, and though we expected to see a number of router make and models affected across the internet, this event was confined to the single ASN,” Thursday’s report stated before going on to note the troubling implications of a single piece of malware suddenly severing the connections of 600,000 routers.

Conclusion:

So what the actual eff is this? As I said in Incredible Intel

Once you’ve got a pile of bones, you have to sit down and decide if you’ve got a dinosaur or a Dodo bird.

Bullet points of theories to consider based on above:

• Relatively new foreign ownership triggers high level hit.

• Corporate move to eliminate DSL goes awry.

• Skiddies seeking control make a huge mess.

And the one I didn’t add above, specifically because of this

which Windstream has yet to explain

Windstream isn’t talking because this is a business impacting event for a broadly held private firm. Windstream isn’t talking because there is probably a 600k times $350 OEM device cost insurance claim. Windstream isn’t talking because those 600k devices are coming back, getting new software, and being placed in a spares pool.

And most of all, Windstream isn’t talking because it’s not clear if this own goal was a compromised administration system, an extraordinarily cheeky decision to install a stock malware kit on all of their client devices, or the least conspiratorial explanation:

A disgruntled employee responsible for the device management was let go, either just before (on the job sabotage) or just after (post employment dead man switch) the event.

Coda:

My last ISP engineering job began when the management learned the former engineer had wiped configs on every radio in an effort to ensure some ongoing work after he left. This was two months before Chet Uber asked me to help start Project VIGILANT, and as the ISP was partly owned and colocated with an electrical coop. This was the proximal cause for my fast track through Infragard vetting.

The company had one other incident along these lines, when a well liked manbaby field service tech cratered the management console for a fiber to the home system whose manufacturer had been swept from the playing field by M&A.

“Can’t you just reinstall it?”

“I have never been able to locate the install media.”

“Will support be able to help?”

“Unlikely.”

Support at this point was a former customer service rep for the company who had made off with the software for unlocking features in various pieces of hardware. They had a quiet business buying line cards from companies removing the system, which they sold to those who were crutching it along. They had no wisdom to offer on the management console.

A cybercrime outfit spotted this and wrote it up from outside. Having been on the inside … we’ll wait and see if there are charges brought.