The Pitter-Patter Of Cybersleet
It's ALWAYS the season for this.
Disinfodrome, a document indexing service I built and operated, was Open Semantic Search systems running on a Proxmox cloud I built for myself. I’ve been responsible for various hosting operations since the late 1990s. I don’t talk about it all that much; it’s a bit far afield from NIB’s usual pursuits.
But having worked 72 of the last 96 hours cleaning up someone else’s mess, I’m exhausted, irritated, and feeling a lil’ bit stabby.
Some of you may be in a position where you’re going to start hosting things on your own, perhaps there will be some wisdom, mixed in with my multifaceted rant.
Attention Conservation Notice:
Did you see the keyword “rant” in the lede? Well, did you? You should know what that means by now. This one is for the technically inclined reader.
Tarpit Time
If you have a virtual private server (VPS) somewhere with a public IP, and it’s brand new with literally nothing running on it except ssh, try this.
apt install tshark
curl ifconfig.me
tshark -i eth0 not host <your IP from previous command>
There’s an endless horde of compromised systems out there and they get put to work scanning for other vulnerable hosts. Below is the capture from a VPS I turned up shortly before I started writing this. What you see below are a couple brute force attempts on the ssh daemon and then some Tor startup stuff. If I left this running and added the qualifier “port 22” to the tshark command, there’d be hundreds of results in the first day of operation.
No matter where your system is, you’ll immediately start encountering this sort of cybersleet. Like the real thing, it’s noisy and you will notice. Unlike hail, you can go for a walk in it. If you drive too fast it can be a hazard, but mostly it’s just a nuisance. Same holds for the cyber form.
The system I’ve been working on has customers in California and some of them travel internationally. We could not just shut out the non-ARIN IP address ranges. If that sounds like Greek to you, read The Shape Of Cyberspace for some background. The internet has five global organizations that manage IP address space.
For the sake of international travelers we’re going to allow these systems to talk to a VPN provider that uses some well known address ranges. The rest of AfriNIC, APNIC, LACNIC, and RIPE get the finger.
Here’s what cybersleet looks like in practice.
We’re using the iptables Linux firewall to set rules for incoming traffic. If the traffic originates from 1.0.0.0/8, a group of sixteen million addresses that used to be called a “class A” block, it will end up in the TARPIT. Since the arrival of BGP back in 1994 those former class A/B/C blocks have been called prefixes. If you’ve got a sharp eye you might have noticed the /7 on the forth line - prefixes are variable length, while blocks were fixed size.
iptables -A INPUT -p tcp -s 1.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 2.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 5.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 6.0.0.0/7 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 11.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 14.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 21.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 22.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 25.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 26.0.0.0/7 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 28.0.0.0/6 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 33.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 36.0.0.0/7 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 39.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 41.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 42.0.0.0/7 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 46.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 49.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 51.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 55.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 57.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 58.0.0.0/7 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 60.0.0.0/7 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 62.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 77.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 78.0.0.0/7 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 80.0.0.0/5 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 88.0.0.0/6 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 92.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 94.0.0.0/7 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 101.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 102.0.0.0/7 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 105.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 106.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 109.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 110.0.0.0/7 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 112.0.0.0/5 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 120.0.0.0/6 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 124.0.0.0/7 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 126.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 130.0.0.0/7 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 133.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 141.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 145.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 150.0.0.0/7 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 153.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 154.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 163.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 171.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 175.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 176.0.0.0/5 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 185.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 186.0.0.0/7 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 188.0.0.0/6 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 193.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 194.0.0.0/7 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 196.0.0.0/7 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 200.0.0.0/6 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 210.0.0.0/7 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 212.0.0.0/6 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 217.0.0.0/8 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 218.0.0.0/7 -m multiport --dports 21,22 -j TARPIT
iptables -A INPUT -p tcp -s 220.0.0.0/6 -m multiport --dports 21,22 -j TARPIT
6.0.0.0/7 is an aggregation of two of the maximum size prefixes, 6.0.0.0/8 and 7.0.0.0/8. This makes it a wee bit harder to read but that’s fine, because we packed 124 slash eights into a total of 63 prefixes from /8 clear up to a /5. These rules get applied sequentially to incoming traffic, so this is great for efficiency. A /5 contains eight /8s, or 128 million IPs.
The default Linux firewall offers things like REJECT, which forcefully ends an attempted connection, or DROP, which refuses to negotiate. The TARPIT extension tells the far end “yes, but I’m a little busy, let me put you on hold …” Eventually the would-be intruder’s system times out, but it can take a LONG time. We’re talking hundreds of seconds, instead of a hundred milliseconds.
The internet’s original sticky trap was named La Brea, in honor of the vast tarpits in Los Angeles. When I operated ISPs I would take some old machines and set them up to do this in IP address space we had, but which had not been assigned to a customer yet. Today’s computers are so much faster we can use the TARPIT extension on systems with other duties.
There is another older firewall technology called TCP wrappers. There’s a configuration file called /etc/hosts.allow, and this is what controls access to the ssh service. This example permits the machine to talk to itself on the 127.0.0.1 loopback IP and it shows how to add the TailScale encrypted overlay network, which uses the reserved Carrier Grade NAT prefix 100.64.0.0/10, and last there are trusted ssh only IPs. The stuff below that is historical; the world was a kinder, gentler place when things like finger were left available.
all : 127.0.0.1 : allow
#CGNAT
all : 100.64.0.0/10 : allow
sshd: 23.57.182.145 : allow
sshd: 52.217.100.131 : allow
# You need to be clever with finger; do _not_ backfinger!! You can easily
# start a "finger war".
fingerd : ALL \
: spawn (echo Finger. | \
/usr/bin/mail -s "tcpd\: %u@%h[%a] fingered me!" root) & \
: deny
# The rest of the daemons are protected.
ALL : ALL \
: severity auth.info \
: twist /bin/echo "You are not welcome to use %d from %h."
I like hosts.allow for a couple reasons - 1) it’s always there, no matter what might be happening with firewall rules, 2) it’s very easy to sight read and verify, and 3) it’s a snap to move it to another system. If you are copying a portion of firewall rules from one machine to another, that’s a fiddly manual merge job.
Conclusion:
If you’ve never used a protocol analyzer like tshark, or its graphical big brother, Wireshark, this is probably all new and mysterious. Given the type of work I do, there are lots of days where I never use one, but it would be rare to go a whole week without at least one issue that required this for debugging.
Limiting one’s attack surface is a fundamental security strategy but I see so incredibly many bad things out there. People will just leave ssh on port 22 open to the world. If their password matches an email password in a leak, they get had. There are daemons like MySQL that should NEVER be exposed, but I find that disturbingly often. If I’m hosting something that’s behind Cloudflare I will create static routes to their CDN, and then set up a simple VPS that provides fail closed firewall services. Any mistakes I might make on setup are not an exposure; the machine is simply unreachable.
I think not many of you do this sort of work, but you may know someone who does, and they might be struggling. I am always happy to pick up new clients, even if it’s just a couple hours work …
