The minivan dilemma: It is the least cool vehicle ever designed, yet the most useful. Offering the best value for the most function to a plurality of American drivers, a minivan can cart seven passengers or more in comfort if not style, haul more cargo than many larger trucks, and do so for a sticker price roughly a quarter cheaper than competing options. Even so, minivan sales have been falling steadily since their peak in 2000, when about 1.3 million were sold in the United States. As of last year, that figure is down by about 80 percent. Once sold in models from more than a dozen manufacturers, the minivan market now amounts to four, one each from Chrysler, Honda, Toyota, and Kia.
On account of the dilemma, a minivan is typically purchased under duress. If you live in a driving city, and especially if you have a family, a minivan conversation will eventually take place. Your older, cooler car—perhaps your Mini Cooper or your spouse’s Honda CR-V—will prove unfit for present purposes. Costco cargo, loads of mulch, sports equipment, and holiday loot all need a place to go. The same is true of car seats, which now are recommended for children as old as 7. And so, before too long: “Maybe we should get a minivan.”
This phrase is uttered with an air of resignation. The minivan was popular, but it was never cool, not even in its youth, during the 1980s. Now it’s middle-aged: The first of its type came out in ’83, which makes the minivan an elder Millennial, and it’s no more attuned than your average 41-year-old to recent trends. But why, exactly, has it earned so much derision through the years? And why was the minivan replaced, almost altogether, by the SUV?
The minivan arrived, way back when, as a savior. When Chrysler, under the former Ford chief Lee Iacocca’s direction, first conceived of the design in the late 1970s, Americans who wanted room to cart more kids and goods had only a couple of options. One was the land-yacht-style station wagon, perhaps in avocado green with faux-wood paneling. Lots of kids could pile onto its bench and jump seats, while the rear storage, accessible by hatch, allowed for easy loading. These cars were somewhat functional, but they didn’t seem that safe. The suburban family’s other choice was the full-size van—a big, boxy transport or utility vehicle. The gas for these was also pricey, and their aesthetic felt unsuited to domesticity. By cultural consensus, vans were made for plumbers, kidnappers, or ex–Special Forces domestic mercenaries.
Chrysler’s minivan would steer clear of those two dead ends, and carry American families onto the open roads toward, well, youth soccer and mall commerce. It really did bring innovation: ample seating organized in rows with easy access, the ability to stow those seats in favor of a large cargo bay, a set of sliding doors, and smaller features that had not been seen before, such as the modern cupholder. And it offered all that at an affordable price with decent fuel economy.
Pickup was quick. In the first year after introducing them, Chrysler sold 210,000 Dodge Caravans and Plymouth Voyagers, its initial two models. Overall minivan sales reached 700,000 by the end of the decade, as the station wagon all but disappeared. But the new design also generated stigma: As the child of the station wagon and the service van, the minivan quickly came to represent the family you love but must support, and also transport. In a nation where cars stood in for power and freedom, the minivan would mean the opposite. As a vehicle, it symbolized the burdens of domestic life.
That stigma only grew with time. In 1996, Automobile magazine called this backlash “somewhat understandable,” given that the members of my generation, who were at that point young adults, had “spent their childhoods strapped into the backseat of one.” Perhaps it was childhood itself that seemed uncool, rather than the car that facilitated it. In any case, minivans would soon be obsolesced by sport utility vehicles. The earliest SUVs were more imposing than they are today: hard-riding trucks with 4×4 capabilities, such as the Chevrolet Suburban and the Jeep Wagoneer. These were as big as or even bigger than the plumber-kidnapper vans of the 1970s, and they got terrible gas mileage, cost a lot of money, and were hard to get in or out of, especially if you were very young or even slightly old. Yet the minivan’s identity had grown toxic, and for suburban parents, the SUV played into the fantasy of being somewhere else, or doing something better.
The SUV’s promise was escape from the very sort of family life that the minivan had facilitated. In 2003, TheNew York Times’ John Tierney recounted how the new class of vehicles had taken over. “The minivan became so indelibly associated with suburbia that even soccer moms shunned it,” he explained. “Soon image-conscious parents were going to soccer games in vehicles designed to ford Yukon streams and invade Middle Eastern countries.” At the same time, the SUVs themselves were changing. The minivan had been built from parts and designs for a car, not a van. SUV manufacturers followed suit, until their vehicles were no longer burly trucks so much as carlike vehicles that rode higher off the ground and had a station-wagon-style cargo bay. Few even had more seats than a sedan. As the early minivans were to vans, so were these downsized SUVs to the 4x4s that came before them.
Functionally, the minivan is still the better option. It is cheaper to buy and operate, with greater cargo space and more seating and headroom. Still, these benefits are overshadowed by the minivan’s dreary semiotics. Manufacturers have tried to solve that problem. When my family reached the “Maybe we should get a minivan” milestone, I noticed that some models of the Chrysler Pacifica now offered, for a premium, blacked-out chrome grills and rims. But to buy a poseur “sport van,” or whatever I was meant to call this try-hard, cooler version of the uncool minivan, struck me as an even sadder choice.
Beyond such minor mods, the industry hasn’t really done that much to shake away the shame from the minivan’s design. I suspect that any fix would have to be applied at the level of its DNA. The minivan was the offspring of the wagon and the van. To be reborn, another pairing must occur—but with what? Little differentiation is left in the passenger-vehicle market. Nearly all cars have adopted the SUV format, a shoe-shaped body with four swinging doors and a hatch, and true 4x4s have been all but abandoned. Perhaps the minivan could be recrossed with the boxy utility van, which seems ready for its own revival. This year, Volkswagen will begin selling a new electric version of its microbus, one of the few direct precursors to the minivan that managed to retain an association with the counterculture despite taking on domestic functions.
However it evolves, the minivan will still be trammeled by its fundamental purpose. It is useful because it offers benefits for families, and it is uncool because family life is thought to be imprisoning. That logic cannot be overcome by mere design. In the end, the minivan dilemma has more to do with how Americans think than what we drive. Families, or at least vehicles expressly designed for them, turned out to be lamentable. We’d prefer to daydream about fording Yukon streams instead.
I sat down with my favorite webcomic creator, Sarah Andersen, and Emmy-winning creator Bobby Chiu to check out the first "An Infinite Story" experience. It's a hidden art game that uses vector art to let you zoom in on art indefinitely without any pixellation.
The Importance of Stupidity in Scientific Research – published in 2008 in the Journal of Cell Science – is one of those wonderful essays that turns your understanding of “being smart” on its head. There are enough implications for education here that it’s worth reading the entire thing.
[W]e don’t do a good enough job of teaching our students how to be productively stupid – that is, if we don’t feel stupid it means we’re not really trying. —Martin Schwartz
[Hat tip to @KevinDKohl for the share on twitter. The original article can be found here.]
The idea of confronting one’s own stupidity and living with it productively – even joyfully! – feels even more central to me in mathematics than in science. The article touches on learning science, but it’s really about research. Stupidity is part of confronting the new, and being a capable science student means you might not have to face it that much when you’re merely absorbing the knowledge that previous scientists have already uncovered.
Math seems different to me. Everyone who learns math is familiar with the experience of being stuck on some new idea or problem, banging their head against it, and then, when they finally understand the answer (or having someone tell them), feeling stupid. There’s something fundamental in the nature of mathematics that makes it easy once you get it, and impossible before.
To wit: there’s an old joke where a professor is lecturing and remarks, “It’s obvious why this equation holds, of course.” A brave student raises her hand and says what everyone is thinking: “Could you please explain it? It doesn’t seem obvious.” The professor looks at the equation. “Isn’t it obvious?” He starts mumbling to himself. He starts to write on the white board, then stops. Finally he just stands there, looking at the board. Ten minutes pass. The students shift awkwardly and the seconds tick by. Suddenly the professor exclaims, “Aha! It is obvious!”
I’ve had numerous experiences in mathematics that left me feeling stupid. The one that made it clearest, I think, was taking real analysis in college. There’s a technical business with epsilons and deltas: one arbitrarily small value has to be smaller than another, thus proving that a function is continuous in a more rigorous way than high school calculus ever bothers to consider. It was arcane and baffling when I took the course. Six months later, it felt so simple that I couldn’t understand what my problem had been. Why had I been so stupid before, that this obvious mathematical idea had eluded me?
My personal theory about this (and this is untested, as far as I know) is that mathematics is somewhat unique in the way new learning develops in the mind. Piaget studied mathematical leaps in understanding in young children, and noticed that there are almost switches that flip at certain ages: before they’ve flipped, an idea is essentially impossible to understand; after, they’re obvious.
A classic example from Piaget is conservation of volume. (It’s worth checking out the video below if you’re not familiar with this one.) A child confirms two quantities of liquid are the same, sees one poured into a taller, thinner glass, and then says that the taller one has more water in it, despite seeing it proved that the two have the same amount of liquid.
Is the boy stupid? No. None of us were, and we all went through this phase. At some age, none of us could see that the amount of liquid stays constant. But once you see it, it’s impossible not to see it. And it’s almost impossible to imagine how it wouldn’t be obvious to anyone.
I suspect that mathematical understanding requires similar mental leaps, no matter how long you study it. It’s almost as if we construct new highways in our mind. As long as the construction goes on, no cars drive on it, and our mind hits the same confusion again and again. Then one day, the new road opens, and we “get it.” Once the cars can drive from one side of our mind to the other, the new idea feels obvious. And more, you are so changed by the new perspective that you can’t even understand your own prior lack of understanding. I don’t know any other field of study that so regularly gives you this experience. And if you track topics in mathematics, you can see this happening again and again: with base 10, fractions, algebra, calculus… it’s all hard, until it suddenly feels easy.
These jumps in comprehension can be thrilling, and they’re one reason math is so fun. But they do create a challenge for the student. The evidence that you learned something hard is that you feel like you’re stupid. That stupidity is essential to the process, not of pushing at the boundaries of what is known, but simply of getting your mind to take in tools and ideas more abstract than it was ever meant to learn. Students need to know that this feeling is the norm when it comes to learning math.
The centrality of stupidity can be especially challenging for the teacher. Being in the position of understanding means that we are distanced from someone who doesn’t understand. It takes an act of remembering that borders on wholesale invention to get a sense of how someone couldn’t see what seems so obvious once you see it. What’s happening in their mind? If you couldn’t understand your own inability from a few weeks ago, how do you understand theirs?
I’ve come to believe that one of the best ways to address the centrality of stupidity is to take on two opposing efforts at once: you need to assure students that they are not stupid, while at the same time communicating that feeling like they are stupid is totally natural. The message isn’t that they shouldn’t be feeling stupid – that denies their honest feeling to learning the subject. The message is that of course they’re feeling stupid… that’s how everyone has to feel in order to learn math!
It’s all just another reason math is deviously difficult, and spectacularly wonderful to learn. Is there any other study that changes you so deeply, and so often?
Welcome back to another watchTowr Labs blog. Brace yourselves, this is one of our most astounding discoveries.
Summary
What started out as a bit of fun between colleagues while avoiding the Vegas heat and $20 bottles of water in our Black Hat hotel rooms - has now seemingly become a major incident.
We recently performed research that started off "well-intentioned" (or as well-intentioned as we ever are) - to make vulnerabilities in WHOIS clients and how they parse responses from WHOIS servers exploitable in the real world (i.e. without needing to MITM etc).
As part of our research, we discovered that a few years ago the WHOIS server for the .MOBI TLD migrated from whois.dotmobiregistry.net to whois.nic.mobi – and the dotmobiregistry.net domain had been left to expire seemingly in December 2023.
Putting thoughts aside, and actions first, we punched credit card details as quickly as possible into our domain registrar to acquire dotmobiregistry.net - representing much better value than the similarly priced bottle of water that sat next to us.
Our view was that as a legacy WHOIS server domain, it was likely only used by old WHOIS tools (such as phpWHOIS, which conveniently has an Remote Code Execution (RCE) CVE from 2015 for the parsing of WHOIS server responses – thus fitting our aim quite nicely).
Throwing caution into the wind and following what we internally affectionately refer to as our 'ill-advised sense of adventure' - on Friday 30th August 2024 we deployed a WHOIS server behind the whois.dotmobiregistry.net hostname, just to see if anything would actually speak to it actively.
The results have been fairly stunning since - we have identified 135000+ unique systems speaking to us, and as of 4th September 2024 we had 2.5 million queries. A brief analysis of the results showed queries from (but certainly not limited to):
Various mail servers for .GOV and .MIL entities using this WHOIS server to presumably query for domains they are receiving email from,
Various cyber security tools and companies still using this WHOIS server as authoritative (VirusTotal, URLSCAN, Group-IB as examples)
However, significant concern appeared on 1st September 2024 when we realised that numerous Certificate Authorities responsible for issuing TLS/SSL certificates for domains like 'google.mobi' and 'microsoft.mobi', via the 'Domain Email Validation' mechanism for verifying ownership of a domain, were using our WHOIS server to determine the owners of a domain and where verification details should be sent.
We PoC'd this with GlobalSign and were able to demonstrate that for 'microsoft.mobi', GlobalSign would parse responses provided by our WHOIS server and present 'whois@watchtowr.com' as an authoritative email address.
Effectively, we had inadvertently undermined the CA process for the entire .mobi TLD.
As is common knowledge, this is an incredibly important process that underscores the security and integrity of communications that a significant amount of the Internet relies upon. This process has been targeted numerous times before by well-resourced nation-states:
While this has been interesting to document and research, we are a little exasperated. Something-something-hopefully-an-LLM-will-solve-all-of-these-problems-something-something.
As always, we remind everyone - if we could do this, anyone can.
Onto the full story...
Setting The Scene
We're sure you’re familiar with the old adage, ‘it never rains but it pours’. That was definitely the case here, where we set out with the intention of just getting some RCE’s to fling around, and ended up watching the foundation of secure Internet communication crumble before our eyes.
Before we get ahead of ourselves, though, let’s start at the beginning, in which we decided to take a quick look at a WHOIS client. The protocol being some 50+ years old, we expected WHOIS clients to be constructed with the same brand of string as an enterprise-grade SSL VPN appliance, and so we took a naive shot and served up some A’s.
This, at first glance, looks like an easily-exploitable crash. We were keen to find more bugs, and keenly started examining some other client implementations - but we were soon interrupted by some vocal killjoys naysayers.
They were quick to remind us that, to get to this state in our lab environment, we’d impersonated a WHOIS server, redirecting traffic from the usual server to our test server via iptables.
How realistic was this attack scenario, the naysayers asked?
We tried to silence the killjoy's naysayers and convince them our attack was plausible - we could find a registrar that allows us to set a Referral WHOIS value, or buy an IP range and control the range ourselves - but they suggested we spend more time doing, and less time playing academia.
The reality was that in order for an attacker to carry out an attack against a WHOIS client, they’d need one of the following:
A Man-In-The-Middle (MiTM) attack, which requires the ability to hijack WHOIS traffic at the network layer - out of reach for all but the most advanced of APTs,
Access to the WHOIS servers themselves, which is plausible but unlikely, or
A WHOIS referral to a server they control.
These are effectively the preconditions of a nation-state or someone who is very comfortable compromising global TLD WHOIS servers in pursuit of exploiting clients.
You would, at this point, be forgiven for thinking that this class of attack - controlling WHOIS server responses to exploit parsing implementations within WHOIS clients - isn’t a tangible threat in the real world.
We were left unsatisfied. We had located some shoddy code, but declaring it out of reach sounded like something you might bill a day rate for.
Perhaps there was another avenue for attack?
Collateral Damage In Pursuit Of RCE
The key to turning this theoretical RCE into a tangible reality is rooted in the tangled mess of the WHOIS system.
One of the biggest ‘kludges’ in the WHOIS system is the means of locating the authoritative WHOIS server for a given TLD in the first place.
Each TLD (the bit at the end of the domain), you see, has a separate WHOIS server, and there’s no real standard to locating them - the only ‘real’ method being examining a textual list published by IANA. This list denotes the hostname of a server for each TLD, which is where WHOIS queries should be directed.
As you can imagine, maintainers of WHOIS tooling are reluctant to scrape such a textual list at runtime, and so it has become the norm to simply hardcode server addresses, populating them at development time by referring to IANA’s list manually. Since the WHOIS server addresses change so infrequently, this is usually an acceptable solution.
However, it falls down in an ungraceful manner when server addresses change. With a little bit of legwork, we found that the WHOIS server for a particular TLD - .mobi - had been changed some years ago from the old domain whois.dotmobiregistry.net to a new server, at whois.nic.mobi.
Of course though, because the Internet is joined together by literal string and hopes/wishes at this stage, somebody had neglected to renew the old domain at dotmobiregistry.net meaning it was up for grabs by anyone with $20 and an ill-advised sense of exploration.
We registered the domain, working on the theory that, while most client tooling would be updated to use whois.nic.mobi, most of the Internet population is still surprised when their 2011 SAP deployment gets popped, and thus WHOIS applications in production had a fairly decent chance of still referencing whois.dotmobiregistry.net.
Of course, this being the Internet, we got a little more than we bargained for.
So What? It's Old
We soon realized the threat model for this attack had just changed.
Now that we control a WHOIS server, we were in the position to ‘respond’ to traffic sent by anyone who hadn’t updated their client to use the new address (auto updates are bad, turn them off).
No longer do we require a Man-In-The-Middle attack, or some exotic WHOIS referral, to exploit a WHOIS client vulnerability - all we need to do is wait for queries to come in, and theoretically respond with whatever we want.
The pre-requisites for real-world exploitation now sat within what we deemed ‘rough reality’.
Things were beginning to escalate.
We had set out to find some simple bugs in WHOIS client tooling, file for some CVEs, get them fixed.. but then we realised that once again we’d probably chewed off more than we intended and things were about to become worse - much worse.
Never Update, Auto-Updates And Change Are Bad
Unfortunately, there is a lot of Internet infrastructure which depends on the antiquated WHOIS protocol.
Starting off slow, we’re now in a position to attack the many websites that run a WHOIS client and echo the results back to the user, injecting XSS or PHP eval payloads. Ethical (and legal) concerns prevent us from doing so, however - and we did not spend $20 to get an XSS.
Of course, our original goal was to find and exploit some 0day in WHOIS clients, or some other system that embeds a WHOIS client (such as a spam filter), similar to the trivial memory corruption we found earlier.
Our biggest hurdle here - as alluded to above - was the simplicity of the WHOIS protocol itself, which is a simple text-based TCP data stream. With so little complexity, there seemed very little room for developers to make errors.
Ha.
Prior Art
To fully understand and look to leverage our new capability and adjusted threat model, we decided to examine the area’s ‘prior art’ in exploitation, looking at historic attacks on WHOIS clients.
We were somewhat surprised that a search for relevant CVE data yielded relatively few results, which we attributed to the area being under-researched - the search return 26 CVE records.
Once we discount the irrelevant results, we are left with only three bugs that are triggered by malformed WHOIS responses.
This small number - three bugs since 1999 - makes it obvious to us that very little research has been done - likely due to the perception that any real-world exploitation comes with difficult prerequisites, such as control of a TLD WHOIS server.
But, there have been some interesting cases - just to give you a taste of where this is going.
phpWHOIS (CVE-2015-5243)
The first bug that our retrospective found was CVE-2015-5243. This is a monster of a bug, in which the prolific phpWhois library simply executes data obtained from the WHOIS server via the PHP ‘eval’ function, allowing instant RCE from any malicious WHOIS server.
The important item is the juicy eval statement in the middle of the snippet, which is fed data returned from the WHOIS server.
While it attempts to escape this data before it evaluates it, it does so imperfectly, only replacing " with the escaped form, \\\\" . Because of this, we can sneak in our own PHP code, which is then executed for us.
Netitude’s blogpost lays out all the details, and even provides us with exploitation code - ”;phpinfo();// - is enough to spawn a phpinfo page.
We tried this out on an application that uses phpWhois, purely to demonstrate, and it worked swimmingly:
Clearly this is a powerful bug - the best part being that phpWhois hardcodes our newly found whois.dotmobiregistry.net in vulnerable versions (it's old, but at a cursory glance no-one appears to have ever updated phpWhois).
What other historic artefacts could we find, though?
Fail2Ban (CVE-2021-32749)
As we continued to examine historic client-side bugs, we came across CVE-2021-32749. This one is again a pretty nasty bug, this time in the ever-popular fail2ban package. It’s a command injection vulnerability, a vulnerability class keenly sought by attackers due to its power and ease of exploitation.
As you may know, if you have administered a fail2ban server, the purpose of fail2ban is to monitor failed login attempts, and prevent bruteforce or password-guessing attacks by blocking hosts which repeatedly fail to log in.
Being the polished package it is, it also includes the ability to email an administrator when an IP address is banned, and - very helpfully - when it does so, it will enrich the email with information about who owns the banned IP address.
This information is gleaned from - yeah, you guessed it! - our friend WHOIS.
Unfortunately, for some time, the output of the WHOIS client wasn’t correctly sanitized before being passed to the mail tool, and so a command injection bug was possible.
Fortunately - or unfortunately, if you’re an attacker - because fail2ban runs a WHOIS query on the IP address rather than, for example, a domain name specified in the PTR record of an IP address of blocked hosts - this attack is not within reach still based on our newly found capability.
For those that control a WHOIS server that is queried for IP addresses, though, exploitation is simple - simply attempt to unsuccessfully authenticate to a server via SSH a few times to trigger a ban, and once fail2ban queries the WHOIS server for information on your IP address - serve a payload wrapped in backticks.
Reality check
So, the burning question on our minds - can we actually exploit these bugs, right now?
Well, at this stage, our view was fairly pessimistic in terms of achieving real-world impact. We saw the following pre-requisites:
The WHOIS client must be querying an old authoritative .MOBI WHOIS server and thus by definition, has not been working for quite a while
To achieve client-side code execution (i.e. compromise) via a WHOIS client vuln - the only public option available to us was disclosed in 2015 and appears to have been rectified in 2018 - likely due to the perceived lack of real-world exploitation mechanisms.
Meh. Our gut feeling remained that most of the Internet and those in the sane world would logically be querying the new .mobi authoritative WHOIS server whois.nic.mobi, rather than the decommissioned dotmobiregistry.net (which we now controlled).
“Surely no large organisations would still reference the old domain”, we thought to ourselves.
Kill WHOIS With Fire
Without skipping a beat and really not considering the consequences, we set up a WHOIS server beneath our new domain at whois.dotmobiregistry.net, and logged incoming requests. We specifically focused on two things:
Source IPs (so we can perhaps begin to work out who exactly was querying an outdated server), and,
The queried domain (because again, this may give off some clues).
We threw together the lglass server to respond to WHOIS requests that found their way to our WHOIS server, and returned:
ASCII art (we were relatively refrained here, but it was a priority)
Fake WHOIS details indicating watchTowr as the owner for every queried entity.
As this was our private server, we included a request for queries to cease (after all, they were unauthorised).
A quick test directly to our new WHOIS server showed that all was working as expected, with the following response provided for a query about google.mobi:
Nice.
Uh…..
Well, it’s 2024 - absolutely no one has the ability to exercise patience, including ourselves.
So, we began just looking around the Internet for obvious locations that could be sending queries our way. Surely, we thought - surely! - the broken clients using an outdated server address wouldn’t be in anything major, that we use every day?
A significant number of domain registrars and WHOIS-function websites
etc (you get the idea)
A screenshot of each WHOIS tool would become repetitive, but you get the idea.
urlscan.io - “A sandbox for the web” - used our WHOIS server for .mobi, too. You can see the results by browsing to a page representing any .mobi domain (like this one).
VirusTotal, the popular malware-analysis site, was querying us! A tool dedicated to the analysis of hostile code seemed like an opportunity for enjoyment.
Sadly, VirusTotal doesn't render our ASCII art properly, but as you can see - VirusTotal is querying our makeshift WHOIS server for this global .TLD and presenting back the results. We were also pleased to see that VirusTotal updated their records of who owns bbc.mobi:
For anyone that has ever worked in offensive security, you occasionally get a sinking feeling where you realize something may be a little larger than expected, and you begin to wonder.. “what have we broken?”.
(Editors note: Technically, this should be ‘what was broken’, because people were querying our WHOIS server without authorisation and we’re very upset - get off our lawn!).
Well, with our WHOIS server clearly working - we figured we’d come back in a few days and see if anything at all reached out to us - giving us us a good excuse to stare at a separate PSIRT response indicating a 2 year lead time to resolve a vulnerability.
Being insatiable and generally finding it hard to focus on anything longer than a TikTok video of a dog in a hat, we took a look to see how many unique IPs had queried our new WHOIS server after a few hours:
$ sqlite3 whois-log-copy.db "select source from queries"|sort|uniq|wc -l
76085
Uh. Yes, that’s correct - this is 76,000+ unique source IP addresses that have sent queries to our WHOIS server in just a couple of hours.
We were somewhat dismayed when, after leaving our server running for around two days, the poor little SQLite DB containing the logs ballooned to some 1.3 million queries! Clearly, we’d stumbled into something more major than we’d anticipated.
We threw the list of IPs at ZDNS and just sat back, as a relatively feeble way of doing attribution:
Some other highlights of source hosts (not exhaustive, but just to give you some idea of just how bad this trash fire appeared to be):
Mail servers! Lots and lots of mail servers.
Spam filters will often do WHOIS lookups on sender domains. We saw a bunch of these, ranging from the aptly-named cheapsender.email through to mail.bdcustoms.gov.bd - which appears to be part of the Bangladeshi government's infrastructure. Yikes! Theoretically, we could cause mayhem by serving responses indicating that the sending domain was a known spammer - and even more mayhem-worthy to start fuzzing the WHOIS parsing code to pop RCE on the mail servers themselves.
(We didn’t)
Leading on from that thought, what other .gov apparatus have we been queried by? Well, we found Brazil in our logs multiple times - for example, antispam.ap.gov.br and master.aneel.gov.br , and Brazil was not alone. We also found .gov addresses belonging to (but again not limited to):
Argentina,
Pakistan,
India,
Bangladesh,
Indonesia,
Bhutan,
Philippines,
Israel,
Ethiopia,
Ukraine,
USA.
Neat.
Militaries (.mil)
Swedish Armed Forces, for example
Universities (.edu)
All of them
We even saw cyber security companies - hey Group-IB, Detectify! - query our WHOIS server (presumably doing threat intel things for .mobi domains).
We saw Censys query us for ‘google.com’ and wondered if we’d get an APT number and a threat intel report shout-out if we’d been actively delivering payloads. Maybe we did? Check your boxen. (We didn't. Or did we?)
We’re still trying to determine what software solutions are in play here/configured to query this WHOIS server for .mobi - let us know if you have any ideas.
Those who are nefariously minded likely realised what we saw as well - with .gov and other mail servers querying us each time they received an email from a .mobi domain - we could begin to passively determine who may be in communication.
This is not ideal. How do we fix this? Well, hold that thought - IT GETS WORSE.
Tales of TLS
TLS/SSL. Everyone knows it - it’s that friendly little padlock icon in the address bar that assures you that your connection is secure. It’s powered by the concept of certificates - sometimes used for HTTPS, sometimes used for signing your malware.
For example, say you’re the owner of watchTowr.mobi. You want to secure communications to your web server by speaking TLS/SSL , so you go off to your favourite Certificate Authority and request a certificate (let’s also pretend you haven’t heard of LetsEncrypt).
The Certificate Authority will verify that you own the domain in question - watchTowr.mobi - and will then sign a private certificate, attesting to your identity as the owner of that domain. This is then used by the browser to ensure your communications are secure.
Anyway, what does this have to do with WHOIS, and what does it have to do with us?!
Well, it turns out that a number of TLS/SSL authorities will verify ownership of a domain by parsing WHOIS data for your domain - say watchTowr.mobi- and pulling out email addresses defined as the ‘administrative contact’.
The process is to then send that email address a verification link - once clicked, the Certificate Authority is convinced that you control the domain that you are requesting a TLS/SSL cert for and they will happily mint you a certificate.
For example:
Perhaps you can see where we’re going with this? sobs
If a TLS/SSL certificate authority is using our WHOIS server for .mobi domains, we can likely provide our own email address for this “Email Domain Control Validation” method.
Uh-oh. Is this a fringe feature supported only by two-bit, poor-quality certificate authorities?
No! Here’s a sample of large TLS/SSL Certificate Authorities/resellers that support WHOIS-based ownership verification:
Trustico
Comodo
SSLS
GoGetSSL
GlobalSign
DigiSign
Sectigo
Going through the normal order flow, we began cautiously - by generating a CSR (Certificate Signing Request) for the fictitious domain watchTowr.mobi - the logic being that as long as our WHOIS server was queried, whether or not the domain was real was irrelevant because we respond positively to absolutely every request including domains that don’t actually exist.
# sudo openssl req -new -key custom.key -out csr.pem
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:SG
State or Province Name (full name) [Some-State]:Singapore
Locality Name (eg, city) []:Singapore
Organization Name (eg, company) [Internet Widgits Pty Ltd]:watchTowr
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []:watchtowr.mobi
Email Address []:
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
We’re not going to walk through each provider - for the purposes of illustration, we’ll use GoGetSSL.
Once we upload our watchTowr.mobi CSR to GoGetSSL, it is parsed, and we continue. The indication of these placeholder email addresses indicates that WHOIS was not successful - instead of the email address that our WHOIS server is configured to respond with (whois@watchtowr.com), we’re presented with only @watchtowr.mobi domains.
That’s something of a relief.
The Certificate Authority has correctly determined that the domain watchTowr.mobi does not exist and thus if WHOIS is working as expected, no email addresses will be returned. We concluded that our newly set up WHOIS server was not being queried by the provider.
At least the world isn’t ending. Right? (spoiler: it actually was)
We carried on trying a few other providers until a thought occurred.
The WHOIS protocol is extremely simple. Essentially it is a string blob returned in various formats depending on the TLD serving it. Each provider implements parsing in their own way. Perhaps, before we write off our theory, we should make sure this verification mechanism is actually working as it is supposed to.
So, we began again - choosing microsoft.mobi as a .mobi domain that appeared to follow a fairly typical WHOIS format (when using the current .mobi WHOIS server).
The screenshot below shows that the legitimate WHOIS record for microsoft.mobi was correctly parsed at Entrust, as the only email addresses available for validation were at the microsoft.com domain:
While the WHOIS record for watchTowr.mobi was not being parsed at all (indicating that Entrust was using the correct WHOIS server, and not ours):
Looks good you think?
WRONG.
We skipped and hopped over to the next provider, GlobalSign. GlobalSign reported that they were unable to parse the WHOIS record of microsoft.mobi:
At this point, something clicked in our minds. Perhaps GlobalSign WAS querying our new WHOIS server - but the string returned by our WHOIS server was incompatible with GlobalSign’s parsing?
We copied the microsoft.mobi output from the legitimate WHOIS server, made it our own, and loaded it into our own WHOIS server - updated to look like the following:
Holding our breath, we then re-triggered GlobalSign with a CSR for microsoft.mobi…
We want to be explicitly clear that we stopped at this point and did not issue any rogue TLS/SSL certificates to ourselves. This would undoubtedly create an incident, and require significant amounts of work by many parties to revoke and roll back this action.
Success!
The GlobalSign TLS/SSL certificate WHOIS domain verification system had queried our WHOIS server, parsed whois@watchTowr.com from the result, and presented it as a valid email address to send a verification email to, allowing us to complete verification and obtain a valid TLS/SSL certificate.
This is then blindingly simple:
Set up a rogue WHOIS server on our previously authoritative hostname, responding with our own email address as an ‘administrative contact’
Attempt to purchase a TLS/SSL certificate for a .mobi domain we want to target (say, microsoft.mobi)
A Certificate Authority will then perform a WHOIS lookup, and email us instead of the real domain owners [theory]
We click the link, and.. [theory]
… receive an TLS/SSL cert for the target domain! [theory]
Now that we have the ability to issue a TLS/SSL cert for a .mobi domain, we can, in theory, do all sorts of horrible things - ranging from intercepting traffic to impersonating the target server. It’s game over for all sorts of threat models at this point.
While we are sure some may say we didn’t ‘prove’ we could obtain the certificate, we feel this would’ve been a step too far — so whatever.
One Last Thing
Please stop emailing us..
Here We Go Again..
We hope you’ve enjoyed (and/or been terrified by) today’s post, in which we took control of a chunk of the Internet’s infrastructure, opened up a big slab of juicy attack surface, and found a neat way of undermining TLS/SSL - the fundamental protocol that allows for secure communication on the web.
We want to thank the UK's NCSC and the ShadowServer Foundation for rapidly working with us ahead of the release of this research to ensure that the 'dotmobiregistry.net' domain is suitably handled going forwards, and that a process is put in place to notify affected parties.
The dotmobiregistry.net domain, and whois.dotmobiregisry.net hostname, has been pointed to sinkhole systems provided by ShadowServer that now proxy the legitimate WHOIS response for .mobi domains.
We released this blog post to initially share our process around making the unexploitable exploitable and highlight the state of legacy infrastructure and increasing problems associated with abandoned domains - but inadvertently, we have shone a spotlight on the continuing trivial loopholes in one of the Internet’s most vital encryption processes and structures - TLS/SSL Certificate Authorities. Our research has demonstrated that trust placed in this process by governments and authorities worldwide should be considered misplaced at this stage, in our opinion.
We continue to hold concern around the basic reality - we found this on a whim in a hotel room while escaping the Vegas heat surrounding Black Hat, while well-resourced and focused nation-states look for loopholes like this every day. In our opinion, we are not likely to be the last to find inexcusable flaws in such a crucial process.
Although subverting the CA verification process was by far the most devastating of impacts that we uncovered, it was by no means the limit of the opportunity available to us as we also found everything from memory corruptions to command injections. Our ‘honeypot’ WHOIS server gave us some interesting statistics, revealing just how serious the issue is, and a large amount of Internet infrastructure continues to query us instead of the legitimate WHOIS servers.
We do not intend to call out any specific organization or maintainer here - the prevalence of this issue and the statistics on hand show that this is not a pure-negligence or competence related issue - but a fundamental flaw in how these processes work together.
It’s worth noting that all the above attacks that we were able to orchestrate given our takeover are also possible by any entity that is able to carry out MITM attacks - such as entities that control or can influence transit backbones. It would be very easy for an attacker with such access to fake WHOIS data for any domain, and thus obtain valid TLS/SSL certificates. Of course, there has been an insurmountable level of effort by major players to add transparency to this process over the years, and thus, 'pulling off' a heist of this scale has its operational hurdles.
At watchTowr, we passionately believe that continuous security testing is the future and that rapid reaction to emerging threats single-handedly prevents inevitable breaches.
With the watchTowr Platform, we deliver this capability to our clients every single day - it is our job to understand how emerging threats, vulnerabilities, and TTPs could impact their organizations, with precision.
If you'd like to learn more about thewatchTowr Platform, our Attack Surface Management and Continuous Automated Red Teaming solution, please get in touch.
On Wednesday, federal prosecutors charged a North Carolina musician with defrauding streaming services of $10 million through an elaborate scheme involving AI, as reported by The New York Times. Michael Smith, 52, allegedly used AI to create hundreds of thousands of fake songs by nonexistent bands, then streamed them using bots to collect royalties from platforms like Spotify, Apple Music, and Amazon Music.
While the AI-generated element of this story is novel, Smith allegedly broke the law by setting up an elaborate fake listener scheme. The US Attorney for the Southern District of New York, Damian Williams, announced the charges, which include wire fraud and money laundering conspiracy. If convicted, Smith could face up to 20 years in prison for each charge.
Smith's scheme, which prosecutors say ran for seven years, involved creating thousands of fake streaming accounts using purchased email addresses. He developed software to play his AI-generated music on repeat from various computers, mimicking individual listeners from different locations. In an industry where success is measured by digital listens, Smith's fabricated catalog reportedly managed to rack up billions of streams.
“It it ain’t broke, replace it with something that is.”
About five years ago I was suddenly, unexpectedly taken ill. Not just
‘that’s a bit nasty’ ill, but ‘prepare for the worst’ ill. One thing
that kept my spirits up in hospital, in the long watches of the night,
was listening to comedy shows and audiobooks. I used my smartphone for
this, since I had it with me, and a pair of old, wired headphones that I
just had time to grab on my way to the ambulance.
I survived, of course, as evidenced by my continued ramblings on this
site. But it was an unpleasant experience, made just a little better by
a simple piece of technology: the 3.5mm headphone jack.
Now, of course, I do own wireless headphones and earbuds – I think
almost everybody does. I also own several of those irritating USB
dongles, that provide a 3.5mm port for devices that don’t have one. But
here’s the problem: I can’t use my Bluetooth earbuds while they’re
charging. And I can’t easily charge my phone whilst it’s connected to
the USB dongle. In a critical-care facility, it’s hard enough to find
one free mains socket to connect a charger to, let alone two.
In the debate about the benefits of wired and wireless headphones, I
doubt anybody is thinking “What if I get sick?” It certainly wasn’t
something I was thinking, either, until I actually did. Still, that
experience made me think about all the advantages of having a headphone
jack, some of which I’d thought about before, and some I hadn’t.
Sound quality is better. Maybe that won’t always be the case, but it
is now. All decent headphones have an analog jack.
Almost any headphone or earphone will work with almost anything with
a standard jack socket. That isn’t the case for USB dongles. Maybe that
won’t always be the case, but it is now.
I can use the headphone jack to connect my phone or tablet to an
amplifier, should the need arise. For certain headphones, that need
does arise.
I can charge my phone whilst listening to music, which I do all
day.
I don’t have to carry around a stupid dongle, which I will
invariably lose.
Ordinary wired earbuds can be small enough to wear whilst sleeping.
I’ve never seen Bluetooth earbuds that are.
The headphone jack is a “just works” kind of technology: there’s
nothing complicated about it, and its not encumbered by patents, so
anybody can make compatible equipment.
So, although “What if I get sick?” probably isn’t at the top of the
list of questions that will guide your buying decision, we have to
wonder what other things we lose, by replacing a well-established,
robust technology with a complicated, flaky one.
On the other hand, the advantages of doing away with the
headphone jack are:
Your cellphone is about ten pence cheaper.
Er… that’s it.
What makes the loss of the headphone jack so hard to bear is that
it wasn’t done for the consumer’s benefit. To be sure,
manufacturers made certain claims about the alleged benefits of losing
the jack, but few of them stand up to much logical scrutiny.
The first manufacturer to make a point of dropping the headphone jack
(I believe) was not Apple – as is commonly believed – but Oppo, and back
in 2014. Their reason for doing so was at least a credible technical
one: they said it made their phones about half a millimetre thinner.
Maybe that was a selling point, maybe it wasn’t. But Apple couldn’t fall
back even on this claim, because people found ways to fit a 3.5mm jack
socket into the iPhones that lacked one, and even posted videos on
Youtube showing how they did it. It wasn’t easy, but it was clearly
possible. If Apple genuinely thought that omitting the jack
would leave more room for other features, they didn’t actually provide
any.
It’s notable that a number of companies mocked Apple’s decision to
drop the headphone jack, before quietly doing the same themselves:
Samsung and Google in particular. Samsung even cynically tried to
withdraw all the advertising in which they had mocked Apple. Of course,
nothing is ever really gone on the Internet, so we can continue to
marvel at Samsung’s barefaced duplicity.
Some manufacturers claimed that the presence of the headphone jack
made it difficult to keep their phones waterproof; but there’s a whole
range of good-quality phones from around 2019-2020 that are waterproof
to a reasonable degree, without sacrificing the jack.
No. All of these weak excuses are simply distractions from the
real reason Apple, Samsung, and Google dropped the headphone
jack: they all have a substantial investment in the manufacture of
wireless headphones.
Apple owns Beats – this was a multi-billion dollar investment, in a
company that manufactures Bluetooth headphones.
Samsung owns Harmon, which is know for the Harmon-Kardon and JBL
audio brands. Again, these were multi-billion dollar acquisitions for
Samsung, in companies with a strong interest in wireless audio.
Google owns Synaptics, Dysonics, RevX, and Tempow, all of which are
active in the development of wireless audio. Google also hired Peter Liu
from Bose, who was one of the original developers of the Bluetooth Low
Energy specification.
It is very much in the interest of companies like Apple, Samsung, and
Google to encourage their customers to buy into wireless Bluetooth
headphones. Or, better yet, to force them to do so, by taking away the
means to do anything else. After all, their executives have to justify
the billions of dollars they’ve spent, acquiring suppliers and
developers of wireless audio equipment.
The 3.5mm headphone jack has almost nothing to recommend it,
technically speaking. It was criticized by hi-fi enthusiasts almost from
its inception. The only things it has going for it are its simplicity,
and the fact that everybody uses it. Well, everybody outside the world
of mobile gadgets, anyway. This simplicity and ubiquity is a great
benefit to consumers, which is why Apple, et al., don’t want to provide
it – they have nothing to gain if consumers spend less money on their
products.
The loss of the headphone jack would have hurt even if there were
good technical reasons for it. As it turns out, there are none – it’s
just another cynical way for big businesses to gouge consumers. A side
effect of their strategy is that wired headphones themselves are
becoming less available, as cellphone users were until recently major
purchasers of these devices. It’s not difficult to get top-quality
headphones with a jack – they all have one; but there are fewer and
fewer mid-priced wired earbuds on the market.
The way to discourage companies behaving this way is for us all to
take our business elsewhere. It’s still possible to get decent
cellphones from Motorola, Asus, and Sony that retain the 3.5mm jack. I
don’t believe that anything recent from Apple, Samsung, or Google has
one, for reasons I explained earlier. Still, there are older cellphone
models from these suppliers that do have a jack, and which are still
good phones. I own the Samsung S10+, S10e, and Note 9, for example.
They’re a few years old, but they still do everything I want and
more.
If you’re a fan of the 3.5mm jack, it’s time to vote with your
wallet.