If you are reading this article, you are most likely using a web browser, and you have some expectations or beliefs about online privacy and security. For example, I do not know what you are reading on other tabs on your web browser, and you would like to keep it that way. But the websites themselves know that you are reading a particular page on their website. They most likely know your IP address and if you are signed in to their website, they also know your identity. This is not unreasonable because you chose to identify yourself in exchange for certain services. That’s how web works.

You might also be heard about cross site tracking using cookies. Cookies are persistent files set on your web browser by a website to identify you later when you visit the same site. Cross site cookies are set by third-party domains present on a website, and the same third-party might also present in other websites as well. Third party domains track you across your browsing sessions and able to identify you uniquely across different websites. That’s how you are shown ads based on your browsing history. Because the third party is usually an advertising company (cough! Google) and they are present in almost all websites. Even though it seems unethical for a third party to track your browsing history, at-least you had control. Web Browsers allows you to delete cookies, so third parties cannot link you back to your past sessions. This is what Private Browsing does. It basically wipes all cookies (and history) upon closing the window.

Browsers like Firefox now ships with advanced protection against this kind of tracking. They isolate third party cookies per website. This means advertisers or third-parties cannot track you across different websites. This affects advertisement companies revenue because they cannot know your full browsing activity and hence cannot show you personalized ads.

Based on your threat model, even being identified by a first party website across different sessions might be uncomfortable for you. So you might set your web browser to automatically clear cookies or use add-ons to do that.

But companies found another way to uniquely identify you across different sessions and websites without using cookies or other persistent storage. It’s called web fingerprinting. Fingerprinting is a more sophisticated approach to identify a user among millions of others. It works by studying your web browser and hardware configuration. Many websites use a fingerprinting library to generate a unique ID. This library collects data from multiple JavaScript APIs offered by your web browser. For example, websites can see web browser version, number of CPUs on your device, screen size, number of touchpoints, video/audio codecs, operating system and many other details that you would not want a typical news website to see.

All of these values are combined to generate a unique ID. Surprisingly, each user’s device and browser specifications differ so much that they get a unique ID among millions.

I did not think web fingerprinting is serious until I came across a company which is actually selling fingerprinting as a service to other websites. I tried their demo and shocked how accurate it is. Many ecommerce websites use it because these fingerprinting companies sell it, saying it prevents credit card frauds and increases security of the websites.

If you are paranoid like me and use private browsers like Firefox Focus or always clearing cookies when you close the browser, it doesn’t really help to protect your privacy. Web Browsers and Web Standards become so complicated that fingerprinting is easier than you think.

Fingerprinting as a Service

We are going to test a product built by a company called FingerprintJS Inc. who is selling fingerprinting as a service. They make JavaScript fingerprinting libraries which are in fact open source and sell it to many websites. There’s FingerprintJS Pro which is an even scarier version of regular fingerprinting library. It doesn’t matter if you are using a VPN or Private Browsing mode, they can accurately identify you. Here’s how they are describing themselves, “The device identity platform for high-scale applications”.

FingerprintJS has a demo built into it’s homepage, https://fingerprint.com. When you visit this website, they generate a visitor ID (fingerprint) which is unique for your browser. So even if you clear the cache (and other site data) or visit the site in Private Browsing mode, they can generate the same ID and correlate with your previous visit.

My tests on popular web browsers

Now we are going to perform the following steps to prove that fingerprinting works and severely undermines our privacy.

Step 1: Visit https://fingerprint.com

Step 2: View the fingerprint generated.

Step 3: Clear browser cache and all other site data.

Step 4: Visit https://fingerprint.com once again.

Step 5: View the fingerprint and also the previous visit history. Even if the browser has no cookies or other site data, their product can generate the same visitor ID and link it back to our previous visit.

Step 6: Clear browser cache and all other site data.

Step 7: Visit https://fingerprint.com in Private Browsing mode.

Step 8: View the fingerprint and see how it is being correlated to the previous two visits we already made. Yes, in Private Browsing mode.

We are going to perform these tests on Firefox, Chromium, and Tor Browser.

Firefox

Notice how different sessions are connected by the same fingerprint generated by FingerprintJS. Firefox in its default configuration is prone to fingerprinting.

Firefox with privacy.resistFingerprinting = true

Firefox has a setting called resistFingerprinting (initially contributed by The Tor Project) that makes it more resistance to fingerprinting. When activated, Firefox tries to mask certain properties like User Agent, CPU Count, Timezone, Screen Resolution etc. uniform for all users. This makes it harder for fingerprinting.

You can enable it by visiting about:config and setting privacy.resistFingerprinting = true in your Firefox browser.

This time, FingerprintJS could not link it with previous sessions. Each session gets a unique ID since Firefox hardens certain APIs against fingerprinting.

Chromium / Chrome

Chromium (Chrome) is built by Google, an advertisement company which tracks its users for showing relevant ads. So naturally it doesn’t have any inbuilt protection against fingerprinting. Chromium (and Google Chrome) is vulnerable to fingerprinting.

FingerprintJS generates the same ID in each Chromium session, thus it can identify its users across different sessions.

Tor Browser

Tor Browser is made by The Tor Project, a non-profit organization. Tor Browser routes internet traffic through multiple relays across the world, thus making user’s browsing sessions more private. It is based on Firefox and many features of Tor Browser has been incorporated back in Firefox.

Please note that Tor Browser always operates in Private Browsing mode. So I did not test it under Private Browsing explicitly.

FingerprintJS could not link two different Tor Browser sessions by the same user. So Tor Browser is more secure against fingerprinting.

Conclusion

Fingerprinting has become a popular method of user tracking due to its ability to connect multiple different browsing sessions even if the user clears browsing history and data. Given there are companies selling fingerprinting as a service, if you want to really protect yourself from fingerprinting, you should use Tor Browser or Firefox with resistFingerprinting=true. If you need to use Chromium, then Brave browser is a good choice. It also randomizes fingerprint for each session, making it harder to link your browsing sessions. However, I do not recommend Brave because it is based on Google’s Chromium engine, thus only encourages Google’s monopoly.

On mobile, only Tor Browser and Firefox with resistFingerprinting=true were able to protect against fingerprinting. Firefox Focus leaks fingerprints even if you clear its session each time. Also note that VPNs does not help with fingerprinting. They only masks IP address.

Adblock test (Why?)

In this blog post, we explore the topic of fake GitHub stars. We will share our approach for identifying them and invite you to run this analysis on repos you are interested in. Click here to skip the background story and jump right to the code.

And if you enjoy this article, head on over to the Dagster repo and give us a real GitHub star!

GitHub stars are one of the main indicators of social proof on GitHub. At face value, they are something of a vanity metric, with no more objectivity than a Facebook "Like" or a Twitter retweet. Yet they influence serious, high stakes decisions, including which projects get used by enterprises, which startups get funded, and which companies talented professionals join.

Naturally, we encourage people interested in the Dagster project to star our repo, and we track our own GitHub star count along with that of other projects. So when we spotted some new open-source projects suddenly racking up hundreds of stars a week, we were impressed. In some cases, it looked a bit too good to be true, and the patterns seemed off: some brand-new repos would jump by several hundred stars in a couple of days, often just in time for a new release or other big announcement.

We spot-checked some of these repositories and found some suspiciously fake-looking accounts.

We were curious that most GitHub star analysis tools or articles that cover that topic fail to address the issue of fake stars.

We knew there were dubious services out there offering stars-for-cash, so we set up a dummy repo (frasermarlow/tap-bls) and purchased a bunch of stars. From these, we devised a profile for fake accounts and ran a number of repos through a test using the GitHub REST API (via pygithub) and the GitHub Archive database.

So where does one buy stars? No need to go surfing the dark web. There are dozens of services available with a basic Google search.

In order to draw up a profile of a fake GitHub account used by these services, we purchased stars from the following services:

• Baddhi Shop - a specialist in low-cost faking of pretty much any online publicly influenceable metric. They will sell you 1,000 fake GitHub stars for as little as $64.
• GitHub24, a service from Möller und Ringauf GbR, is much more pricey at €0.85 per star.

To give them credit, the stars were delivered promptly to our repo. GitHub24 delivered 100 stars in 48 hours. Which, if nothing else, was a major giveaway for a repo that, up until then, had only three stars. Baddhi Shop had a bigger ask as we ordered 500 stars, and these arrived over the course of a week.

That said, you get what you pay for. A month later, all 100 GitHub24 stars still stood, but only three-quarters of the fake Baddhi Shop stars remained. We suspect the rest were purged by GitHub’s integrity teams.

We wanted to figure out how bad the fake star problem was on GitHub. To get to the bottom of this, we worked with Alana Glassco, a spam & abuse expert, to dig into the data, starting by analyzing public event data in the GitHub Archive database.

You might be tempted to frame this up as a classical machine learning problem: simply buy some fake stars, and train a classifier to identify real vs fake stars. However, there are several problems with this approach.

• Which features? Spammers are adversarial and are actively avoiding detection, so the obvious features to classify on - name, bio, etc - are generally obfuscated.
• Label timeliness. To avoid detection, spammers are constantly changing their tactics to avoid detection. Labeled data may be hard to come by, and even data that is labeled may be out-of-date by the time a model is retrained.

In spam detection, we often use heuristics in conjunction with machine learning to identify spammers. In our case, we ended up with a primarily heuristics-driven approach.

After we bought the fake GitHub stars, we noticed that there were two cohorts of fake stars:

• Obvious fakes. One cohort didn't try too hard to hide their activity. By simply looking at their profiles it was clear that they were not a real account.
• Sophisticated fakes. The other cohort was much more sophisticated, and created lots of real-looking activity to hide the fact that they were fake accounts.

We ended up with two separate heuristics to identify each cohort.

During our fake star investigation, we found lots of one-off profiles: fake GitHub accounts created for the sole purpose of "starring" just one or two GitHub repos. They show activity on one day (the day the account was created, which matches the day the target repo was starred), and nothing else.

We used the GitHub API to gather more information about these accounts, and a clear pattern emerged. These accounts were characterized by extremely limited activity:

• Created in 2022 or later
• Followers <=1
• Following <= 1
• Public gists == 0
• Public repos <=4
• Email, hireable, bio, blog, and twitter username are empty
• Star date == account creation date == account updated date

Using this simple “low activity” heuristic, we can detect many (but hardly all) suspected fake accounts that starred the same set of repositories, using nothing but data that’s available from the GitHub API.

The other cohort of fake accounts was quite challenging to identify. They had realistic human activity: they had profile pictures, bios, and realistic contribution activity. The simple heuristic above didn't identify these accounts as fake, even though we knew they were from our purchase from vendors. How could we identify these realistic—yet known to be fake—accounts?

Clustering intuition

We ended up settling on a technique called unsupervised clustering. Conceptually, we wanted to construct a set of features per account. Normal users should be fairly diffuse; that is, their features should be fairly unique to them and they should not belong to any large clusters. Fake users, on the other hand, would all share similar features and would cluster together when visualized. In order to detect fake users, we'd simply check to see if they are part of a suspicious-looking cluster.

This is easiest to understand with an example. Consider "activity date". Most users on GitHub don't have public activity every day. If an account uses GitHub a few days a month, and those days are the exact same as another account, and they share similar activity on those days, it's a sign that, potentially, both accounts are controlled by the same underlying script.

To get a sense of what this looks like, we can plot the set of users who starred a given repo by the number of activity dates they share with other users (x-axis) and the total number of repositories they interacted with (y-axis):

Here’s an example of what that looks like for our dummy repo, which has nearly 100% fake stars:

And for the Dagster repo, which has (as far as we know) zero fake stars. Notice the tiny yellow dots in the bottom left corner, which represent a handful of false positive accounts (false positive rate = 0.17%):

And finally, for something in the middle, we can take a look at one open source project repository that has a large number of suspected fake stars mixed in with real engagement. The clusters of fake GitHub users stood out.

Improving the clustering

While this initial technique was interesting, it wasn't good enough at identifying high-confidence fake users on its own. We needed to improve on it.

After digging into the data from this initial intuition, we discovered another pattern. Though these sophisticated fake accounts were interacting in realistic ways, we found that all the fake accounts tended to interact with a small set of repositories shared between them. Essentially, each fake account seemed to be adding stars to a subset of overarching "suspicious repositories."

Unfortunately we couldn't just find that list of repositories and be done as spammers would constantly rotate through new sets of repositories to star. However, we could use our unsupervised clustering techniques to identify these suspicious repos automatically. Then we would know which accounts were fake based on whether (and how much) they interacted with these suspicious repos.

Specifically, here is how we identified whether or not a user was suspicious:

• First, we pulled a list of all the users who starred the repository we're interested in analyzing.
• Then, we identified a set of potentially suspicious repositories based on high overlap with other users in this set. Remember, since these users are already related by having starred the same repository initially, it could be suspicious if they star a large number of overlapping additional repositories. (It could also just mean that this cluster of repositories is legitimately interesting to the same set of users - which is why the additional step below is important!)
• Finally, we looked for accounts with relatively low activity levels, where the vast majority of their activity was on the set of suspicious repositories identified above, and that had no additional legitimate-seeming activity. These were our fake accounts.

When we tested this heuristic on the known fake stars in our dummy account, we found that while it could be very computationally expensive, it was both very good at detecting fake accounts and also extremely accurate (98% precision and 85% recall). Things are bound to be a bit messier in a real repository, so we were eager to test this out.

By putting these two methods together, we can get a more complete picture of the suspicious engagement found on a given GitHub repository and the recall of each method:

Footnote: Due to the compute cost of running the calculations, the GitHub Archive analysis done on BigQuery was limited to stars gained starting Jan 1st 2022. For the GitHub Archive analysis, a slightly different method was used to identify the same type of suspicious "low activity" accounts that were found in the GitHub API analysis. These accounts were combined with the additional suspicious accounts identified through our clustering method, to give us the total suspected fake star count.

If you would like to analyze other Github repositories using this logic, you will find a full Dagster and dbt project on Github here. The simple heuristic is implemented in Python, so all you will need is a Github account and an access token.

The unsupervised clustering method is implemented as a dbt project, so you'll need a Google Cloud BigQuery account to run it. Note that for large repos it may be quite costly to run.

Conclusion

Building models for detecting fake accounts (or other forms of spam) with 100% accuracy is hard. We can develop techniques with fairly high precision and recall but they become computationally expensive. Furthermore, the world is continuously changing around us, and many models need constant adjustment. Sometimes going with a simple heuristic can get us enough data at a fraction of the cost, so it's always good to keep a range of options in mind. And finally, just because a repo has some fake stars, it doesn't necessarily mean that the owners bought them, as spammers often try to hide their fake accounts by starring legitimate repos.

Given the incentives at play here, and the ease with which one can buy fake GitHub stars, it's heartening to see that this is not a widespread phenomenon, and speaks well of the values of the broader developer community.

However the option of stuffing a repo with fake stars exposes gaps in GitHub's own trust and safety mechanisms and there are certainly projects that have exploited them. We've shared these findings with the GitHub team, so don't be surprised if all of the fake accounts we detected in this tutorial disappear in the coming months, and the GitHub star count on those repositories takes a dive.

---

We're always happy to hear your feedback, so please reach out to us! If you have any questions, ask them in the Dagster community Slack (join here!) or start a Github discussion. If you run into any bugs, let us know with a Github issue. And if you're interested in working with us, check out our open roles!

Follow us:

Adblock test (Why?)

Over the last week, three U.S. banks have failed. More banks are under extreme stress. This stress is not new and was not unknown but is becoming common knowledge rapidly. We may be in the early stages of a banking crisis.

“Crisis” is a bit of a strong word, even when invoked as a potential outcome, and I try to be fairly sober-minded. I’d like to explain how we got here, how the relevant institutions are generally expected to work, what seems to be different this time, and what smart people who are not normally professionally engaged in this might find relevant to know about the infrastructure that we all depend on.

Short disclaimer: I worked at Stripe (which is not a bank, but works with many banks) for six years prior to leaving full-time employment recently. I am an advisor there now. My views are entirely my own, and my analysis is only informed by publicly available data. I put a longer disclaimer at the bottom.

Why are banks failing?

As we previously covered in a discussion about deposit insurance, now unfortunately topical, banks do not fail in a day. The seeds of their destruction are sewn and watered for years, and then they are reaped quickly.

Importantly, these do not say “seeds of destruction: definitely don’t plant these!” on the package. People have a great desire for there to be a narrative here; for a bank failure to require stupidity or malfeasance or ideally stupid malfeasance.

The thing killing banks is a very simple idea with profound consequences. It is not a secret: when interest rates rise, all asset prices must fall. This is both almost a law of nature and also perpetually underestimated in how much it affects the world outside of asset prices. For example, in January 2020, I pointed out that obviously engineering compensation includes an interest rate derivative, because it includes equity. This is very not obvious to many people in tech, including financially sophisticated people!

But equity is not the only thing that embeds an interest rate derivative. All prices embed an interest rate derivative. The price of eggs embeds an interest rate derivative, among many other things, like how it reflects the cost of grain. The price of grain embeds an interest rate derivative. The world sits atop four elephants who stand astride the risk-free rate, and then it is interest rates all the way down.

The price of eggs, and other important parts of the consumer basket, is a major contributing reason why we are here. The United States (through the Federal Reserve) made a considered decision to manage inflation by hiking interest rates. That is, explicitly, an intervention to push down the price of eggs (and other things), via a lever which happens to be much more amenable to direct action than other available levers for controlling egg prices. This lever can be applied across the entire consumer basket in parallel. And so it was.

If you recall the ancient history of *checks notes* the past 15 months, we went from a regime where prevailing interest rates were just above zero to almost 5%. This was the most aggressive hike in rates since World War II, or to put it another way, in the history of the modern economic order.

The decision to sharply manage down the price of eggs was, indirectly but inescapably, also a considered decision to cause large notional losses to all holders of financial assets. That includes everyone with a mortgage, every startup employee with equity, and every bank.

That is the proximate cause of the banking crisis, if in fact we are in a crisis. Three banks failed first, because for idiosyncratic reasons they were exposed to sudden demands for liquidity, which makes large declines in the value of one’s assets unsurvivable. But there are many more banks which have a similar issue on their balance sheet.

A useful heuristic from bond math

I apologize for a very 101-level financial math lesson but it’s unavoidable, useful, and may not have featured in your education (and, of course, Matt Levine beat me to mentioning it): there is a heuristic for the value of bonds.

Every bond and instrument created on top of bonds has a “duration”, which you can round to “how much time left in years until we expect this to be paid back?” And every bond and instrument on top of bonds has its market price move down by 1% per year of duration if interest rates move up 1%, and vice versa. (There is better math available but this is math you can trivially perform in your head, and is close enough to blow up large portions of a financial system.)

So if you held ten year bonds and interest rates went up 4% in a year, your ten year bonds are down, hmm, somewhere in the 35%ish range. This is true regardless of whether the bonds are good bonds. If you want to sell them today, the people buying them have better options than you had a year ago, and to induce them away from those better options you have to give them a 35%ish discount.

We now come to one of the most important charts in the financial world, courtesy of the FDIC in February:

$620 billion. The U.S. banking system lost $620 billion. Six hundred twenty billion dollars. That is a loss no less real than if money had been loaned out to borrowers who defaulted. It might be temporary! If interest rates go down, bond prices will recover. (And sometimes defaulting borrowers receive an inheritance or get bailed out! But one doesn’t generally want to count on that.)

But, for the moment, banks are out $620 billion and the Fed recently signaled more aggressive rate hikes.

Was this because the banks invested in poor credit? No. The price of everything embeds an interest rate derivative, including definitionally perfect credit like U.S. Treasuries. The type of security most numerically relevant here is functionally immune to credit risk: agency-issued mortgage-backed securities.

You might remember that financial instrument from 2008. Many people are going to fixate on that coincidence far more than is warranted. In 2008 those embedded bad-and-mispriced credit risk which had an uncertain backstop. In 2023 the losses are caused by a bad-and-mispriced interest rate risk with a rapidly evolving backstop. But all asset prices include an interest rate derivative.

“Why do banks buy exotic assets with lots of letters in the name, like MBS from GSE? Why can’t they just do banking? Like, make regular loans to real people and businesses with income to service them? That would surely solve this, right?”

It would not. If they created loans with fixed rates, just plain vanilla loans warehoused on their own balance sheet in the “traditional business of banking”, the rate environment would have exactly the same effect. It already has had this effect.

In addition to the $620 billion in losses in securities, there exist staggering losses in the loan books of every bank that wrote fixed rate loans in 2021. And 2020. And 2019. And 2018. And 2017. And 2016. And 2015. And 2014. And 2013. And 2012. And 2011. And 2010. And 2009.

Most people sensibly don’t care about any of this, and only care when a financial product which is core to their lives—bank deposits—suddenly and unexpectedly ceases to function. Bank deposits are much more complicated products than they are believed to be. When banks fail, the most important societal impact is that deposits, which are money no less real than physical script and in many ways much more real, suddenly have an unanticipated risk of not being money.

Maturity transformation

What is the connection between deposits, bank runs, and the value of ten year bonds in conditions of rising interest rates? I’m glad you asked.

You pay an explicit bill to most businesses which provide you valuable services. You get deposits for free*, emphasis on the asterisk. The tellers and the lawyers and the engineers and the regulators and the insurance company and the equity providers who collectively must labor diligently to give you deposits still need to get paid. They get paid largely by harvesting the option value from depositors as a class and creating something new out of it.

Banks engage in maturity transformation, in “borrowing short and lending long.” Deposits are short-term liabilities of the bank; while time-locked deposits exist, broadly users can ask for them back on demand. Most assets of a bank, the loans or securities portfolio, have a much longer duration.

Society depends on this mismatch existing. It must exist somewhere. The alternative is a much poorer and riskier world, which includes dystopian instruments that are so obviously bad you’d have to invent names for them.

Take an exploding mortgage, the only way to finance homes in a dystopian alternate universe. It’s like the mortgages you are familiar with, except it is callable on demand by the bank. If you get the call and can’t repay the mortgage by the close of the day, you lose your house. What did you do wrong to make the mortgage explode? Literally nothing; exploding mortgages just explode sometimes. Keeps you on your toes.

Exploding mortgages don’t exist and can’t exist in our universe. But it is important that, from a bank’s perspective, the dominant way people bank sometimes explodes. That asymmetry is the mismatch. We expect banks to manage this risk, and we expect society to tolerate it (and sometimes cover the bill for it), because exploding mortgages are worse than this risk.

We have moved some of this mismatch out of the banking system, by e.g. securitizing mortgages and selling them to pension funds which can match them against natural liabilities (e.g. actuarial tables of when pensioners will retire and require their payouts). But the banking system holds a lot of duration mismatch risk, and likely always will.

This is, like all the other risks to banks, something which is managed and regulated. Sometimes management screws up or priorities their bonuses over prudential risk mitigation. Sometimes regulators are, feel free to choose your phrasing, asleep at the switch or not sufficiently empowered.

Can I excerpt that FDIC speech from three weeks ago? While the FDIC obviously must moderate their public comments, this is the payload:

> Unrealized losses on available–for–sale and held–to–maturity securities totaled $620 billion in the fourth quarter, down $69.5 billion from the prior quarter, due in part to lower mortgage rates. The combination of a high level of longer–term asset maturities and a moderate decline in total deposits underscores the risk that these unrealized losses could become actual losses should banks need to sell securities to meet liquidity needs.

This is very measured language. Equally true language is: about a quarter of all equity in the banking sector has been vaporized by one line item. I was surprised to learn this.

The sacred duty of equity is to protect depositors from losses. After it is zeroed, the losses must come from somewhere. We do not celebrate equity getting vaporized, except insofar that sacrifice of oneself in satisfaction of a duty to others is generally praiseworthy, but we certainly want to be aware that it happened.

The world is, belatedly, realizing that this did actually happen. Past tense.

This realization creeped in around the edges with e.g. Byrne Hobart on February 23rd noting that one of the U.S.’s largest banks was recently technically insolvent but almost certainly in a survivable way. And, to be fair, a few short funds and the Financial Times had come to this realization a bit before Byrne. Then, a few weeks later, the entire financial system almost simultaneously discovered how much they doubted precisely one half of his thesis.

I submit to you that the regulators probably did not understand a few weeks ago that this situation was factually as concerning as it is.

Don’t read this as a statement about competence or the lack of it; just read it as a factual claim about the constitution of the Problem Bank List. The Problem Bank List is figurative state secret, specifically to prevent inclusion on the PBL from causing a run on the bank if it were to become common knowledge.

At least one bank which failed last week was not a Problem Bank three weeks ago. Reader, that should not ever happen.

“How do you know this if the Problem Bank List is a state secret?” Because they report the aggregate total of the assets of all banks on the list and publicly available data plus math a 4th grader can do in their head suffices to prove this claim.

Finance is an industry with many smart people in it. The same goes for regulatory agencies. You’re welcome to your guess of how many of them asked a 4th grader “Were all the banks which failed this week on the Problem Bank List or do we have an unknown unknown?” prior to reading this paragraph.

There exists this same problem at banks that are not on the Problem Bank List. I would normally hedge that sentence with something like “likely”, but the market has woken up and is now aggressively repricing risk and publishing findings. Those findings are deeply concerning and, for social reasons, I must direct you to the financial media of your choice to read them.

We went multiple years without a bank failure, of any size, in the United States. We then had three in a week, including one (by some measures) larger than any during the last financial crisis. It would take a very brave and confident person to forecast no additional bank failures in the next two weeks. It would take a very interestingly calibrated person to say that, contingent on there being a bank failure, that that bank must necessarily have been on the Problem Bank List.

Liquidity problems are the proximate cause of bank failures

The reason for relative sanguinity about unrealized losses in the banking sector denominated in the hundreds of billions to low single digit trillions of dollars, and forgive me for harping on that fact but it is a fact about the world we live in, is that banks do not need to pay out all deposits simultaneously. Functionally no bank anywhere could do that, and the theoretical exception is considered not desirable as a matter of public policy and therefore does not exist.

Banks designate certain assets on their books as “available for sale”, those which they expect to perhaps sell to raise liquidity, and “held to maturity.” Losses in the ATS portfolio are relatively noisy, because they immediately ripple into one’s income statement, are reported quarterly, and are extremely salient for all stakeholders. Losses in the HTM securities are basically fine until they aren’t.

This isn’t entirely because management prefers to keep its head in the sand. Banks are institutions designed to exist over timelines longer than interest rate cycles. This implies certain assets of theirs will always be underwater and certain assets of theirs will always be “worth more than we paid for them.” To the extent that the bank is simply holding the asset to collect the income from it this all comes out in the wash. The day-to-day movements are in normal times a distraction and get relegated to a footnote.

We do not expect the footnote to swallow the bank, and that is an important update to our model of the world. We do not expect it to swallow multiple banks. We do not expect to not have a high-quality estimate for how many banks it will swallow in the next two weeks.

The three bank runs which already happened had idiosyncratic causes, but “if accounted for accurately, the bank is insolvent” is the sort of thing which, if one stipulates to it, one would suggest might generate bank runs in the near future. And so there was a policy response, which much commentary has assumed is primarily about the banks which no longer exist, and the satisfaction of their depositors, and which is actually much more about banks in danger which might yet be saved.

Trying to forestall a banking crisis

The losses banks have taken on their assets are real. They already happened. They are survivable if banks remain liquid.

The Federal Reserve, Department of the Treasury, and Federal Deposit Insurance Corporation released a joint statement over the weekend to adjust people’s expectations regarding banks that still exist. The key element of the response is a temporary extension of credit to banks collateralized by high-quality assets at their par value, rather than their market value. This is called the Bank Term Funding Program.

The hope is that a bank facing liquidity pressure could tap this credit program, in addition to existing credit programs and source of liquidity, and thereby avoid a downward spiral of selling assets, realizing losses, pushing asset prices down, spooking markets and depositors, and repeating at a very high cycle rate until the bank doesn’t exist.

We recently went through that cycle faster than we thought possible with regards to a bank which responsible people considered very safe. According to the official record, one of the institutions went from being financially healthy one day to insolvent the next. I believe that narrative to be face-saving, but it is what The System currently is messaging as the truth, so let’s accept it for now. If this is the truth, what unfortunate truths might we learn in the near future?

This is a temporary program; banks can only tap this liquidity for about a year. In the ordinary course, bank runs don’t last for a year; they either cause an institution to fail very quickly or peter out. But the other reason this is time-bounded is to defang the moral hazard, on behalf of both banks and their customers. (Moral hazard in insurance is when the existence of insurance makes it incentive-compatible for you to be imprudent in your own risk taking, expecting someone else to bear the consequences.)

Banking regulators want banks to take the strong medicine solution to the problem.

If banks have experienced hundreds of billions to single digit trillions of dollars in losses, realized or no, they have a very limited set of options. Hoping for a miracle is one. Experiencing a sudden dramatic shift downwards in interest rates, which would cause them windfall gains for exactly the reason they experienced windfall losses, is another. Grinding out many years of profits in the ordinary business of banking to fill the hole is a third.

But the thing which is actually within their immediate ability and control is simple and painful. The sacred duty of equity is to take losses before depositors do. Equity has taken losses. Depositors must be shielded. Equity must be raised to take the losses again.

Equity, of course, has a choice in a free market system as to which risks it wants to take. It flowed into banks in good times at prices banks were reasonably happy with. They now need to raise in what is no longer a good time, at prices banks (and existing equity holders, etc) will not be happy with, because the new marginal equity appreciates the risk environment it is entering more than the equity raised a while ago.

This is the short explanation for why bank stocks are getting hammered right now. A share is a one-over-some-denominator claim on the equity of the bank. Sophisticated people are realizing that the numerator is lower than they expect and the denominator is shortly to be larger, and potentially much larger, than they expect. Existing shares are perforce worth less than they were before we woke up to this realization. Banks will need to go to the market to sell new shares at these less favorable prices.

Count this as another knock against the strong-form efficient market hypothesis. None of these dynamics are particularly complicated by the standards of finance. The core facts are not secrets; they were exhaustively disclosed on a quarterly basis. Charts were made.

Anyone could have made a killing if they put two and two together even a week ago. A killing was, mostly, not made. (Killings perhaps remain available as of this writing, if that is your thing.)

Deposit insurance expansion

Bank deposits in the U.S. are insured up to $250,000 per depositor per account type per institution. The exact definition of “account type” is a sort of wonky detail; just assume it is $250,000 historically per depositor/institution pair and you’ll save some braincells for the meatier issues.

By special and extraordinary action, the FDIC has announced that two recent bank failures will backstop all deposits, not just all insured deposits. Much commentary has focused on the decision to create winners out of losers vis depositors at those two institutions.

This is an effect of the policy but is neither the intent nor the rationale.

Let me speculate about some things which may have happened this weekend, with arbitrarily high confidence.

Over the weekend, the regulators made some calls and asked regional banks what deposit outflows looked like on Friday and how many wires were queued up for execution Monday morning. This was complicated by some banks finding it surprisingly difficult to add numbers quickly. You see, the core puts the queued wired requests in a different part of the system than Friday’s outflows. We have a report of Friday outflows, but it gets crunched by an ETL job which only finishes halfway through Saturday, and Cindy who understands all of this is on vacation, and… and eventually very serious people said Figure Addition The #*(%#( Out And Call Me Back Soonest.

Regulators then heard the numbers, did a bit of modeling in Excel, and then went into wartime execution mode. Regulators have, of course, not declared this war, because it is a war on the public’s perception of reality, and to declare war is to surrender.

The $620 billion in losses on securities and the concomitant loss on loans is not distributed evenly across the U.S. banking sector, but it is distributed across the U.S. banking sector. Every institution thanking its risk managers for them having a below-average amount of it implies that some other institution has more of it.

And so we are in a situation where some institutions, whose names are not yet in headlines but may be very shortly indeed, are under acute stress. And we are also beginning to understand a mechanism by which a handful of institutions fell off a precipice, where we understand the edge of that precipice to be eroding, because we currently believe interest rates will go up again. (That belief is shifting rapidly; the rapid decline in 2 year Treasury yields is a sign that the markets are adjusting expectations and beginning to doubt the forecast future sharp hikes.)

Financial institutions are also adjusting to the new reality rapidly. Over the weekend, like every other customer of a particular bank, I got an email from the CEO explaining that they had ample liquidity but had just secured a few tens of billion of additional liquidity, prudent risk management, no problems here, all services are as up as ever, yadda yadda yadda.

Securing more liquidity may be prudent, and the announcement of securing liquidity may be prudent, but this is not an email you send to all customers in good times. Banks typically take communications advice from the Lannisters: anyone who needs to say they have adequate liquidity does not have adequate liquidity. History is replete with examples. Bank CEOs know this. They know their sophisticated customers know this. And yet that email was still written, reviewed by management and crisis comms and counsel, and then sent.

Deposit insurance also some legacy issues

Deposit insurance is an important piece of social technology, and so successful that some believe that it is the primary reason deposits are safe. It is, of course, the backstop to the primary things which make deposits safe, which is the ordinary risk management of banks, a complex and mostly effective regulatory regime, and $2.2 trillion of private capital that signed up to be incinerated if there are faults in earlier controls. The deposit insurance fund, by comparison, is about $130 billion, which you can compare to that $620 billion in losses number prior to thanking capital for its service to society.

But, much like we’ve previously talked about how credit cards are legacy infrastructure, deposit insurance is also legacy infrastructure. It is designed to adjust the expectations of large numbers of relatively slow-acting low-sophistication users by credibly dampening the pain to “regular users of the banking system” that banking stress threatens.

But the world deposit insurance now protects is different than the one it was developed in, and I think it may need to be updated. One much remarked upon elsewhere is that some banks have hypernetworked customer bases who can through relatively independent action tweet and WhatsApp themselves to withdrawing $42 billion in a day.

But deposit insurance is institutionally aware that some institutions have concentrated deposits and lots of deposits are controlled by sophisticated actors. We had capital-intensive businesses with chainsmoking professionals who'd prefer their businesses to survive a bank run during all the relevant crises. The architects of deposit insurance knew these people exist and that they were a primary vector for runs historically. This problem is planned for. It was not created by Twitter.

Let's talk about the problem it doesn't institutionally prepare for. The entire edifice of deposit insurance rests on the assumption the primary harm from bank failure, at least that worthy of societal attention, falls first on direct depositors of the bank and secondly on spillover stress in the rest of the system.

This is a reasonable model, and like all models it is wrong but useful.

Consider the case of Rippling, a startup I have no affiliation with. Rippling has a complicated business; one portion of that is being a payroll provider. Payroll providers, as a type of business, are much older than iPhones but effectively younger than many policy measures designed to mitigate banking crises. (Rippling is a tiny one; some exist in the Fortune 500.)

When Rippling’s bank recently went under, there was substantial risk that paychecks would not arrive at the employees of Rippling’s customers. Rippling wrote a press release whose title mostly contains the content: “Rippling calls on FDIC to release payments due to hundreds of thousands of everyday Americans.”

Prior to the FDIC et al’s decision to entirely back the depositors of the failed bank, the amount of coverage that the deposit insurance scheme provided depositors was $250,000 and the amount it afforded someone receiving a paycheck drawn on the dead bank was zero dollars and zero cents.

This is not a palatable result for society. Not politically, not as a matter of policy, not as a matter of ethics.

Every regulator sees the world through a lens that was painstakingly crafted over decades. The FDIC institutionally looks at this fact pattern and sees this as a single depositor over the insured deposit limit. It does not see 300,000 bounced paychecks.

Payroll providers are the tip of the iceberg for novel innovations in financial services over the last few decades. There exist many other things which society depends on which map very poorly to “insured account” abstraction. This likely magnifies the likely aggregate impact of bank failures, and makes some of our institutional intuitions about their blast radius wrong in important ways.

What would happen if my bank were to go into receivership this weekend?

We covered this previously, but the dominant answer historically is that it is sold and you have a new bank on Monday with functionally nothing else changing. The system has worked very well; we have gone years since the last bank failure, most failures are small, most are entirely resolved by the following Monday, and even deposits over the limits held at banks which failed have rarely taken losses over the last few decades. On the few occasions they have, those losses have been miniscule.

The system recently looked at the combination of published rules, availability of a transaction over the weekend, degree of surprise, and preparedness of suitors… and it blinked, because of what it could actually have delivered on Monday (yesterday).

That would have been full satisfaction of insured deposits, perhaps fifty cents on the dollar satisfaction of uninsured deposits, and a few months of uncertainty as to the timing and level of eventual satisfaction for the remainder. Actual losses would have probably been zero or a few cents on the dollar, eventually, probably.

That resolution is a much worse resolution than the one the system typically obtains and it would have affected many more people than is typical. This may be, if not the new normal, a new concerning potential recurring pattern during uncertain times.

People may have a mental model that a bank keeps a list of all its customers and can therefore quickly calculate e.g. who is insured and to what degree, so that it can pass this list to the FDIC, so that those people can get their money on Monday. This is a useful mental model for first approximations and does not actually describe the world you live in.

For example, FDIC insurance insures the “actual owners” of accounts, and not the entities those accounts are titled to. One important type of account which exists in the world is the For Benefit Of (FBO), where someone might hold money in trust for someone else in their own name.

FBO aren’t newfangled things dreamt up in Silicon Valley. Trusts as an institution date back to the middle ages; regulations have successfully anticipated how they used to be used.

Decades ago, the dominant mental image people might have had for FBO accounts was Lawyer Larry holding a settlement on behalf of Client Carla because lawyers are more like banks than regular people are like banks. The FDIC insures Carla, not Larry, even if Larry has fifty Carlas commingled in a single account and the bank only knows them as “names available on request.” (This is perhaps surprising for people who think banks need to Know Your Customers. The bank customarily adheres to its written policy about KYC for FBOs. Their regulator is OK with the policy. All of this is the normal business of banking and entirely uncontroversial.) To make Carla whole, it has to learn Carla exists first, which implies a process that cannot conclude by Next Monday.

Well that’s an edge case, right. Lawyers and FBO accounts have to be a teeny tiny percentage of all deposits and, while this would be greatly inconvenient for Carla, presumably if she is still banking through her lawyer in 2023 she is rich and sophisticated.

Let’s talk about fintech.

Many fintech products have an account structure which looks something like this sketch: a financial technology company has one or several banking relationships. It has many customers, enterprises which use it for e.g. payment services or custodying money. Those services are not formally bank accounts, but they perform a lot of feels-quite-bankish-if-you-squint to the people who rely on them to feed their families. The actual banking services are provided to those users by the banks, who are disclosed prominently on the bottom of the page and in the Terms and Conditions.

Each enterprise has their own book of users, who might number in the hundreds of thousands or millions, in a single FBO account at the bank, titled in the name of the enterprise or the name of the fintech. The true owners of the funds are known to the bank to be available in the ledgers of the fintech but the bank may have sharply limited understanding of them in real-time.

And so I ask you a rhetorical question: is this structure robust against the failure of a bank handled other-than-cleanly, such that, come the following Monday, those users receive the insurance protection which they are afforded by law? Mechanically, can that actually be done? Is our society prepared to figure that out over a weekend? Because during this past weekend, that sketch I wrote out about banks being confuddled by addition for a few hours almost certainly happened.

There are a sharply finite number of hours between Friday and Monday and we cannot conveniently extend them to cover multiparty discussions about how to get a core system to import a CSV dumped by a beleaguered data scientist from Jupyter based on a hopefully up-to-date MongoDB snapshot so that it can be provided to the FDIC agents on site.

I am very frustrated by political arguments about desert, which start with an enemies list and celebrate when the enemies suffer misfortune for their sins like using the banking system.

Be that as it may: most enemies lists do not include taxi drivers, florists, teachers, plumbers, etc etc you get the drift literally every strata of society is exposed to products which bank for them in complicated ways. These people will be hurt by bank failures. We as a society do not accept this, which is a large portion of why they are protected if they bank directly with a financial institution, and why we promise they are protected if their money is in a more complicated account structure.

I am very sure our society and institutions are operationally capable of delivering on the promised and counted-upon protection for some of the ways these depositors access banking services.

Many people who read this might feel a bit of negative surprise that structures like this sketch exist in the world and are deployed pervasively. ("Was that allowed?! Where were the regulators?" Yes. The usual places.)

Interestingly, that has not been the dominant worry about the adequacy of deposit insurance in the fintech industry. The dominant worry, among clueful people on this narrow and wonky topic, has been that deposit insurance would not protect some people exposed to structures where the bank survived but the fintech did not.

Given this worry, fintechs trumpeting FDIC insurance to mean that users faced de minimis risk of loss of funds felt like misselling what they were offering.

The good news: it seems like the problem we’re immediately faced with is the sort of thing that deposit insurance actually insures against: the failure of financial institutions. The hypothetical losses would be covered. The bad news: banks are failing and more may fail, potentially including some banks with customers that have business models younger than Its A Wonderful Life (1946).

It is not obvious to me that people, including people in positions of authority and responsibility, understand that society has wandered its way into commitments shaped like this one. But it has, and so maybe they should (while dealing with the other fires) seek to gain more understanding of the current operation of financial infrastructure that is pervasively deployed and pervasively relied upon by many people, including arbitrarily sympathetic people.

Not that I think someone needs to be sympathetic to be worth a duty of care here. Infrastructure undergirds society; failures of it are a per se emergency. Anyone who cheers an infrastructure failure because of the first order consequences of it will find themselves negatively surprised.

What should users of the banking system do?

I suggest that you go to someone who actually has a professional duty of care to you, but that feels unsatisfying, and so let me make some general observations.

One is that the banking system is more resilient than appreciated, even under conditions of immense stress. From the perspective of a typical consumer using the banking system, you can probably blithely ignore that this is happening. Nightmares for systemic stability might be utterly non-events for you personally.

To the extent one wants to take low-cost actions one is unlikely to regret, I would suggest one has at least one backup financial institution. If one hypothetically does not, I would observe that opening bank accounts rounds to free. Thousands of perfectly good financial institutions exist. If one were to put money into a backup account, perhaps enough money to get through a weekend or to get through a payroll cycle, one would have access to money even if one’s primary financial institution was unexpectedly unavailable for a short time due to serious issues. (Having credit available at diverse institutions is, of course, another option.) This has the added benefit of helping if the issue is, for example, total computer failure at the bank rather than financial catastrophe. It has been known to happen.

If one has more money in a financial institution than applicable insurance limits, and one does not have a professional advisor about that money, and one does not feel capable of confidently answering questions about their risk management, one should probably find a clueful advisor. I have no particular advice on sorting clueful advisors from many who passed the relevant exams, charge outrageously, and know even less about this subject than non-experts currently Googling while stressed.

My observations for businesses would be more complicated.

Many people believe that businesses should have a treasury department who considers liquidity and risk management to be literally the only thing they do. That sounds great in theory, but in the world we actually live in, you will actually hire a treasury department a few hundred employees after your bank account is above FDIC coverage limits. (Deposit insurance was designed for a world with sharply different employment patterns!)

And so, if you are a founder in the substantial chunk of the economy between those two goalposts, you should breathe a sigh of relief that the FDIC and other regulators are going into crisis management mode.

Many banks and technology firms have, and some will quickly rush to market, various automated treasury management solutions. These do some of the work of a treasury department, at a tiny fraction of the cost of expensive professionals.

It seems to be popular right now to shame businesses and suggest they need to manage the counterparty risk their bank represents. This is actually advocacy for the most sophisticated and largest financial firms in the world to have a new high-margin revenue stream renting this solution to the substantial fraction of the economy too large to benefit from deposit insurance and too small to hire a treasury department.

The basic offering here, which I will avoid endorsing any particular provider of, is “We will establish relationships with N financial institutions in parallel. We automate money movement between them on your behalf, such that you can treat your money as being in one logical pile. However, at legally relevant times, in legally relevant ways, you only have a maximum of $250,000 in each institution. This will allow you to effectively 5X or 10X or… well there are thousands of banks and we are tireless in finding partners… the deposit insurance limit. This will cost you money, just like all financial services cost you money, and it may or may not be 100% obvious exactly how much money it costs you.”

I will note that there is an interesting policy angle on whether we, as a society, would prefer for deposit insurance to be effectively unlimited if and only if one is smart enough to pay a software company (or financial services firm, but I repeat myself) to do this for you.

In addition to “treasury management”, sometimes firms phrase this offering as “cash sweep”, which I mention in case you’re wondering what words you need to say to a salesman to get the pitch. The offering largely does what is says on the tin. Despite the above policy response, I’d expect the salesmen of it to be booked beyond capacity signing up new customers this week, at every firm that has it in-market and at some which are rushing to fix their lack of it.

Any parting thoughts?

The banking system is well-regulated, resilient, and strong. Most institutions in the U.S. are comfortably OK at the moment. Some may well not be. Failures, and particular surprising failures, in heavily interconnected core infrastructure have a worrisome tendency to cascade.

This is not the end of the world, but the last five days (!) include a material and negative update on our understanding of the state of the world. It has surprised many people at many different institutional vantage points who would expect to not be surprised by this exact issue.

You’re probably going to end up hearing a lot more about this. If for some reason you don’t read Byrne Hobart or Matt Levine, fix that. For breaking news, your financial news outlet of choice will be all over this for the foreseeable future. I recommend moderating one’s degree of reliance on group chats or Twitter, less because they are likely to be less accurate than media coverage (very not obvious to me) and more because your degree of risk here is likely lower than justifies 24/7 monitoring of this situation unless you have reasons why that is obviously not the case.

A long and boring disclaimer relegated to a footnote

Market observers have a purity ritual where they exhaustively disclaim whether they have financial interests in stocks they are discussing. I think that’s irrational in my case but rituals are useful things, so here’s a longer disclosure statement than you probably want:

I don’t and won’t short bank stocks, mostly because it’s impossible to do when keeping my nose provably clean given my position in the information graph. I do invest in individual bank stocks, but not materially (they’re a sixth of the economy and maybe 1% of the part of my portfolio I can conveniently price?), and for an idiosyncratic reason.

My life is weird by the standards of retail bank consumers—”business owner with American citizenship plus Japanese residence” puts me in a reference class of only a few hundred people banked in either nation. Banks will routinely steamroll reference classes of a few hundred people, by accident. You can buy a bank’s attention to bespoke needs by bringing it deposits, but it takes a lot more money than I have. Or you can buy a trivial number of shares of the bank and call Investor Relations if you have any problems.

This is one of many fun hacks I picked up over the years as an unpaid advocate for people with routine banking issues. Customer Service might fob off a retiree who wants a NSF fee reversed. Investor Relations, on the other hand, is socialized to guess that anyone calling it is more likely to be a pension fund manager and less likely to be a pension fund beneficiary. And so they can use very free calendars, no managed-to-the-minute-CS-drone quota, and substantial organizational heft to escalate things to any department on your behalf, with the implicit endorsement that Capitalism Called And It Requires You Resolve This Immediately.

Another weird thing about me: some people collect baseball cards. I collect bank accounts. I never set out to do this but by the time I realized I had a collection I had some borderline rational reasons for not reversing the decision, like “well I have to understand professionally how banking apps work and it is useful to have a survey of them installed” and “if I ever lose a U.S. bank account as a non-resident opening a new one is a pain in the keister so maybe I should have, oh, five backups... per account type... including for my LLC... in each country."

And so, when you combine these two facts, I am directly exposed to a lot of bank stocks, but in relatively tiny amounts. This includes banks under substantial stress. I have not sold and have not changed my banking as a result of risk of failure.

Why do I believe this is an irrational disclosure, despite general support for this ritual? Because I live in a society, which is sufficient information for you to know that I’m structurally levered long to the stability of the banking system, much like you are.

Adblock test (Why?)

My text editor of choice when I'm in Linux-land is Vim. One of its most powerful features is the ability to use regular expressions when searching. This capability makes hunting through giant log files so easy. However, I ran into a problem today that I wasn't sure how to work around.

I was looking through a large log file, trying to find the lines that included the text Internal Server Error. Making this difficult, however, were hundreds of such entries that were calling a known failing case. I wanted to weed out these known cases, and with regular expressions, I was able to. The known case I wanted to ignore looked something like the following:

[02/Feb/2023 16:06:14] ERROR [log.py:224] Internal Server Error: /api/v1/some_bad_endpoint/

This endpoint is the only one in the /api root, so I wanted to ignore /api as a part of my search. The magic is through negative look-ahead assertions, the syntax for which I was unfamiliar (\@! being the key). My search command in Vim ended up being:

/Internal Server Error: \/\(api\)\@!

Using this regular expression helped me jump to every instance of the Internal Server Error text that wasn't a call to the known-failing /api root. So handy!

In September 2022, after watching many YouTube videos of other people on long-distance Amtrak trips, I finally embarked on a journey of my own: I took the Amtrak Southwest Chief train from Chicago to Los Angeles. Continue reading to learn more about it and why I’ll do it again on another route.

“Why would you want to do this?”

This is a question I got a lot. When I told friends that I booked a ticket to be on a train for over 40 hours, many didn’t understand why anyone would want to do this when you can fly for cheaper and in so much less time, enjoying comforts like in-flight entertainment, drinks, snacks, and the airport lounges.

The idea behind such a trip was to experience the country differently. I moved to the United States from Germany only 7 years ago and have flown all over it so many times, but never seen much of it up close. I can give you directions to a good bar or restaurant in dozens of cities, but I have never seen Arizona. I can navigate 20 airports blindly but never saw a sunrise in Kansas. Taking a train slows you down and gets you 34,000 feet closer to life on earth.

Treating the Trip Like a Cruise

I had no reason to be in Chicago or Los Angeles. In fact, I flew to Chicago only to take the train, and from Los Angeles, I flew straight back home. Yes, it will take so much longer to get to your destination, but in this case, the journey is the goal.

Not only can you experience much more of the country, but you are also almost forced to do nothing. On many other trips, I pack my schedule full of things I want to do. It is easy to be stressed about the feeling of missing out on something and being so busy that you don’t get to relax very much. On a train trip? Not so much. You can do nothing except eat, drink, listen to music, and watch the scenery pass by. Internet access is unreliable, and you can’t do much with your phone. It’s you and all the other people on the train with no schedule except the stops on the journey. More on this later, but I found this trip to be extraordinarily relaxing and recharging.

On top of all that, you will meet people. Everyone has nothing to do, and if you want to talk, you’ll find someone else who also wants to talk. I’ve had fantastic conversations that stuck with me, and I made a new friend.

Most importantly, I am constantly searching for silence. An Amtrak Superliner pulled by 4,000 hp diesel locomotives, rumbling over New Mexico standard gauge railway tracks, is not silent. But you cannot achieve real silence ever. If you are not meditating, there will be your inner monologue; even if every sound is drowned out, you’ll start hearing your own heartbeat.

For me, silence is the absence of noise. Noise is having to do something else than just being. Noise is the chores of the day. Phone notifications are noise. On a train, in the middle of nowhere, with no mobile reception and a cold beer in my hand, and nothing to do but watch the mountains pass by, I am experiencing silence.

Picking a Train

For many, Amtrak is an operator of commuter trains. Readers on the eastern seaboard will likely have experience with Amtrak as an easy way for short to medium-distance trips between the significant population centers. Those commuter trains are similar to most trains I’m used to from when I grew up in Europe: Somewhat comfortable but designed to move people around for only a few hours. You might get a snack, but meals are not a priority or available at all. You can enjoy the scenery from your window like on any other train, and there are no sleeper cars.

The long-distance fleet of Amtrak is designed to move you across the entire country. While many people travel long distances on a simple chair in Economy class, you can also book different types of rooms in sleeper cars. An observation car with huge windows and outward-facing seats is available for sleeper car passengers, and meals are made to order in an onboard kitchen.

Every long-distance route is served by trains with specific names. For example, the train taking you from Chicago to San Francisco or the other direction is called the “California Zephyr.” I was interested in three routes:

All those trains had an enticing distance, each en route for at least 40 hours.

I decided to take the Southwest Chief for my first-ever Amtrak long-distance trip. The reason for me was simple: The California Zephyr route takes you through the Rocky Mountains, and I wanted to take it in the winter. Also, I wanted to take a slightly less famous or beautiful route to get some experience first. The Empire Builder route was another option, but I felt like that particular route would be more beautiful in the winter, too.

The Chicago to Los Angeles route is served by two trains: The Southwest Chief and the Texas Eagle. The Southwest Chief takes you south of the Rocky Mountains through Kansas, Colorado, New Mexico, and Arizona. If you take the Texas Eagle, you will go south to Texas and west to California – A significantly longer route and the longest scheduled travel on Amtrak.

I went with the Southwest Chief because it seemed to be a good combination of long but not-too-long travel time on a beautiful but not the most famous route. I was setting myself up to take another trip on the California Zephyr with more experience at another time.

Planning the Trip

Now that I knew which train I wanted to take, I could start planning the trip and buying tickets.

Amtrak makes it very easy to buy train tickets using their website, and it’s very similar to buying a plane ticket. I could not find a pattern of pricing differences, so I simply picked a ticket for a week that worked well with my work schedule: Departure from Chicago on Labor Day. Prices did not seem to be very different.

Flying from my hometown of Houston to Chicago is easy; both cities are United hubs. I booked a one-way ticket to Chicago and a one-way ticket back home from Los Angeles.

You can get a bedroom or roomette on an Amtrak sleeper car. A roomette fits two people, with the second person having to sleep on a bunk bed contraption under the room’s ceiling. I was going alone on this trip, and a roomette seemed to be the best choice.

The ticket cost me around $1,200. I could have bought an economy ticket for $150, but sitting for so long was out of the question. If I treat this as a cruise, I want a good level of comfort. Also, economy does not include meals in the dining car and gets you no access to the observation car. With two flights and the hotel room for a night, I ended up spending a little over $2,000. Not cheap. At least the food and some drinks on the train will be included in the price.

Long-distance Amtrak trains are known to often accumulate enormous delays. Amtrak does not own pretty much any rail lines (the physical tracks on the ground) outside of the Northeast USA, and freight trains have priority. You can easily be 10 hours late at your destination. If you treat the trip like a cruise, like I did, you might treat this as a positive thing. It will, however, complicate your flight schedule. I decided to play it somewhat safe and book my flight back for 12 hours after the scheduled train arrival in Los Angeles. Worst case, I could have always rebooked.

During booking, I could not pick a room or car. I ended up in a room in the center section of the upper floor and heard that those are better because you are further away from the vibrations and sounds of the wheels and track. On this route, I didn’t care much about which side of the train I was on because the scenery is very similar, no matter which way you look. This is more important if you take a train along the coastline, for example, the Coast Starlight from Los Angeles to Seattle. I have no idea if you could call Amtrak and get another room if you ask nicely. Still, I believe there is a chance depending on availability.

The departure timing is the other side of the trip you want to play safe. A delayed flight should not jeopardize making it in time for the train; this is supposed to be relaxing, and I hate travel stress. That’s why I decided to fly into Chicago the day before the train departure and spend a night there.

Packing

It was clear to me that I wanted to travel with a single backpack. Hauling a second bag or carry-on through the airports and train stations did not seem like a great plan. Also, the somewhat uncertain exact time of return from Los Angeles called for maximum flexibility. On top of that, it was only a three-day trip, and packing a single bag was absolutely possible.

One complication was that I also wanted to pack some camera gear, but my tiny Ricoh GRIIIx, Sony a6400, and few lenses don’t take up much space.

I chose my favorite backpack for short, single-bag trips: the Peak Design 45L Travel Backpack. It’s large, but carry-on approved for the flights. The small camera packing cube was enough for the photo and video gear, leaving me with enough space to comfortably pack clothes and other utensils.

Everything else was just like packing for a regular short trip. Toothbrush, deodorant, cables, chargers, Kindle, iPad, and so on. The Amtrak sleeper car rooms have USB charging ports and reliable power outlets. I recommend packing 6-10 foot charging cables because the outlets are only on one side of the room, and it might not be the closest to you, depending in which direction you’d like to sleep.

Your room will have a dial to control the temperature, but the hallway of the sleeper car was extremely cold on my trip. I recommend you bring a hoodie or something for when you want to leave the door open.

You won’t need anything special for “life” on the train. You will have breakfast, lunch, and dinner included and can always get small meals, snacks, drinks, and even alcohol in the little store on the lower level of the observation car. You can pay with a credit card and even Apple Pay there. I recommend bringing cash to tip the dining car staff and your sleeper car attendant. More on that later.

In Chicago

I had been at the Chicago O’Hare airport many times before but never in Chicago itself. I was happy to learn that Chicago has a decent public transit system. You can quickly get from O’Hare to downtown using the blue line train. The trip takes about 45 minutes and is not very scenic, but effective.

My train left in the early afternoon on Monday, so I had a lot of time to explore the city on Sunday night. My friend Kyle recommended the Centennial Bar on La Salle St for dinner and beers. It turned out he quickly texted the owner, letting him know that I was coming. It’s a short walk from downtown, and the food, beer, and conversations were fantastic. 10/10. Thanks, Kyle!

I checked out from my hotel the following day and walked to the Chicago Union Station. Amtrak owns and operates it; it serves as a long-distance train terminal and handles several commuter lines. It opened in 1925, replacing an older station built in 1881. Today, it handles about 140,000 passengers on an average day and is the busiest Amtrak station outside of the Northeast corridor.

At the Train Station

I had never taken an Amtrak before and planned two hours of a buffer. I used that time to familiarize myself with the station and where to find the trains. The trains leave from a separate building, but access is easy, and many signs point you in the right direction.

I immediately noticed a lot of Amish people sitting in the large hall. This is because travel by plane is considered too modern and travel by train appears to be very popular amongst them. Something to read more about.

As an experienced and frequent flyer, I headed to the Amtrak lounge after feeling familiar enough with the station. The lounge was quiet, clean, and comparable to a US domestic airline lounge, with snacks, a little bar with wine (out of plastic cups), beer, and plenty of space to sit. You can access the lounge if you have a certain member level or a same-day first-class or sleeper car ticket.

I spent about an hour in the lounge, waiting for my train to board. The staff at the lounge will print out a physical paper boarding pass if you prefer that over the mobile app. (I believe you don’t have to have a printed boarding pass.)

Boarding

I can only describe the boarding process I experienced from within the lounge. I was ready to head to the tracks a few minutes before boarding time, just like you would at an airport. While sitting at the lounge and enjoying a very average red wine, I heard announcements for passengers of other long-distance trains (I believe the California Zephyr and the Capitol Limited) asking to meet at the front desk for boarding. It turns out that all passengers with lounge access can board their trains directly from the lounge: An Amtrak attendant will guide the group through the train station and straight to the train. This pleasant surprise made the whole thing feel like a class trip.

If you are not in the lounge, you can just walk to the platform when it’s time to board.

A conductor checks your ticket at the train, directs you to your car, and tells you if your room is upstairs or downstairs.

Settling in on the Train

I boarded the train and headed upstairs through a small staircase to find my roomette. The room had about the size I expected and was very clean and fresh.

After spending a few minutes exploring the room, I was greeted by the car attendant Tony. She quickly ensured everyone felt at home and explained how things worked on the train and in the room.

A roomette has a bunk bed layout with two seats connecting into a bed and another bed under the ceiling. I was traveling alone and did not need the top bed, which also had no window view.

Tony explained that I could reserve a time for breakfast, lunch, and dinner in the restaurant car. They do this because the car can’t hold all eligible passengers at once. She took my reservation for dinner and the next day’s breakfast.

I spent the few hours until dinner in my room, watching parts of Illinois and Iowa pass by.

The Southwest Chief has 31 stops on the 2,265-mile-long journey. You can get out for some fresh air at every stop, but there will be an announcement to let you know how long you can expect to have. Some stops are just a few minutes long, and some are long enough to explore the area around the train station if you are brave enough to risk getting lost or losing track of time. In case of delays, you could be at a station for hours. The train staff set expectations very clearly and communicated frequently.

The First Dinner

I had a 7pm reservation and walked over to the car right in front of mine, the restaurant car. Because space is limited, you will sit with other passengers at tables holding 4 people. This is something I personally enjoy, but you should consider this if you are a little more introverted.

I sat with a British couple that night and had a great conversation. Those random collisions in the middle of nowhere are a great part of this slow form of travel.

An Amtrak Superliner has a full kitchen on the lower deck of the restaurant car. This is why the food is surprisingly delicious. I heard the steak was good and picked it for my dinner. It was cooked to order and tasted great. I had not expected such good food on the train.

You get one alcoholic drink with each meal but can always order more-Which I did.

Tips are appreciated, and, just as with any other service job, if you tip well, you might end up with a lot of free wine for you and a friend you made on the way the next day.

After returning from dinner, I discovered that Tony had used the time to convert my room from the “day configuration” to the “night configuration.” This means she set up the bed and brought fresh bed linen, pillows, and fresh towels for the shower.

The shower was hot, with good water pressure and plenty of space. I was ready for the first night.

During the Day

I slept well, but please take that with a grain of salt because I can sleep in pretty much any environment. If you can sleep on a international first-class plane seat, you will sleep comfortably in an Amtrak room. The upstairs rooms on a Superliner are further away from the tracks, and my room was in the middle of the car, at the most distance away from the wheels. All this reduced noise and vibrations, and I was very much unbothered.

I fell asleep and woke up still in Kansas. It turns out that state is really large if you cross it diagonally.

After brushing my teeth, I walked over for breakfast. I sat with a guy from the East Coast, and we had a great conversation. The eggs were fresh, and the fruit was delicious. My new friend and I had a great discussion, so we decided to check out the observation car together.

The observation car is located right behind the restaurant car. It can also only be accessed by first-class and sleeping car passengers. (UPDATE 2/3/2023: Readers on Hackernews pointed out that this might not be the case and that everyone has access to the observation car. I might be wrong here. UPDATE 2: Yes, I was definitely wrong. Amtrak says on their website that everyone can access the observation car.) Not only is there a bar with snacks and drinks downstairs (it takes Apple Pay!), but also it has seats facing outside and huge windows. At some stops, workers will even clean the windows from the outside.

I spent almost the entire day in the observation car, with only a break for lunch, where I met a gentleman who had boarded earlier and was going to California.

The many stops along the route allow for some fresh air and stretching your legs.

You will have an entire stretch of daylight to see the most interesting sceneries on the westbound Southwest Chief route.

It is an amazing experience to sit down with a drink and watch the ever changing landscapes of Colorado, New Mexico and Arizona pass by in front of you.

For dinner, we closed the place down. The friendly staff had a bottle of wine to empty, and we were there, talking until almost midnight. This was one of my favorite experiences of the entire trip.

Arriving in Los Angeles

After another good night of sleep, I woke up in California and headed for breakfast around 7am to find a crowded restaurant car. I sat with a large group of travelers connecting to the Coastal Starlight Amtrak train to Seattle. Many boarded in San Bernadino to take the short trip to Los Angeles.

I packed my things and waited for our arrival in Los Angeles. After arriving, I spent some time in the Amtrak Lounge and took a FlyAway shuttle bus from the station to LAX airport. There was no delay, and we were actually a few minutes early, so there was no rush for me to get to the airport. We arrived before 8am, and my flight was not until 6pm. I had chosen a late flight on purpose and was ready to rebook in case of a significant delay of the train.

What’s Next?

This trip was great. I was able to completely shut down and find some silence without all the noise and distractions. I cannot think of a single uncomfortable moment along the journey.

I plan to take the California Zephyr to San Francisco or the Empire Builder to Seattle next. A friend even had the idea of going from coast to coast and starting in New York City.

I am encouraging you to explore the Amtrak network yourself and give it a try. It is worth it.

Adblock test (Why?)

Have you been worried that ChatGPT, the AI language generator, could be used maliciously—to cheat on schoolwork or broadcast disinformation? You’re in luck, sort of: OpenAI, the company that made ChatGPT, has introduced a new tool that tries to determine the likelihood that a chunk of text you provide was AI-generated.

I say “sort of” because the new software faces the same limitations as ChatGPT itself: It might spread disinformation about the potential for disinformation. As OpenAI explains, the tool will likely yield a lot of false positives and negatives, sometimes with great confidence. In one example, given the first lines of the Book of Genesis, the software concluded that it was likely to be AI-generated. God, the first AI.

On the one hand, OpenAI appears to be adopting a classic mode of technological solutionism: creating a problem, and then selling the solution to the problem it created. But on the other hand, it might not even matter if either ChatGPT or its antidote actually “works,” whatever that means (in addition to its limited accuracy, the program is effective only on English text and needs at least 1,000 characters to work with). The machine-learning technology and others like it are creating a new burden for everyone. Now, in addition to everything else we have to do, we also have to make time for the labor of distinguishing between human and AI, and the bureaucracy that will be built around it.

---

If you are a student, parent, educator, or individual with internet access, you may have caught wind of the absolute panic that has erupted around ChatGPT. There are fears—It’s the end of education as we know it! It passed a Wharton MBA exam!—and retorts to those fears: We must defend against rampant cheating. If your class can be gamed by an AI, then it was badly designed in the first place!

An assumption underlies all these harangues, that education needs to “respond” to ChatGPT, to make room for and address it. At the start of this semester at Washington University in St. Louis, where I teach, our provost sent all faculty an email encouraging us to be aware of the technology and consider how to react to it. Like many institutions, ours also hosted a roundtable to discuss ChatGPT. In a matter of months, generative AI has sent secondary and postsecondary institutions scrambling to find a response—any response—to its threats or opportunities.

[Read: ChatGPT is dumber thank you think]

That work heaps atop an already overflowing pile of duties. Budgets cut, schoolteachers often crowdsource funds and materials for their classrooms. The coronavirus pandemic changed assumptions about attendance and engagement, making everyone renegotiate, sometimes weekly, where and when class will take place. Managing student anxiety and troubleshooting broken classroom technology is now a part of most teachers’ everyday work. That’s not to mention all the emails, and the training modules, and the self-service accounting tasks. And now comes ChatGPT, and ChatGPT’s flawed remedy.

The situation extends well beyond education. Almost a decade ago, I diagnosed a condition I named hyperemployment. Thanks to computer technology, most professionals now work a lot more than they once did. In part, that’s because email and groupware and laptops and smartphones have made taking work home much easier—you can work around the clock if nobody stops you. But also, technology has allowed, and even required, workers to take on tasks that might otherwise have been carried out by specialists as their full-time job. Software from SAP, Oracle, and Workday force workers to do their own procurement and accounting. Data dashboards and services make office workers part-time business analysts. On social media, many people are now de facto marketers and PR agents for their division and themselves.

No matter what ChatGPT and other AI tools ultimately do, they will impose new regimes of labor and management atop the labor required to carry out the supposedly labor-saving effort. ChatGPT’s AI detector introduces yet another thing to do and to deal with.

Is a student trying to cheat with AI? Better run the work through the AI-cheater check. Even educators who don’t want to use such a thing will be ensnared in its use: subject to debates about the ethics of sharing student work with OpenAI to train the model; forced to adopt procedures to address the matter as institutional practice, and to reconfigure lesson plans to address the “new normal”; obligated to read emails about those procedures to consider implementing them.

At other jobs, different but similar situations will arise. Maybe you outsourced some work to a contractor. Now you need to make sure it wasn’t AI-generated, in order to prevent fiscal waste, legal exposure, or online embarrassment. As cases like this appear, prepare for an all-hands meeting, and a series of email follow-ups, and maybe eventually a compulsory webinar and an assessment of your compliance with the new learning-management system, and on and on.

---

New technologies meant to free people from the burden of work have added new types of work to do instead. Home appliances such as the washing machine freed women to work outside the home, which in turn reduced time to do housework (which still fell largely to women) even as the standards for home perfection rose. Photocopiers and printers reduce the burden of the typist but create the need to self-prepare, collate, and distribute the reports in addition to writing them. The automated grocery checkout assigns the job of cashier to the shopper. Email makes it possible to communicate rapidly and directly with collaborators, but then your whole day is spent processing emails, which renews the burden again the next day. Zoom makes it possible to meet anywhere, but in doing so begets even more meetings.

ChatGPT has held the world’s attention, a harbinger of—well, something, but maybe something big, and weird, and new. That response has inspired delight, anxiety, fear, and dread, but no matter the emotion, it has focused on the potential uses of the technology, whether for good or ill.

The ChatGPT detector offers the first whiff of another, equally important consequence of the AI future: its inevitable bureaucratization. Microsoft, which has invested billions of dollars in OpenAI, has declared its hope to integrate the technology into Office. That could help automate work, but it’s just as likely to create new demands for Office-suite integration, just as previous add-ons such as SharePoint and Teams did. Soon, maybe, human resources will require the completion of AI-differentiation reports before approving job postings. Procurement may adopt a new Workday plug-in to ensure vendor-work-product approvals are following AI best practices, a requirement you will now have to perform in addition to filling out your expense reports—not to mention your actual job. Your Salesforce dashboard may offer your organization the option to add a required AI-probability assessment before a lead is qualified. Your kids’ school may send a “helpful” guide to policing your children’s work at home for authenticity, because “if AI deception is a problem, all of us have to be part of the solution.”

Maybe AI will help you work. But more likely, you’ll be working for AI.