You’re reading Age of Invention, my newsletter on the causes of the British Industrial Revolution and the history of innovation. This edition went out to over 22,000 subscribers. To support my work, you can upgrade your subscription here:

Most people binge on television shows. I binge on reading eighteenth-century travel diaries — especially when they tell me fascinating new things about the inventors of the time.

One that especially caught my eye this week was a kind of pilgrimage to Leicestershire in 1785 by two young Frenchmen, Alexandre and Francois de La Rochefoucauld — the sons of a duke — to visit a man famous for his cattle.1 They went to Dishley, just a couple of miles from Loughborough, to the 500-acre farm rented by Robert Bakewell.

Bakewell’s story could have been an unremarkable one. He was born, farmed, and died at Dishley, much like his father before him. But Bakewell, unlike most people, caught the improving mentality, or attitude — the one thing all inventors, both then and now, have in common — which had him viewing everything around him in terms of its capacity for betterment. The improving mentality was a reframing the status quo as a problem to solve. A habit of optimisation. A compulsion to perfect.

It is unclear where exactly Bakewell picked up the improving mentality. His father’s and then his own landlord, Sir William Gordon, appears to have been an agricultural improver. He conducted a tour of the West Country at some point in his youth, presumably coming into contact with more improvers — though I suspect he was already motivated to travel by having already caught the improving bug. When starting his own experiments with cattle, he drew his original stocks from some of the better-known improvers of cows and sheep.

Regardless of how Bakewell picked up the improving mentality, however, he definitely had it. As the young French aristocrats put it, after spending just a little time with him, “he showed us how well, beneath a heavy and rough exterior, he had been making observations, and studying how to bring into being his fine breed of animals with as much care as one would put into the study of mathematics or any of the sciences.”

Bakewell’s core objective, in improving cattle, was to make them as profitable as possible. His core insight was that you could divide a cow with two horizontal lines, with butchers paying the most for the topmost meat on cows’ backs, fit for roasting — “gentlemen’s meat” of the sirloin and fillet. The middle section was less valuable. And the lowest, generally boiled, was “only fit for the army”. Over the course of decades he thus selectively bred cows with the largest possible backs — where the most valuable meat was — and the leanest possible lower parts, their bellies forming a sort of triangle.

This strategy was not just about maximising the price of the cows when sold for meat, but about making them as efficient as possible when it came to feeding them. The less feed spent on developing muscle in the lower, less valuable parts, the better. Bakewell ended up doing similar for his sheep, “so plump that we have measured several of them and found them broader than they are tall … having on each side a great lump of fat which the farmer called cloven flank”. In sheep he selected for smaller bones — it was all about their fat and meat. And he improved horses — used for pulling ploughs, driving mills and hauling carts — developing a breed that was short, had low joints, and a broad chest, “strong enough to pull a house along”.

As one of his French admirers put it, “little by little, by dint of trials and the coupling of what he discovered to be the best animals, he came in the end to create the most superb breed of animals I have ever seen.”2 This was not just by trial and error: Bakewell separated a his farm into many hedge-enclosed fields of just 6-10 acres, in which he conducted controlled trials, for example buying a particular county’s typical sheep and turning it out onto the same kinds of fields with the same feed and upbringing, to see how his own breeds differed. This had the added advantage of being highly effective advertising for his own breeds: “he demonstrates the comparison to everyone who visits his farm, and it is undeniable that the others are thin and lean compared with his”.

Bakewell’s cows and sheep became extraordinarily valuable when sold for meat, though he soon discovered he could make even more money by leasing out the young males of his breeds to other farmers so that they could improve their own — “but never as good as that of Mr Bakewell who has both the male and the female”. Recognising how essential it was that he not lose his competitive advantage, he even set up his own abattoir and sold only dead meat, for fear that an unscrupulous butcher might be tempted to breed from the live animals sent to slaughter.

One of his French visitors judged that “he has the glory of being the only man in England, perhaps in the world, who has reached such great perfection in cattle, sheep and horses. I think he has reached the point he was aiming at and no longer seeks to improve them, only to lease them out.”

But perfecting the breed was still not enough. The improving mentality was a compulsion — there was and is always a way to optimise, to tweak and finetune. Bakewell thus perfected the way his cattle were raised. The agricultural writer Arthur Young was astonished at the cows’ extraordinary docility, so that “by a little swish”, even a young boy could gently direct them wherever they wanted, being “accustomed to this method from being calves.”3 That docility meant they were less likely to harm one another and get into accidents too.

Bakewell even conducted a cost-benefit analysis of whether to let the cattle roam free or have them tethered nearer to their stalls. Letting them roam free had the advantage that they would manure the land, meaning that he would not have to pay people to cart the dung out onto the field, and to harvest hay from the fields and bring it to the stalls. But he worked out that letting them roam free would also require spending more on people to watch them, that they would only manure the fields unevenly anyway, and the cattle often tended to trample what they were supposed to eat. Having them tethered meant that “nothing is lost, all is eaten, and the manure is better rotted.”

Bakewell even configured the layout of his farm so that “all is in tremendous order, everything arranged to avoid wasting time in carting from one place to another”. If it could be optimised, it was. Despite the fact that he rented his farm rather than owning it, Bakewell even had a network of wooden aqueducts and about ten miles of canal dug across his lands in order to irrigate them. He even told his French visitors in 1785 that he planned to make the canals navigable by boat, so that he could float heavy loads of grain or turnips from one side of the farm to the other.

The irrigation was essential for yet another task, which was to make every acre of the farm count in terms of fodder for his cattle. He showed his French visitors that he could get almost seven times as much hay from an acre of carefully manured and watered meadow than from one that was not. Naturally, he experimented continually on getting it exactly right. To the improver, there was always something else to be improved.

With such attention to detail, it’s no wonder that the English became such prodigious meat eaters in general. Our same French visitors in 1785 commented throughout their journey at the extraordinary diets enjoyed by everyone, even working people down mines or in factories. Indeed, they noted a peculiarly “English drowsiness that immediately follows a meal”, characterised by a roundly extended belly, though it came also came at a cost: “much breaking of wind, which the English generally take no trouble to control.”

If you’d like to support my work, please consider upgrading your free subscription to a paid one here:

Adblock test (Why?)

08 Nov 2023 - Tags: sage

So I was on math stackexchange the other day, and I saw a cute post looking for a book which lists, for many many integers, facts that Ramanujan could have told Hardy if he’d taken a cab other than 1729. A few days ago OP answered their own question, saying that the book in question was Those Fascinating Numbers by Jean-Marie De Koninck. I decided to take a glance through it to see what kinds of facts lie inside (and also to see just how many integers are covered!). Not only was I overwhelmed by the number of integers and the number of facts about them, the preface already includes one of the single wildest facts I’ve ever heard, and I have to talk about it here! Here’s a direct quote from the preface:

37, the median value for the second prime factor of an integer; thus the probability that the second prime factor of an integer chosen at random is smaller than 37 is approximately $\frac{1}{2}$;

My jaw was on the floor when I read this, haha. First it sounded totally unbelievable, since 37 is a tiny number in the grand scheme of things. Then it started to sound slightly more plausible… After all, about half of all integers have $2$ as their smallest prime factor. It makes sense that smaller primes should be more frequent among the smallest factors of numbers! But then I thought “how can you possibly prove this!?”. I’m not much of an analytic number theorist, but I know that they have good estimates on a lot of facts like this. I decided it would be fun to try and find and understand a proof of this fact, and also write some sage code to test it!

So then let’s go ahead and do it ^_^

First, I think, the sage code. I want to know if this really works!

“Obvoiusly” there’s no uniform distribution on the natural numbers, so what does it even mean to choose a “random” one? The way the number theorists usually solve this problem is by fixing a large number $N$ and looking at the probabilities when you pick a random number between $1$ and $N$. Then you look at the $N \to \infty$ limit of these probabilities.

So for us, we’ll want to first fix a large number $N$ and then work with numbers $\leq N$. For $N$ kind of small, we can just find the second prime factor of each number $\leq N$ and check the median!

When I first ran this code, it honestly felt like magic, haha. What the hell is going on here!?

The key idea, found in a paper of De Koninck and Tenenbaum, is that we can compute the density of numbers whose second prime is $p$ (which the authors denote $\lambda_2(p)$) by cleverly using the ideas in the Sieve of Eratosthenes!

Let’s do a simple example to start. What fraction of numbers have $5$ as their second prime? In the language of the paper, what is $\lambda_2(5)$?

Well it’s not hard to see that the numbers whose second prime is $5$ are those numbers whose prime factorization looks like

or

so we need to count the density of numbers of these forms.

But a number is of the first form ($2^a 3^0 5^b \cdots$) if and only if it has a factor of $2$, a factor of $5$, and no factors of $3$.

To bring this back to elementary school, we can highlight all of our numbers with a factor of $2$

numbers with no factors of $3$

and numbers with a factor of $5$

Then the numbers whose prime factorization starts $2^a 3^0 5^b \cdots$ are exactly the numbers highlighted by all three of these colors!

It’s intuitively clear that $\frac{1}{2}$ the numbers are blue, $\frac{2}{3}$ are orange, and $\frac{1}{5}$ are pink. So taken together, $\frac{1}{2} \cdot \frac{1}{5} \cdot \frac{2}{3} = \frac{1}{15}$ of numbers are of this form!

So now we have our hands on the density of numbers of the form $2^a 3^0 5^b$, but this is only one of two ways that $5$ can be the second smallest prime. A similar computation shows that $\left ( 1 - \frac{1}{2} \right ) \cdot \frac{1}{3} \cdot \frac{1}{5} = \frac{1}{30}$ of numbers are of the form $2^0 3^a 5^b$.

It’s easy to see that these sets are disjoint, so their densities add, and $\frac{1}{15} + \frac{1}{30} = \frac{1}{10}$ numbers have $5$ as their second smallest factor!

Now with the warm-up out of the way, let’s see how we can compute $\lambda_2(p)$ for our favorite prime $p$!

We’ll play exactly the same game. How can $p$ be the second smallest prime? Exactly if the prime factorization looks like

for some $q \lt p$.

But we can count these densities as before! For each choice of $q$, we know that $\frac{1}{p}$ numbers are multiples of $p$, $\frac{1}{q}$ are multiples of $q$, and for each $r$ we know $\left (1 - \frac{1}{r} \right )$ numbers are not multiples of $r$! For each $q$, then, we want to land in the intersection of all of these sets, then we want to sum over our choices of $q$. Taken together, we see that

But remember the goal of all this! We want to know the prime \(p^*\) so that half of all numbers have their second prime \(\leq p^*\). That is, so that the sum of densities

But we can implement $\lambda_2(-)$ and just check for which prime this happens!

Again we see that $37$ is the prime where roughly half of all numbers have something $\leq 37$ as their first prime! So we’ve proven that $37$ is the median second prime!

Also, this shows that we expect the actual density to be $\approx .5002$. If we set $N = 10^7$ in the code from the first half to get a better approximation, we get $.5002501$, which is remarkably close to the truth!

Thanks for hanging out, all! This was a really fun post to write up, and I’m really excited to share it with everybody! This fact about $37$ was all I could think about for like a week, haha.

I have more blog posts coming, of course, so I’ll see you all soon!

Stay safe, and stay warm ^_^

Adblock test (Why?)

Published on 11/09/2023, 1396 words, 6 minutes to read

Just in time for me to start a new job at a new place, my RSI has decided to flare up.

Cadey>

For the record, I'm fine, I've known this has been coming for a while. The flare-up is on its exit anyways. I've gotten lucky and I'm going to be fine. But it's still a bit of a bummer.

The last time this happened, I was able to get by by doing mostly writing about technology things, but I think I'm going to need to be able to program again. I know I'm a bit of an emacs user, but for this I've been using visual studio code because of one extension in particular: Cursorless.

Cursorless is a plugin that integrates with voice control software to let you do AST level code editing with your voice. This is crazy alien magic from the future.

I've talked about cursorless before on my blog, but I have decided to really get deep into it this time around. The last time I used it, I didn't actually use it for much more than moving around the screen, but this time I'm going to try to use it for everything.

I wish I had this as an input method for slack and discord messages.

The most magic parts about this are the ideas of destinations and targets when it comes to cursorless inputs. Targets are individual anchors in a document and destinations are places relative to individual targets. Every single token in a document is given a hat over a letter with a color. These hats act as anchors that let you give commands based off of locations, destinations, and paths between them. Here's a simple example. Consider this code:

function fetchBlog() {
  fetch("https://xeiaso.net/blog.json")
    .then((response) => {
      if (!response.ok) {
        throw new Error("Network response was not ok");
      }
      return response.json();
    })
    .then((data) => console.log(data))
    .catch((error) => console.error("Error:", error));
}

This is a fairly standard looking JavaScript function. But, cursorless puts a bunch of hats over all of the code so it may look something like this:

Aoi>

So that's why your editor is full of random video artifacts?

Cadey>

They're not artifacts, they're targets!

Take a good look at that picture again:

The hats are color coded above individual letters. The position tells you the name and the color tells you how to disambiguate it. For example, That word function would be referred to as green urge because the hat is green over the letter u. If I wanted to delete that word for some reason or if I wanted to move it somewhere else, I could use green urge as the target for that action.

By itself, this gives you some pretty powerful actions and effectively lets you do spoken vim motions. But, that is only thinking in terms of simple actions that you can do with your editor. The real power of cursorless comes in from not only the idea of paths (such as green urge past green bat to select the function fetchBlog in that screenshot), but the fact that cursorless knows what the AST of the language is doing. This means that you can do things across the entire function, like deleting it or moving it somewhere else. As an example, here are the lambdas of this function visualized separately (with the "visualize lambdas" command):

These AST units are also targets. This means that I can do things like select the body of a definition and then work off of that. So if I want to refactor this into an asynchronous function, the refactoring becomes trivial:

Aoi>

That's pretty cool, but I don't think I'd be able to remember all of those commands.

Cadey>

After a while, they just become second nature like Vim commands do. I have been forcing myself to use this over and over again for the past few days and it's starting to become second nature. I'm introducing individual commands at one of the time and building up into bigger and better things.

The real magic comes when you start writing your own commands with the full power of Cursorless and Talon. In that example I just showed you, I have a action for inserting "async " before the function definition. Here is the code for that:

[state] async <user.cursorless_destination>:
    user.cursorless_insert(cursorless_destination, "async")

You can break talon commands into two basic parts: patterns and captures. Patterns are the spoken words that you say and captures are the things that you want to extract out of what you say. In this case, the pattern is just the word async and the capture is the destination that you want to insert the word async before. The <user.cursorless_destination> capture is a special capture lets you specify if you want something before or after a target. Of course, this is just a very simple example and it can get way more intricate than this.

Here is the most complicated Talon rule I've written so far:

(method|meth) <user.letter> [<user.go_pointer>] [<user.go_visibility>] <user.text> [over] [<user.go_visibility>] named <user.text> [over]:
    user.go_method(go_pointer or "", letter, go_visibility_1 or "public", text_1, go_visibility_2 or "public", text_2)

Aoi>

What the heck is going on there?

This looks like a lot, but it is actually really simple. This lets you declare a method in Go. In Go, a method looks like this:

func (reciever *Type) MethodName() {
    
}

Pedantically, Go doesn't have methods in the traditional sense, it just has functions that take structs as the receiver (read: a hidden first argument but in a way that is namespaced to that struct in particular). Without something to automate writing this for you, you would have to say something like this:

state funk args word reciever space star hammer type over go right space hammer method name args go right brack enter

That's a lot of words to say. But, with this talon rule, you can just say:

meth r raised type named method name over

Aoi>

That's still a lot of words.

Cadey>

Well, yes, you're not going to be able to get over that. But this is at least more efficient and it makes more sense. I'm not going to be able to get rid of all of the words, but I can at least make it so that it's close to how I conceptualize it in my head.

Aoi>

I guess that makes sense, but what do you mean by raised? I don't think go has raised types I know it has pointers but not "raising". What is raising?

I'm glad you asked! This is something that I'm experimenting with to try to find a different way to explain the concept of pointers in Go. I think that one of the oversights in the Go language is that pointers use C-style syntax. This specifically has you use an * to lower a value from a pointer value to a normal value and & to raise the value from a normal value into a pointer value.

Since I'm taking the opportunity to radically redesign the Talon bindings for Go, I want to try unifying the syntax of pointer values into the idea of raising and lowering to see how it makes it easier to understand Go programs. I don't know if this is a good idea, but you have to fuck around in order to find out.

Maybe some parts of our industry are actually good. I really hope that I get led into the GitHub copilot voice beta soon, I want to compare how Talon does voice coding versus how copilot voice does it.

Facts and circumstances may have changed since publication. Please contact me before jumping to conclusions if something seems wrong or unclear.

Tags: cursorless, vscode

Adblock test (Why?)

Imagine you are in an interview and you get asked "can you implement bubble sort"...and you answer the interviewer "sure, do you want that in JavaScript or CSS".

What a flex would that be?

Now, I know what you are thinking. "no interviewer will be impressed by you creating an animation that simulates bubble sort". And you would be right!

But what if we could create a functioning bubble sort algorithm...in pure CSS...and add visualisations to it?

Well, you have come to the right place!

The Demo

Instructions: There are 5 values at the top of the :root element:

:root{
    --val1: 12;
    --val2: 16;
    --val3: 9;
    --val4: 14;
    --val5: 7;

These are our unsorted array!

So the above represents: [12,16,9,14,7].

And you can change those values (any value between 0 and 20) and then press "run" in the Codepen UI and it will actually sort them for you!

warning: on mobile the last few animations might not play and just go blank. On PC your fan may spin up!

This is a limitation of using so many calculations that rely on previous calculations...I am not sure if it runs out of memory or what, but I defo pushed the limits of CSS here!

Anyway, with the warnings out of the way, give it a go (you may need to press "rerun" in the bottom right as the animation only plays once)!

Explanation

Look, this is silly, so I am not going to do a tutorial, but there are a couple of things that are interesting:

Getting a boolean for v2 > v1

--is-1-greater-2-step-1: Min(1, Max(var(--arr1s0) - var(--arr2s0), 0));

Looks complicated, but it isn't, we are performing the following operations:

• subtract the value of position 2 from the value of position 1 in our array. (lets call this "diff1and2" for ease)
• find the maximum of "diff1and2" and 0. We do this as a way of saying "if 1 is bigger than 2, we want a positive value, but if 2 is bigger than 1 we want to return 0". Lets call the result of this "1greaterOrZero".
• then we take "OneGreaterOrZero " and make sure it is not greater than 1 by using Min. So if "OneGreaterOrZero" was 6, we would reduce that to 1, but if it was 0 then we would return 0.

Still confused? Here it is in JS effectively:

let pos1 = 7;
let pos2 = 15;

let diff1and2 = pos1 - pos2; 
//if "diff1and2" is negative the next step will change it to 0;
let OneGreaterOrZero = Math.max(diff1and2, 0);

let result = Math.min(1, OneGreaterOrZero);

console.log(diff1and2, OneGreaterOrZero, result); //always between 0 and 1 as false / true representation.

swapping array positions

So how do we swap positions in our "array"?

For bubble sort to work we need to be able to swap 2 values if the first value is larger than the second.

We can't do any temp variable magic like we would in JS.

Well, that is what this is for:

--arr1s1: calc(var(--is-2-greater-1-step-1) * var(--arr1s0) + var(--is-1-greater-2-step-1) * var(--arr2s0));
--arr2s1: calc(var(--is-1-greater-2-step-1) * var(--arr1s0) + var(--is-2-greater-1-step-1) * var(--arr2s0)); 

Yet again, looks complicated, is actually reasonably simple in principle.

In our previous "function", we created a boolean to see if 1 was greater than 2. So we have either a 1 or a 0 there.

We can also easily get the inverse of this:

--is-2-greater-1-step-1: (1 - var(--is-1-greater-2-step-1));

The beauty of this is that we can now do the following trick:

// in bubble sort, if 1 
origArray = [7,2];

// we run our previous functions to get our 2 variables:

oneIsGreater = 1;
twoIsGreater = 0;

// we can now multiply the values together. If [2] is greater than one then we will return the same value. But if [1] is greater than [2] then we will swap the values.

newArray[0] = (twoIsGreater * origArray[0]) + (oneIsGreater * origArray[1]);  
newArray[1] = (oneIsGreater * origArray[0]) + (twoIsGreater * origArray[1]); 

// which is the same as this:
newArray[0] = 0 * 7 + 1 * 2;  //2 
newArray[1] = 1 * 7 + 0 * 2;  //7

Neat trick huh? and if oneIsGreater and twoIsGreater were swapped then it would just return the original values!

origArray = [7,2];
oneIsGreater = 0;
twoIsGreater = 1;

//same "function"
newArray[0] = (twoIsGreater * origArray[0]) + (oneIsGreater * origArray[1]);  
newArray[1] = (oneIsGreater * origArray[0]) + (twoIsGreater * origArray[1]); 

// which is the same as this:
newArray[0] = 1 * 7 + 0 * 2;  //7 
newArray[1] = 0 * 7 + 1 * 2;  //2

And that is effectively all we need to do a bubble sort!

The only reason we have so much CSS is because we cannot do loops in vanilla CSS (yet). So we have to manually write out the swaps for each stage of bubble sort:

• check and if needed swap 1 and 2
• check and if needed swap 2 and 3
• check and if needed swap 3 and 4
• check and if needed swap 4 and 5
• check and if needed swap 1 and 2
• check and if needed swap 2 and 3
• check and if needed swap 3 and 4
• check and if needed swap 1 and 2
• check and if needed swap 2 and 3
• check and if needed swap 1 and 2

That's all folks!

As I said, not a tutorial. But I did want to introduce some interesting CSS "switches" and "booleans" that may become useful in some strange scenario in the future for you!

I hope you enjoy my silliness.

Now I just have to wait for someone to ask me if I can implement Bubble sort so I can flex on them with my CSS silliness! 🤣💗

Adblock test (Why?)

In the months before The Boy and The Heron’s Japanese debut, Studio Ghibli made a point of keeping details about the latest animated feature from Hayao Miyazaki to a minimum. For those who’ve been trying to avoid spoilers, it’s definitely gotten trickier what with all the buzz coming from its run on the film festival circuit. But as much as you might expect the trailer for GKids’ English dub to give away, it’s a surprisingly restrained and tantalizing look at just what has everyone so worked up.

It’s difficult to parse where and when The Boy and the Heron’s young protagonist Mahito (Luca Padovan) is in the movie’s new trailer, but it is clear that all of the odd and inexplicable occurrences around him have something to do with a strange...

Continue reading…

Oct 31^st, 2023 @ justine's web page

Cosmopolitan Honeybadger

After nearly one year of development, I'm pleased to announce our version 3.0.1 release of the Cosmopolitan library. The project is an entirely new animal. For starters, Mozilla sponsored our work as part of their MIECO program. Google also awarded me an open source peer bonus for my work on Cosmopolitan, which is a rare honor, and it's nice to see our project listed up there among the greats, e.g. curl, linux, etc. In terms of this release, we're living up to the great expectations you've all held for this project in a number of ways. The first is we invented a new linker that lets you build fat binaries which can run on these platforms:

• AMD64
  • Linux
  • MacOS
  • Windows
  • FreeBSD
  • OpenBSD
  • NetBSD
• ARM64
  • Linux
  • MacOS
  • Windows (non-native)

It's called apelink.c and it's a fine piece of poetry that weaves together the Portable Executable, ELF, Mach-O, and PKZIP file formats into shell scripts that run on most PCs and servers without needing to be installed. This is an idea whose time has come; POSIX even changed their rules about binary in shell scripts specifically to let us do it. So we've been using it to create a "Fat Linux Distro" which I've named the "Cosmos". In the Cosmos, every program is statically linked and contains a PKZIP central directory where its /usr/share dependencies are embedded. You can think of it as a coalition of individualistic executables, where each program can be separated from the whole and run on other OSes. So far it includes programs like Emacs, Vim, CoreUtils, Curl, Git, etc.

cosmos-3.0.1.zip
213mb - PE+ELF+MachO+ZIP+SH executables
For AMD64+ARM64 on Linux+Mac+Windows+FreeBSD+NetBSD+OpenBSD
ce256ededf106748a09f13bf47ace9ef0e6f115d963353d3d63c21302c5f28f4

More specifically, the above zip file contains fat binaries for ape, python, lua, qjs, vim, emacs, emacsclient, nano, llama, bash, dash, less, git, grep, curl, wget, tidy, zip, unzip, zstd, bzip2, sqlite3, life, nesemu1, make (GNU + SECCOMP + Landlock), gmake (GNU), redbean, greenbean, datasette, assimilate, rusage, ctags, wall, pledge, verynice, find, tree, basename, chgrp, cp, date, du, expr, groups, ls, mknod, nl, pathchk, pwd, rm, seq, shred, stat, tee, uname, users, who, basenc, chmod, csplit, dd, echo, factor, head, mktemp, nohup, pinky, rmdir, shuf, stty, test, true, unexpand, vdir, whoami, awk, chown, df, false, id, link, mv, nproc, pr, rsync, sleep, sum, truncate, uniq, yes, b2sum, chroot, dir, fmt, install, md5sum, numfmt, printenv, readlink, runcon, sort, sync, timeout, tsort, unlink, base32, cat, cksum, cut, dircolors, env, fold, join, ln, mkdir, od, printf, realpath, script, split, tac, touch, tty, wc, base64, chcon, comm, dirname, expand, kill, logname, mkfifo, nice, paste, ptx, sed, sha1sum, sha224sum, sha256sum, sha384sum, sha512sum, tail, tr, ttyinfo, and uptime.

This only became possible in the last few months, in part thanks to Gautham Venkatasubramanian, who spent a few weekends of his PhD studies modifying the C language so it's possible to build conventional software with Cosmopolitan. Since then it's been out of the frying pan and into the fire, testing our library to see if it can support some of the most complex and mature projects in the open source community. Running the ./configure scripts and make check rules of projects (e.g. GMP) has done so much to help us fix bugs and battle test new features.

One of the things we're most happy with, is that Cosmo's cross platform support is now good enough to support Cosmo development. We've traditionally only compiled code on x86 Linux. Devs using Cosmo would build their programs on Linux, and then copy the binaries to other OSes. Focusing on Linux-only helped us gain considerable velocity at the start of the project; the Cosmopolitan monorepo has two million lines of code. Today was the first day the whole thing compiled on Apple Silicon and Microsoft Windows systems, and using Cosmo-built tools.

Windows Improvements

In order to get programs like GNU Make and Emacs to work on Windows, we implemented new libraries for POSIX signals emulation. Cosmopolitan is now able to preempt i/o and deliver asynchronous signals on Windows, using a SetThreadContext() trick I learned from the Go developers. Cosmo does a considerably better job spawning processes now too. For example, we wrote a brand new posix_spawn() function that goes 10x faster than the posix_spawn() included with Cygwin. Cosmo's execve() can now reparent subprocesses, inherit non-stdio file descriptor, and our read() function now contains a termios driver which, for the first time, lets us poll() standard input on consoles. Cosmo binaries cleanly integrate with WIN32, depending pretty much only on KERNEL32, and your fat binaries won't have to live on a separate partition like WSL.

MacOS Improvements

While MacOS may not be the prodigal child of our support vector, this release brings improvements to MacOS users that are equally important. For starters, we now have first-class native ARM64 support. APE Loader also now dynamically links the officially blessed Apple libraries (e.g. libSystem.dylib) on ARM64, so there's less chance that Apple will break your binaries. We've also made semaphores and futexes much better on XNU, thanks to Grand Central Dispatch, and ulock on AMD64.

Portability and Performance (Pick Two)

The end result is that if you switch your Linux build process to use cosmocc instead of cc then the programs you build, e.g. Bash and Emacs, will just work on the command prompts of totally different platforms like Windows and MacOS, and when you run your programs there, it'll feel like you're on Linux. However portability isn't the only selling point. Cosmo Libc will make your software faster and use less memory too. For example, when I build Emacs using the cosmocc toolchain, Emacs thinks it's building for Linux. Then, when I run it on Windows:

It actually goes 2x faster than the native WIN32 port that the Emacs authors wrote on their own. Cosmo Emacs loads my dotfiles in 1.2 seconds whereas GNU Emacs on Windows loads them in 2.3 seconds. Many years ago when I started this project, I had this unproven belief that portability toil could be abstracted by having a better C library. Now I think this is all the proof we need that it's not only possible to make software instantly portable, but actually better too. For example, one of the things you may be wondering is, "these fat binary files are huge, wouldn't that waste memory?" The answer is no, because Cosmo only pages into memory the parts of the executable you need. Take for example one of Linux's greatest hits: the Debian Almquist shell.

$ ls -hal /usr/bin/dash
-rwxr-xr-x 1 root root 107K Nov 21  2022 /usr/bin/dash
$ ls -hal /opt/cosmos/bin/dash
-rwxr-xr-x 1 jart jart 983K Oct 15 19:14 /opt/cosmos/bin/dash

Here we see Cosmo's six OS + two architecture fat binary dash is 30% bigger than the one that comes with Alpine Linux (which only supports x86-Linux and dynamically links a separate 600kb Musl library). But if I run them:

$ rusage /usr/bin/dash -c true
took 231µs wall time
ballooned to 688kb in size
needed 183us cpu (0% kernel)
caused 34 page faults (100% memcpy)

$ rusage /opt/cosmos/bin/dash -c true
took 217µs wall time
ballooned to 544kb in size
needed 172us cpu (0% kernel)
caused 36 page faults (100% memcpy)

Here we see Cosmo's fat binary version of dash went faster and used less memory than an x86-Linux-only binary built for Musl Libc. This is due to (1) the magic of modern memory management, where CPU MMUs lazily load 4096 byte blocks at a time; and (2) how carefully apelink plans your executable layout. For example, all that code which is needed to support Windows (it takes a lot of code to support Windows) gets linked into its own special section of the binary, far away from what the MMU will want to page on UNIX systems. The same goes for the embedded ARM64 build. Since I'm running on AMD64 here, the ARM64 code is never loaded off disk.

You no longer need to choose between the amalgamation release or the cosmo monorepo. We now have a third preferred option, which is our new cosmocc command. It works pretty much the same as the cc command you already know. The Cosmopolitan README file has getting started instructions, which basically boil down to this:

sudo mkdir -p /opt
sudo chmod 1777 /opt
git clone https://github.com/jart/cosmopolitan /opt/cosmo
export PATH="/opt/cosmo/bin:/opt/cosmos/bin:$PATH"
echo 'PATH="/opt/cosmo/bin:/opt/cosmos/bin:$PATH"' >>~/.profile
ape-install        # optionally install a faster systemwide ape loader
cosmocc --update   # pull cosmo and rebuild toolchain

If your development environment isn't x86_64 Linux, then you'll need to download the latest toolchain, rather than using the one that's vendored inside the cosmo repository.

cd /opt/cosmo
rm -rf o/third_party/gcc
wget https://github.com/jart/cosmopolitan/releases/download/3.0.1/cosmocc-0.0.16.zip
unzip cosmocc-0.0.16.zip

If you're on Windows, then you need a shell before you can do any of the above. If you download the cosmos.zip file at the top of the page, it'll have a bin/ and libexec/ folder. Just extract those to your C:\ drive. Then install Terminal Preview from the Microsoft Store, and configure it so that it launches C:\bin\bash -l as your shell. You'll naturally need a good unzip program in order to do this: one that supports symbolic links (a.k.a. reparse points). The cosmos zip includes InfoZIP, but you can download it directly here: unzip.exe.

Now that your cosmo compiler is installed and added to your $PATH, you can build standard C and C++ programs as follows:

cosmocc -o foo foo.c
cosmoc++ -o bar bar.cc

If you want to build programs as fat binaries, there's a special compiler now which can make the process easier than using the x86_64-unknown-cosmo-cc, aarch64-unknown-cosmo-cc, fixupobj, and apelink commands directly. This compiler works by manipulating your compiler flags so that a concomitant .aarch64/foo.o gets created for every foo.o you compile. It runs the x86_64 and aarch64 compilers in parallel too, so it doesn't go much slower than cosmocc at the end of the day.

fatcosmocc -o foo foo.c
fatcosmoc++ -o bar bar.cc

Integrating with GNU Autotools projects is easy as well.

export CC=cosmocc
export CXX=cosmoc++
./configure --prefix=/opt/cosmos
make -j
make install

Refer to the README file for more detailed instructions. Also be sure to check out the ahgamut/superconfigure repository for the largest source of high-quality examples on how to build open source software using Cosmo.

This cosmos release includes the latest version of the redbean web server. This is a fat single-file forking Lua+SQLite+MbedTLS stack written by the Cosmopolitan authors originally to showcase the capabilities of the library for greenfield development. Cosmopolitan is good for more than just building old GNU code and redbean proves that. It's the third most upvoted hobby project in Hacker News history. A few weeks ago Berwyn Hoyt independently determined it to be the fastest Lua web server too!

One of the reasons why redbean is a forking web server is because we didn't develop our own POSIX Threads implementation until last year. So we put a lot of thought into writing an example of how you can build a bare minimal threaded web server that's even faster than redbean, and it's called greenbean. There's a prebuilt fat binary for it in the cosmos.zip distribution above. Greenbean is 400 lines of liberally commented perfection. I still can't believe how good we got its memory usage, thanks to tricks like MAP_GROWSDOWN. On Linux, if I ask greenbean to spawn over 9,000 persistent worker threads:

$ sudo prlimit --pid=$$ --nofile=18000
$ sudo prlimit --pid=$$ --nproc=18000
$ rusage greenbean 9001
listening on http://127.0.0.1:8080
listening on http://10.10.10.237:8080
greenbean workers=0 connections=0 messages=0 ^C shutting down...
took 7,959,324µs wall time
ballooned to 40,352kb in size
needed 929,917us cpu (92% kernel)
caused 10,039 page faults (100% memcpy)
54,689 context switches (99% consensual)

Then it somehow only uses 40mb of peak resident memory, and according to htop, greenbean's virtual memory usage is 76,652kb. That's for 9,001 threads. Like redbean, greenbean is able to handle hundreds of thousands of requests per second on my Intel Core i9-9900, except (1) greenbean has better shared memory support, (2) it sets up and tears down connections faster, and (3) it lets you experience the joy of using Mike Burrows' *NSYNC library, which is the basis of Cosmopolitan's POSIX synchronization primitives. If you're not familiar with the man, he's the guy who coded Chubby and Altavista, a global search engine which was so efficient it only needed to operate on a single server. But you wouldn't think *NSYNC is as prolific as it is if you're only going off star count.

My favorite thing about greenbean is how elegantly it reacts to CTRL-C. When you interrupt greenbean, it'll use the POSIX pthread_cancel() API to immediately terminate all the worker threads, so that shutdown happens without delay. Cancelation is one of the trickier concepts for a C library to get right. It's up there with threads, signals, and fork in terms of how its ramifications pervade everything. So similar to Musl Libc, we've put a lot of thought into ensuring that Cosmo does it correctly. Cosmo avoids cancelation race conditions the same way as Musl and Cosmo also implements Musl's PTHREAD_CANCEL_MASKED extension to the POSIX standard, which has seriously got to be one of Rich Felker's most brilliant ideas. Read the commentary in greenbean.c if you want to learn more.

Some of you might view greenbean as just a toy example. In that case, if you want to see something based on greenbean that is actually running in production, then pay a visit to https://ipv4.games/ whose source code is in net/turfwar/turfwar.c and net/turfwar/.init.lua. This is a hybrid redbean + greenbean service, where redbean does the HTTPS frontend, and greenbean does the HTTP backend. Hackers love to unleash their botnets on the IPv4 Games, and it honestly isn't that hard to withstand a DDOS with 49,131,669 IPs when your web server can do a million requests per second. That's for a service where 99% of the requests are write requests. If you want to have fun with this server, then you're also welcome to check out our monitoring metrics while you do it.

This cosmos.zip release includes several games you can play which have been vetted on all our supported platforms. For example you can actually play Nintendo in the terminal and it'll even work inside the Windows 10 command prompt. Getting poll(stdin) to work in Windows is a messier problem than even naming or cache invalidation, so it's super sweet that Cosmopolitan Third Edition is now able to abstract that complexity. Although in the case of our port of Bisqwit's fabulous NESEMU1 program, it turned out we didn't need poll() at all! Just calling read() and letting it be EINTR'd by setitimer() sixty times a second to pipe audio did the trick. So playing these games helped battle test our new signals implementation too.

We may not have GUI support yet, but you can use your mouse in the terminal. On Windows, your mouse cursor will generate the same ANSI XTERM style control codes as it does on Linux, MacOS, BSD, etc. Try running the life program that's included in Cosmos. Left click draws cells. Space runs an iteration of the life game. Right click can drag the display. You can also ctrl+wheel to zoom the display in and out. To read the source code to this simple program, check out tool/viz/life.c.

Boulder startup dylibso just announced a few weeks ago that they've adopted Cosmopolitan for their new product Hermit: Actually Portable Wasm. Hermit lets you create secure cross-platform executables for WebAssembly modules. It's worth a try! Cosmopolitan has a long history of serving the needs of the Wasm community. Our first major adopter back in early 2021 was actually the wasm3 project which provides a similarly great solution to running Wasm outside the browser. I'm happy to see these projects benefiting from the advantages Cosmopolitan has to offer. Will you be the next adopter? If so, feel free to reach out to me personally and I'll see what I can do to help you be successful: jtunney@gmail.com. You're also invited to join our Discord community.

Funding for Cosmopolitan Third Edition was crowdsourced from Justine Tunney's GitHub sponsors and Patreon subscribers, the backing of Mozilla's MIECO program, and the generous contributions of our developer community on Discord. Your support is what makes projects like Cosmopolitan possible. Thank you!

Adblock test (Why?)