Don't use musl if you care about performance

I've spent most of my career working inside the JVM, with a sidebar of Python. So the first time I hit incompatible libc with containerized Bifrost, and GPT suggested musl as the solution, not only did I jump on it, I moved my other Rust projects to musl as well. Self contained binaries with a single implementation backing them? Yes, please!

Then my colleague Ryan mentioned that musl is known to have a suboptimal allocator, and pointed me to Daniel Raneland's article. Uh-oh. I decided to measure it to see how bad it was.

So, yes: musl's allocator is indeed bad. Terrible, even. And (contrary to some of the articles I've read) not only in high concurrency scenarios; these numbers are from 4 core EC2 VMs, and Bifrost sizes its thread pools accordingly.

This is a really bad footgun to leave loaded for users and honestly, I think musl should ship without an allocator and make you choose one. Then if you choose the really bad one for some reason (maybe the code footprint is really tiny? idk) then that's on you, instead of "oh sorry did you not read the fine print? lol wasn't that a fun surprise!"

But unfortunately "just use mimalloc" [or jemalloc] is not a magic wand that gets musl performance parity with glibc; musl with mimalloc is still 26% slower. So I dug deeper into the two task types that showed the worst regressions.

scan_usages does get some benefit from mimalloc, but remains slower than on glibc. structural_clone_smells on the other hand barely allocates; the difference on musl with and without mimalloc is noise. Yet s_c_s suffers even more, proportionally, from musl than does scan_usages.

It turns out that the allocator isn't the only suboptimal code in musl, and several common memory routines are particularly slow:

Simplicity still isn't free

I'm going to keep smaller projects where 25% slower doesn't matter (like Hel) musl-only for simplicity—with the addition of mimalloc. But in the spirit of not leaving more loaded footguns around, we're removing musl as an prebuilt option from the much-more-performance-sensitive Bifrost.