These short tips are useful to remember when writing high-throughput code. You may already know most of them, and if so then please spread the word — friends don’t let friends write performance bottlenecks.
In a high-throughput hot loop:
- Avoid holding locks or other resources, unless you know it won’t block another performance-sensitive thread. Definitely don’t acquire any new locks or other resources! (While you’re at it, avoid closing resources too… sure, that .Close() may claim to be nonblocking, but can you really ever be sure?)
- Keep all such blocking operations outside performance-critical sections, so in those sections you can run without yielding. That includes watching out for the OS scheduler: Start that section as soon as possible after an OS context switch so that it’s as close as possible to a fresh OS thread quantum, and make sure it can finish before the OS gets tired of waiting and does the next context switch.
- Don’t do I/O in hot loops. No! Don’t! Not even intermittently, which can cause gaps in your throughput. Crikey, I sometimes still see people doing console logging inside a critical loop. In 2020.
- I/O is still okay, just defer it: Inside the hot loop, buffer up any such side effects into something like a non-allocating lock-free ring buffer in memory, or similar guaranteed-nonblocking guaranteed-throughput service. Then, after the hot loop ends, send all the buffered work out — and then you can also do all your other deferred locking, resource closing, and other blocking work, as much as you want. It’s all good when you can afford to wait.
Those are just a few tips, but they’re so popular that you may remember learning them as a rhyming song as a child. For high throughput, if you’re going to play the game, boy, you’ve got to learn to play it right:
You got to know when to hold ’em
Know when to close ’em
Know when to block and wait
Know when to run
You never cout your message
When your throughput should be stable
There’ll be time enough for cout-ing
When the hot loop’s done
…
But seriously: Thank you, Kenny. We miss you.
Sutter’s Mill 