Most folks aren’t aware that when a married guy turns 50, AARP sends a welcome package that includes dress socks to wear with sneakers, a copy of Ken Burns’ Baseball on VHS, 1.5 power reading glasses, an aluminum can crusher, and a large-print book on smoking meat.
I’ve been steadily working through my box for the last couple of years and am solidly into the BBQ part of the program. Things are getting serious this Father’s Day as I have been (early) gifted a Bluetooth-equipped Masterbuilt Pro electric smoker. It is so awesome.
Anyways, as seems to occur with each new endeavor, I’ve discovered that there’s at least one interesting-to-me parallel between this one and large-scale software development: The Stall. Let’s discuss.
Smoking 101
The basics here are pretty simple: prep the meat with some sort of brine or rub, let it hang around for awhile, then leave it to cook in the smoker at low temperature (225-250 F), with some variety of smoldering wood chips, for a long time. A “long time” depends on the cut and size of meat plus the weather, and can range from 60 minutes (salmon filets) to 3-4 hours (sirloin roasts) to 10-12 hours (beef brisket).
There is no end of mostly conflicting advice on the details — types and lengths of prep, water in the smoker or not, fat cap on the top of bottom, when to add smoke, blah blah blah. Most of this is just playing at the margin because almost anything you cook this way is going to end up super yummy.
But there is one real challenge: the stall. Time can be an ok estimate, but the only real way to know that your smoke is done is by internal temperature. This is of course where Bluetooth comes in; I can wander around the house watching the temperature on my phone, or more likely not watching because I get a push notification when it’s ready. We live in a world of miracles.
At the start, internal temperature rises steadily. But inevitably things hit a point — the stall — where the temp flattens out and just doesn’t budge, sometimes for hours. Knowing this will happen doesn’t make it any less unnerving, because it’s the difference between dinner at 6pm as planned or at 11pm in your jammies.
If the Internet is to be trusted, the stall occurs when the cooling effect from surface evaporation equalizes with the heat in the smoker. When (or if) it happens depends primarily on moisture levels in the air, the meat and the smoker’s water pan. Eventually enough surface moisture is gone for things to start heating up again and you’re on your way.
As dinner time approaches, there are a few ways to break through this plateau. Easiest is to just crank up the temperature — but this can seriously compromise the final product. Slightly more acceptable is the “Texas crutch,” which involves wrapping the meat in tin foil or parchment. But far and away, the best and most consistent advice is to just stay the course and wait it out. Starting early (and maybe letting things rest a little longer) makes for a far, far better end result than forcing it.
So now you know why the old guy (or gal) looks increasingly panicky into the late afternoon and keeps obsessively checking the thermometer. They are doing their best to get through the stall without a mental breakdown. Beer helps.
The Stall in Software
Pretty much every mid- and large-scale software project I’ve been involved with also hits the stall … even the best ones. I swear this isn’t just some weirdly-forced made-up comparison; I just didn’t have a name for it until now.
In the early stages of a project, you hit new, visible milestones almost every day. The first on-paper design, the first successful build, the first UI screen, the first end-to-end transaction, the first automated test case, the first demo. The nonstop adrenalin rush is the best, and the high can last a long time, especially if you’ve done a good job on system design, as you see early choices pay off in the real world.
But eventually, usually just before or after going beta, things bog down. All the edgiest edge cases start popping up, and the fixes just don’t feel “right.” Too many special cases make you question your abstractions — did we miss something fundamental? Is [insert new technology name here] going to work or do we need to rip it out? Triage meetings start to feel like whack-a-mole as every fix seems to uncover a new bug and the active count just never seems to drop.
Of course, there are “Texas crutches” in this world too. The most common is to reduce system scope and just accept a less capable product. You can take a huge schedule slip, or add (human) resources, or even go nuclear and change up the core design. There’s a time and place for all of these approaches, but nine times out of ten the right move is to just stay the course and wait it out. The real world is complicated and our abstractions can only approximate it … there are always special cases and they just need to be handled.
I am terrible during this phase, constantly fretting and asking what-if and generally being a jerk about every proposed fix. But that’s why you build up a network over the years. My friend Dave manages the stall better than anyone I’ve ever known. He was my first office mate, way back on Microsoft Works for DOS, and we’ve built many systems together since then. He’s relentless and unflappable. Pick up each bug, debug the crap out of it, write the fix, test the fix. Next. Next. Next. Next. Next. And every time, my neuroses aside, we get to the end and everything is a-ok. Thanks Dave.
Back here on Whidbey, there is a lovely bottom round roast sitting in the fridge waiting to go into the smoker Sunday morning. If you see me pacing the deck about 4pm you’ll know why … but I’m working hard at learning the lessons that life keeps putting in front of my face too. Patience pays off, and I still have like 100 hours left on that Baseball documentary anyways. Sláinte!