Defenders of the Codebase

Good CTOs spend a lot of time reviewing designs and code. You simply can’t “stay above the details” if you really want to know what’s going on, i.e., do your job. Somewhat counterintuitively, I have found that my most useful contributions during these sessions are almost always subtractive, not additive:

  • Why do you need that?
  • Why did you pick up that dependency?
  • Did you consider this existing code we already use?
  • Do you really need a new process / machine / database / language / environment?

Engineers love new shiny things. Once upon a time the way that manifested was “not invented here” — building everything anew, even when great alternatives were available. These days (and of course I’m speaking in generalities), that’s flipped 180 degrees — not a week seems to pass without somebody insisting that we have to pick up the new whosywhat that somebody just dropped onto Github.

As an aside, I’m honestly shocked that this doesn’t give more people security-related anaphylaxis. With one line in a build file you have invited who the hell knows onto your development team. The code they wrote almost certainly will run at high privilege in your internal systems. People get super-worked-up about China slipping backdoors into hardware, but seriously how hard would it be for a bad actor to build a really useful open source library ready to attack from the inside? If it hasn’t already happened, it’s going to.

Setting my paranoid fever dreams aside, there is a larger point here about adding anything new to your codebase: more often than not it’s the wrong choice for a ton of reasons. Whether you build it yourself or pick it up with a package manager, the added complexity always comes with downsides that aren’t immediately apparent to an engineer excited about what they did with File / New:

  • New bugs to find.
  • New JARs or DLLs that may conflict with existing residents.
  • New abstractions for team members to learn.
  • New configuration models that have to be integrated into build and deploy.
  • New logging models that have to be integrated into monitors and alerts.
  • New surface areas for security review and patch management.
  • I could make this list really long but I think I’ve made my point.

OK, so it’s important for a CTO to ask “why” and say “no” a lot — check. But the astute reader may detect one tiny, minor flaw this approach — if you never create anything, you never create anything. And it’s kind of your job to create things. Hmm. I’m sure there are many ways to resolve this dilemma; I’ve accidentally developed this framework over the course of my career:

  1. When you need a new capability, start with these questions:
    1. Do you already have something in the environment that can do the job, even if it needs a little work (more on “a little work” to come)? If yes, done.
    1. Is this so far outside of your competency / mission that it’s worth taking on a third party dependency? If yes, do your research and done.
    1. If you get here, you’re going to build something. Read on.
  2. Build the first version for your immediate need. Resist the temptation to make the world’s most awesome reusable widget. It is always easier and safer to build something for a specific purpose than for a general one. You may find some obvious places to wire for reuse, but be super-careful here, because experience says the “next” use case may well never materialize anyways.
  3. Re-evaluate your approach with each successive use case. The second (and third) time you need a capability, you have a lot more information.
    1. What is the gap between what you have and what you need? Is it a natural extension or a big refactor? Has the existing code been a big source of bugs already? Is there some competitive advantage in your version?
    1. Depending on the answers to these questions, make a judgment — and this is just a judgment — whether to stay the course or to pick up something new and more general.

One note: if/when you choose to flip to a new approach, you’re almost certainly to hear somebody say “we should have done this from the beginning.” F*ck that guy. Defending your codebase is a long-term process of optimization and evolution; lots of folks just never figure that out.

The struggle is real

It turns out that I’m also an engineer that likes shiny new things — so even as I play with my own projects I have to keep this stuff in mind. A few months ago I needed a very simple web server as part of a collaborative radio project. I really wanted the code to stand on its own, so I just used the built-in Java HttpServer and was good to go. The SMART project required a few extra features, so I added a bit of abstraction and ended up with WebServer and SecureServer. These did a solid job of serving content that was either static or fully programmatic, perfect for the task at hand.

With my current weather station project, yet again I need a web server. The existing version was OK, but the HTML this time is more complex and needs some server-side manipulation. It honestly just wants a classic web framework like Play or Ruby, but I’m feeling stubborn and don’t want to get sucked into that noise. The “build” option here is a simple templating engine that has at least the following features:

  • Token replacement
  • Loops (including nested loops)
  • HTML escaping

How hard could it be? Well, 375 lines of code later you can be the judge. This is frankly my favorite kind of code to write … a neatly contained package with some interesting algorithms, interface design, and a few picky little loops to get right. Templates are just text files with embedded directives enclosed in double-curly-quotes. Directives can perform simple token replacement or slightly fancier stuff like looping or calling out to an external process. You can also run custom code if the built-in capabilities don’t do the trick. It works quite well.

But should I have done it? Long-term, will I be happy with this code, or wish I’d used an existing framework? It’s gotten pretty complicated and has a bunch of weaknesses — no directive escaping, no compute within the template itself (e.g., for dynamic looping logic), no hot reload, etc. etc.. It also probably still has bugs in it.

On the other hand, I’ve already reused the capability unaltered for email and SMS message generation. And it turns out to be handy for simple CGI scripts that transform process output into HTML tables. And my world today is different from a real CTO’s … these projects are more avocation than vocation after all, so “it’s fun” can legitimately carry some weight.

Who knows? I’d love to hear what you think. At the end of the day, it’s a judgment call and only the passage of time will reveal its wisdom, or lack thereof. We’ll see!

Until next time, when I hope to have that weather station ready to show off….. maybe.