Tag: startups

Turtles all the way down

~ Sean Nolan
image credit Wikipedia

Every business is a process shop — a tangled mess of human and automated activities that work together to produce something folks are willing to pay for. And even as AI starts to handle specific jobs and tasks, that inherent complexity doesn’t go away.

Enterprise software is the glue that keeps the machine running, and if you’ve ever been on the hook for it working correctly, you’ve implemented some kind of monitoring solution. Log processors, web pingers, process monitors, “on call” scheduling — there’s an entire industry of software that just watches other software (and people, and AI) to make sure all is well.

And yet, we still get surprised by catastrophic failure — the backup that we thought was happening every night; the SSL certificate we swore was on auto-renewal; the battery-operated door lock that failed-open over the weekend; the ETL job configured to run under that retired guy’s account.

So what do we do? Monitor the monitors, of course. But what if they fail? One of my favorite old saws is turtles all the way down — it seems like there’s no bottom to this stack! That’s why, everywhere I’ve ever been, I’ve built something like backstop. Even in my retired life, it’s an essential tool.

Backstop

The idea behind backstop is to have one authoritative, affirmative check on your world, generally once per day (I run mine about 4am). Backstop is the heartbeat of your enterprise, showing up on schedule with a single, consolidated, explicit, proactive look at everything that matters.

One person needs to expect the backstop email every morning. If it doesn’t appear, silence is not golden: find out why. If it shows errors or warnings, find out why. If anything looks funny, track it down. (Honestly, this person should be your CTO or CIO — nothing else gives better intuition for “how it’s going,” and that awareness is gold.)

This doesn’t obviate the need for any of your other monitoring and alerts — they are more timely and more detailed. Backstop is an assurance that the machine is working and that nothing is falling through the cracks. A good backstop has four critical properties that need careful attention:

1. Bulletproof

The most important feature of a good backstop is that it finishes and reports. Every exception needs to be caught; every hung request needs to timeout. Each metric you’re measuring needs to be checked independently — failure of one check cannot stop evaluation of another (for example, here and here).

This is easy to get wrong, especially because you’re likely to be relying on a bunch of third party libraries to monitor proprietary services and apps. That’s why the human element is absolutely critical. If the backstop email doesn’t show up on schedule, a real person needs to notice and they need to fix it.  

2. Complete

Asking a human to check in on dozens (hundreds) of independent subsystems is untenable; a backstop fixes this by creating one single tip to the spear. For that to work, it needs to be a complete look at your environment.

Your best friend in this endeavor will be something like ProcessResource — a type of checker that can run an arbitrary sub-process. While I’m the first to advocate for limited dependency, reality is complex and so is your environment. You surely rely on some system that only has a node or python client library, and another that has its own native client, etc. etc.. In the backstop use case, completeness is more important than consistency, so hold your nose and script away.

It’s also important to evolve “completeness” over time. Of course adding support for new systems, but also catching up old ones. Unless you suck at your job, most outages reveal new failure modes — adding new checks to your backstop should be a routine part of your post-mortem process.

3. Clean

Nothing spikes my blood pressure quite like somebody saying “oh that happens all the time, we just ignore it.” It’s not just lazy, it’s corrosive — not only will your “real” alerts get lost in the noise, but the “ignorable” ones are almost always worse under the covers than you think.

You have to be able to see what’s wrong. If you’ve accidentally coded a bad metric, change or remove it. If something is time-bound — e.g., you’ve already set a plan to fix it on a specific date in the future — implement a pause that wakes up if that date is missed. But under no circumstances can you allow errors and warnings to persist over time. Please trust me on this.

4. Actionable

Last — every resource you track should include a link that gets you to the right place to investigate and learn more. By definition a backstop problem is an exception, which means a disruption to your carefully-curated calendar of stupid meetings. It’s imperative that you can dive in quickly and figure out what’s up.

This link can be a lot of things — a more detailed look at the resource itself; a pre-filled form to open a trouble ticket; a diagnosis cookbook on a wiki; whatever works. But especially if you have a junior or specialized engineer looking at the backstop error list, knowing where to start can make a huge difference.

Important Metrics

Age / Activity

This is probably the most important backstop metric, because it’s the one that is most often missed by traditional monitors. Some process (or a monitor!) just stops working, but we don’t notice until it’s too late, because silence seems golden.

These failures also tend to create the worst headaches, because they cause damage over time. Backups that don’t get done, key indicators missing critical inputs, that kind of thing.

Trigger Dates

A bunch of processes happen on what I call the “slow clock” — stuff you have to do every quarter, every year or even every few years. In my retired life these are things like renewing my driver’s license or cleaning the air filter in my furnace. In enterprises they’re more like audits, disaster recovery exercises, and domain renewals. Calendars help with these, but slow clock reminders can get lost amongst daily meetings and more immediate events.

Levels

Things rarely fall apart overnight — they slowly degrade, unnoticed, over time. Smoke alarm batteries are a great example, and the water level in our community storage tank.

When these alert at night or in the middle of the day, busy humans tend to ignore them (“I’ll get to that later”). But as a backstop metric, they become visible in the right context — at the right time, together with other outstanding issues.

Availability

This is the OG monitoring classic: is the web server responding? And if you’re fancy, can you perform basic tasks like login or search? These aren’t usually the most important backstops, but they can be useful checks, especially for lesser-used services that otherwise are ignored until the moment they become critical.

My Backstops, aka Code is All That Matters

I’ve written my own backstop harness because, well, I get to choose. I actually don’t know of a commercial or open source tools that really does this job, but there probably is one. Mine is written in Java; it’s free to use and modify on Github. If you’ve got a system with git, java and maven installed you can try it out like this:

git clone https://github.com/seanno/shutdownhook.git
cd shutdownhook/toolbox
mvn clean package install
cd ../backstop
mvn clean package
java -cp target/backstop-1.0-SNAPSHOT.jar \
    com.shutdownhook.backstop.App \
    config-demo.json PRINT

You’ll see a bit of log output but then most importantly a couple of lines like this:

OK,Google,,2138 ms response
OK,Proof of Life,,I ran, therefore I am.

The “demo” configuration file contains two resources: one that simply reports back “OK” and one that checks availability of https://google.com. The “PRINT” argument tells the app to just output to console rather than sending an email.

What’s Going On Here

The code is pretty simple, and purposefully so — its job is to be rock-solid and always, always, send an email at the end. Plus, we want to collect as much useful information as we can, so failures in one resource can’t impact the others.

Configuration starts with a list of “resources”, each defined by a name, url, java class name and map of class-specific parameters. A resource class must implement the Checker interface, doing whatever it needs to and returning results as zero or more Status objects, where zero means all is well. Checkers also have access to a convenience object offering common services like web requests and JSON management.

In the normal case, the entrypoint in Backstop.java just: (1) uses an Executor pool to tell the checkers to do their things; (2) collects and sorts the Status responses into a single list with the worst offenders at the top; and (3) Uses Azure to send an HTML email with the results.

Again, you’ll notice a ton of defensive code throughout — Backstop is a special snowflake.

Not counting my favorite existential DescartesResource, so far I’ve implemented five resource checker types for my personal use:

TriggerResource

This resource type reads “slow clock” events out of a Google Spreadsheet and alerts when deadlines are approaching or past. The best way to get a sense of this is to look at a few items from my household triggers:

My dog Copper needs his flea and tick pill once a month and we always used to forget. I’m secretary of our community HOA on Whidbey and that means some paperwork every year. My beloved electric boat has old-school batteries that need topping off once in awhile, and my license is going to expire next year.

The trigger resource code simply loads up rows from a spreadsheet like this and checks to see if each due date is past (ERROR) or upcoming within an optional WARNING period.

While many of these are recurring, the sheet isn’t smart about that. Once a row “fires”, the only ways to turn it off are to edit the spreadsheet (using the link from the backstop email) and change the “Due Date” to the next occurrence OR add a “Snooze Until” date.

Snooze is useful for things like my license — I set up my appointment for next month, so until then there’s no reason to pollute my backstop list. As simple as this is, I find it pretty transformational. Adulting is chock full of stupid things you’re supposed to remember — maybe you’ll get a reminder or maybe not. A backstop trigger list is the perfect security blanket.

Sending Email

I’ve chosen to use Azure Communication Services to send the backstop email. SMTP used to be so easy — but that was before spam and phishing and all the other nasties that took advantage of its simplicity. These days, reliably sending email that doesn’t land right in the Junk folder is a big hassle. Azure makes this pretty easy, and it’s dirt cheap — less than a dollar a year for once-a-day emails!

I don’t love the dependency, but it seems like the right balance.

Deployment and Logistics

The “last mile” for backstop is deciding where it should run and how it should be triggered. It is not a resource-intensive operation, so the old school option isn’t a bad one: dedicate a single small server or VM to the job, triggered with cron once a day. Sorted!

But this simplicity does come with a big downside — patching that server and keeping it up to date. In an enterprise you may already have good infrastructure for this, and if so go for it. But in my world, servers left on their own tend to decay over time.

I’ve tried to avoid this by using a couple of Azure services to do the job for me. The first I like a lot — the script docker-build.sh creates a container that runs in Azure Container Instances without a dedicated server. The container does its thing and then shuts down, so it’s also dirt cheap, just pennies a month.

That leaves just the cron part — something has to trigger the container to run every morning. I’m pretty surprised this isn’t just part of ACI, but it’s not. The solution I landed on is a timer-based Azure Function. My function uses a cron-style schedule to run each morning, scripting a start to the proper container.

This was a bear to get right. I’m not going to let myself spiral into yet another rant about how poor the Azure developer experience can be — just know it is rubbish. You know who really helped out here? My good friend Claude; way better than any Azure help resource I could find. Whew.

There’s Always Another Resource

I have a pretty long list of resources I’m planning to add to my backstop:

  • FLO whole-home water shutoff
  • Various GE appliances, in particular for rinse-aid in the dishwasher (finally we’re getting down to the real problems)
  • Tesla Powerwall and Enphase panels/inverters                   
  • More shutdownhook demo apps
  • The Rivian!
  • Electric, water and gas usage
  • … and on and on …

Our lives and our enterprises are pretty complicated — and every new piece of smart technology that seems (and is) so great carries its own tax. Servers, services, accounts, batteries, it adds up. To keep things humming you really do need a backstop. A single tip of the spear from which all of the mess can be corralled and observed. I hope you’ll give it a try — with my code or your own. Until next time!

I Want This Future

~ Sean Nolan

Stellar is a bonkers book, full of crazy word salad arrow-filled diagrams that make very little sense. At the same time, it makes a credible argument for an amazing future — an optimistic, beautiful, qualitative leap forward for humanity that I want to be part of. Even if you don’t buy all of their conclusions, Arbib and Seba set a challenge and opportunity worthy of our attention. Stellar should be required reading and part of the conversation at every high school around the world. Buy a copy for your young adults!

Because it’s a non-fiction book, by law (apparently) it’s too long. I’m going to try to boil it down; let’s see how I do. You’ll have to suspend disbelief a bit — I’ll offer some critical thoughts towards the end. Here’s their pitch:

1. Extraction & the Growth Imperative

The emergence of agriculture thousands of years ago changed us from a species that lived as part of nature (hunter-gatherers) into one that sees nature as a source of resources to be extracted and refined into the stuff we use to live.

This “extraction” economy has been an amazing, wonderful ride — no back-to-the-land nostalgia here. But it has also shaped virtually everything about modern humanity, individually and collectively. Extraction requires a constant, never-ending source of input materials (land for agriculture, fossil fuels, etc.), and those inputs are (at least regionally) scarce, finite, and often unpredictable (droughts, etc.).

The impetus, then, is to capture as much input material as we possibly can, plus extra as a buffer against disruption. This is the “growth imperative” — an extraction economy must grow forever to keep a steady supply of inputs. This is of course ultimately unsustainable, but it also impacts us dramatically in the near term. Nation states, war, religion, worker exploitation, monopolistic markets … they all can be traced back to the need to maintain a steady supply of extraction inputs. Even counter-concepts like the welfare state are part of the machine, creating just enough stability to keep folks “in the game” so they keep producing.

2. SWB, AI and the Zero-Input Economy

For only the second time in our history, humanity is right now on the cusp of a fundamental transformation. We are witnessing the convergence of a set of technologies that will completely obsolete the extraction economy, leading to a society so different from today’s that it’s tough for our extractive minds to even imagine.

Solar, wind, and batteries are the key technologies — the trinity that makes everything else possible. Unlike other energy sources, SWB (once built) uses zero “extractive” inputs. The hardware just sits there, year after year, generating power. It is very reasonable to believe that we can build enough SWB capacity to not just carry us through our darkest days, but to deliver near-infinite surplus during the good ones.

Of course there are inputs — but sunshine (and therefore wind too) is effectively infinite and non-extractive. No matter what we do, it just keeps showing up. And yes, of course SWB infrastructure doesn’t keep working forever; it requires maintenance and replacement. But performance over the last few decades has improved by orders of magnitude; we are approaching “close-enough-to-zero” to call it good.

Further, with the emergence of AI and next-generation robotics, it won’t be long before we don’t even need human labor inputs to keep this machine running. Proactive maintenance, repair and replacement will all be handled by AI, leaving humans to pursue fulfillment and learning far beyond what we currently consider “work.”

And once you have infinite power, all the extraction dominos start to fall. Clean water through desalination instead of aquifer depletion. Food manufactured from base molecules with precision fermentation, rather than covering the planet with farms. 3D-printed environments instead of concrete and lumber. Even climate change can be reversed.

3. Overcoming the Extraction Mindset

The potential changes are so profound and feel so magical that it’s really hard to stop my inner cynic from calling bullsh*t and just checking out. Today’s world is so messed up and just gets worse every day. The Stellar promise feels like an easy cop out — an imaginary white knight excusing us from the hard work of making things better right here, right now.

So don’t feel bad if your spidey sense is tingling; it may be right! But that’s also same reflex that has caused so many of us to miss the boat on other smaller but still dramatic transformations. And it’s also something the Stellar authors don’t shy away from — they have a lot to say about the “transition” and how it could easily go one of three ways: (a) successful transformation; (b) civil collapse; or (c) a hybrid “chimera” where Stellar technologies are hoarded and controlled by today’s capitalist class.

It’s important to note that Arbib and Seba are careful to keep from falling into today’s unhelpful zero-sum “capitalism vs. socialism” tropes. They believe that individual ownership will be unnecessary in a Stellar economy, but that’s because the dynamics of infinite energy will simply make ownership irrelevant. If you never need to worry about having “enough,” there’s no reason to carry around extra baggage.

These are the kinds of things that I’d like our next generation to be thinking about. Maybe there really is an opportunity to create a world without scarcity. And if so, what would humans born into that world look like? Is it actually possible that greed, fear, envy and the rest are all just the side-effects of living in an extractive world? We’ll never know unless / until we get there. But to me, it looks like the only credible offramp from our current run towards the cliff.

4. Living the Transition

As confident they are about the potential, Arbib and Seba freely admit they’re pretty clueless as to how we’ll actually make the transition. A lot of stuff we’ll need has yet to be invented. And many comfortable, familiar ways of looking at the world will have to be reimagined. A few that resonate with me:

  • We can’t retreat our way to success. Less extraction is still extraction; we’re never going to become sustainable by living smaller. Infinite energy is the means by which we will solve our problems.
  • Work is going to go away. Outsourcing jobs to AI and robots is inevitable. And that’s actually a good thing, freeing humans to explore and create, not just survive. But losses will hurt before they help — solutions like UBI will be critical to soften the transition.
  • We need a new economic vocabulary. Metrics like GDP only make sense in an extractive world. We need new numbers to measure our progress: watts per person, happiness indices, calories per watt, etc..

Perhaps most of all, we’re going to have to force ourselves, again and again, to question things that we “know to be true” — but are really just byproducts of extraction and the growth imperative. My generation, and probably the next, and definitely the one before — are going to suck at this. We’re just going to have to figure it out as we go, and hope that a Stellar “north star” will help us correct mistakes before they become too entrenched.

Or maybe that spidey sense is right and we’re just screwed. But the dead-end nature of extraction makes intuitive sense to me, and the numbers for infinite energy actually pencil out pretty good. And trying to build a Stellar world will be a heck of a lot more fun than living through the alternative. So let’s get busy.

110 Posts and Counting

~ Sean Nolan

I realized today that I’ve written 109 (well I guess 110 now) long-form posts on shutdownhook.com since February 2021, the month I quit my job at Adaptive Biotechnologies and updated my LinkedIn bio to “Retired?” (now it’s got an exclamation point). That’s about two a month; sometimes more, sometimes less. Not bad.

I do link my posts on social media, and I love it when one strikes a chord with folks. But mostly I write for myself — a deal I made with the lazy part of me that would be happy to sit on the porch with a Coke Zero and just watch the world slide by. “You can retire, as long as you keep building and learning and trying new stuff.” Writing down that “stuff” gives it shape, and helps me calibrate whether I’m spending my gift of time well (or, sometimes, not).

I do like the writing part, too. A different form of creation, transforming a jumble of experiences and ideas into an efficient, coherent progression of words that tell their story. Somewhere around one or two thousand words usually does the trick. My friend Claude put together this little timeline of my posts (click for an interactive version):

Pretty cool, but it also highlights a slight but steady decline in frequency, especially over the past two years. Some of that is just time spent on other projects that may land on the gallery or book journal or (gasp) offline altogether. But that’s not the whole story.

Quieting

On the positive side, I think it mirrors a general “quieting” of my life that I’m a big fan of. I talk a lot less than I used to (Lara who has to live with me all the time now may disagree) — to the point where if I spend much time actively engaging with people it leaves me physically hoarse and exhausted.

This is a little weird after a decades-long career in an industry known for being loud and interruptive and full of voices. I thrived in that setting, but I like the alternative a lot. My ex-colleagues may derive some measure of amusement from this.

It’s not about less intensity or enthusiasm or fun or depth; it’s just different. It sounds so woo-woo I’m embarrassed to write it down, but the conversations and debates I used to have with people, I find myself having with wood, physics, nature, computers, animals, beaches, books, plants… it’s kind of neat.

Garden Party

I’m also writing less about “the old days,” except when some specific lesson or event has bearing on something new. I’ve heard loud and clear that folks love to hear about old Microsoft and the dot com days, and I get that — there are some great stories.

But like Ricky Nelson, I’m just more interested in looking forward than looking back. That’s why I moved to the West Coast in the first place, and why I jumped from industry to industry, and why I retired to explore new corners of the world. “If memories were all I sang, I’d rather drive a truck.” Although to be fair, “driving a truck” actually sounds super-fun too. Next month I’ll be road-tripping a cargo van across the USA; not quite the same but maybe a taste.

Of course there’s comfort and satisfaction in looking back, and there’s nothing wrong with that. I love beers with old friends, swapping stories and carping about kids these days. But I’ve seen too many people become frozen in time — the “old days” morph from real history into idealized fantasies that don’t serve anybody. Especially at a time when super-freaking old folks are clinging with their fingers and toes to political power.

Toxic Commons

Which brings me to the last, and most negative, thing slowing me down — a deep and real disillusionment with the state of our country. I try hard to separate my daily explorations from the endless stream of crazy news, but it doesn’t always work. And it probably shouldn’t, because what’s going on now is fundamentally sad, unnecessary and destructive.

It’s not all about “politics,” although the politicians and influencers have certainly figured out how to capitalize. It’s about an ignorant, petty mob of children vandalizing civilization and progress — far, far closer to the Red Guards than anything remotely American.

Actually, I keep finding myself using that term “American” as it has lived in my head since I was a kid. But I have to remind myself — WE voted for this, twice. WE are continuing to support it. WE love having our public health systems ruined by lying charlatans without medical experience. WE love having our capital city occupied by federal troops for no reason other than intimidation. WE love vilifying the poorest and most vulnerable to satisfy our pathetic, racist, everything-phobic egos. WE love that our president is a disgusting bloated coward, because it gives us an excuse for being awful ourselves.

The truth is, I don’t really know how to live in public anymore — so I do less of it. I’m still in the fight where I can have some impact, but it gets harder with every company and college and ex-colleague that bends their knee.

Whew. Sorry for the navel-gazing. But hey, it’s my site and I get to be a downer at least once every 110 posts. Next one will have some witty observations about AI or my latest power tool or how kids don’t know how to code anymore — I promise!

Rivian v Tesla (I’m a Mac, and I’m a PC)

~ Sean Nolan

Update 8/12: We just got the car back from the Bellevue Rivian service center and it does appear that the car had a problem with the cooling system (the infamous 5-way valve others have posted about, e.g., here). Service took way too long — they’re clearly struggling with the logistics of scale-up — but they got me a reasonable loaner and (I think) found the root cause. Hopefully we’ll have a smoother fast charging experience when we head back to California in a couple of months. Woo hoo!

As I mentioned a couple of weeks ago, we recently traded in our trusty 2019 Tesla Model X (“Miss Scarlet”) for a 2025 Rivian R1S (“Gonzo the Great”). We bought the Tesla for a combination of selfish and political reasons, and traded it in for those same reasons. But I have strong feelings about the legacy auto industry and EVs — first because of the decades they spent trying to slow-boat the transition, and second because their EVs just kind of suck. I want a software car, not a legacy gas car with the engine swapped out.  

To me at least, Rivian stands out as the next generation of true EV first technology. Their experience with delivery trucks has given them a ton of experience in a tough environment, and Keep the world adventurous forever speaks not just to a sustainable future, but an awesome one.

So — into the breech we went. Our primary use cases for the car are pretty straightforward and unique to our weird lives:

  1. Short trips around town
  2. Ferry rides to and from Whidbey
  3. Multi-day road trips up and down the West Coast
  4. Drive in movies

I waited to write this until we had our first experience with #3. And while we did run into one significant unpleasantness (see “The Ugly” below), on balance I’m pretty enthused about the vehicle. Fingers crossed…

Purchase and Delivery

Industry inertia (and lobbies) are amazing things. Thanks to a bunch of archaic regulations wrt dealer networks, you can’t actually buy a Rivian (or a Tesla) in Washington State. When we went to test drive the car at University Village, the reps were very clear: we can’t talk about selling you this car, the price, your trade in, or anything like that. We’re just letting you take a ride in this wonderful vehicle and answering questions about it!

When it came time to actually buy the car, we had do so online, nominally from Illinois. They’ve done a good job with the process of providing them with documentation, signing notary forms, wiring money, etc. — actually way better than the typical dealer circus.

The car showed up at the Bellevue service center, and a super-enthusiastic lady spent an hour with us walking through all of the features. She was a bit over the top — but I appreciated what seemed to be honest enthusiasm for the car and the company. A good start! We waved goodbye to Miss Scarlet and drove off home.

Fun related fact — when the registration showed up in the mail, it showed a (large) itemization for use tax, the kind you pay to the state when buying a used vehicle from a private seller. This was in lieu of WA sales tax, and our last reminder of the dealership lobby — we actually bought the car “used” and imported it from Illinois!

The Good (a lot)

Our Rivian is an R1S “Tri-Motor” (SUV style). Fully-charged range is just under 350 miles, and those extra miles (compared to the X) really shine. It is super-fast and accelerates like a beast. Definitely BIG and TALL, even in “kneeling” mode — we added EV Sportline running boards to help folks get in and out more easily (a nice product BTW — we also bought their new Aero Cover Plates but I’d skip those next time).  

The suspension is crazy nice … the Tesla raises and lowers itself as well, but with a spread of 6.5 inches (from 8 to 14.5) the Rivian’s ability to adjust is insane. The road to our Bellevue house has these killer abrupt speed bumps — in the Tesla it was bone shaking; in the Rivian I barely notice them at speed (sorry neighbors).

Everything about the cabin is nicer as well. The boxy shape of the Rivian makes it way easier to transport stuff, the seats are cushy on my back even over a 12-hour day, and the console is well-designed so I can see everything (on the Tesla I kept having to peer around the wheel to see important stuff on the heads-up).

A frunk you can hose out is a great touch for those of us who use it to move trash.

I am in love with the wifi hotspot. Being able to cast any video to the car is super, although I have had an issue or two getting the phone to “see” the Rivian.

Dog mode (I mean “creature comfort”) is basically equivalent to the Tesla, but Camp mode is better in the Rivian. There’s a lot more control of what is on and off, and the automatic leveling feature is pretty nifty. Love the ability to illuminate around the vehicle easily, especially picking up mail or dropping trash late at night.

The adaptive/matrix headlights are simply magic. I posted a picture of their “charging” display last time, but the really cool bit is how they work when driving. Instead of just two headlights, there’s a whole bar of individually-focused LEDs which stay in high beam mode permanently. When they detect an oncoming vehicle or person they turn off only the lights directed at that object. The difference in visibility is striking.

Some of these are just better because the car is five years newer than our Tesla (Canada started allowing adaptive lights in 2018). But that doesn’t make me any less giddy; e.g., I cannot express how much I love the synthesized overhead view for parking. Did I mention it’s a big car? Having perfect view of lines and curbs is life changing.

The Less Good (Driver Assist)

Rivian has some great automated driving features, but it doesn’t touch the Tesla. I loved Full Self-Driving on our X. It isn’t perfect by any means, but the idea that the car really knows how to drive itself anywhere and in virtually any environment is a big, hairy, wonderful goal. And Tesla is getting there.

The Rivian is much more about helping you drive conveniently and safely — and it honestly does a very good job of that. It’s just not the same thing, at least not yet. It currently has three assist modes:

  1. Adaptive cruise control.
  2. Driver Assist. This is available on most highways and will keep you in your lane. It will perform safe lane changes on request only (using the turn signal).
  3. Enhanced Driver Assist. The same as #2 but hands-free. Available about 2/3 of the time that regular Assist is enabled.

Adaptive cruise works great, easy peasy. The quality of driving in the Assist modes is solid but it does get fooled sometimes; our working theory is that it struggles with spotty road coloring and lane markings. A few times we got into a weird harmonic where it kept swerving back and forth within the lane, and twice on our big drive it lost track without knowing it.

The other challenge is that the Assist features are clearly map-driven. Of course the car is adapting to its immediate situation, but the features are enabled/disabled based on map location and how much the system knows about the road layout. Because roads are always changing, this means that the feature frequently turns itself off and on. This is especially true with Enhanced mode — there are significant parts of I5 where it’s just not worth using it, because it’ll just be off again in thirty seconds.

The eye tracking is also really aggressive — I can barely pick a song on the main console without it yelling at me to look at the road.

All in all — by the middle of day one I’d figured out the idiosyncrasies and we were humming along pretty well. The overall experience is a lot like the Tesla before FSD. But I expect it will take a big stair step before they’ll go much beyond that — we’ll see.

The Ugly (Battery Temperature)

OK, this one was a killer. I’m optimistic that it’s not a fatal flaw in the car — when we’re back in Bellevue I’m going to have them see if we may be a victim of a bad cooling valve; hopefully that’s all it is. Still, it sucked.

We were impressed with our first stop at an EVgo station in Aurora, OR. The charge was amazingly fast compared to what we were used to with the X. In and out, off we go.

The second stop was a Rivian Adventure Network station in Roseburg, OR. It was pretty warm out at this point, about 92. Started fine (and fast), but after taking the dog out to walk for about ten minutes I returned to find the charger at about 2kW (for reference this is about what we get off the 120V outlet at our house). Huh.

Here’s where I think I made things (much) worse, although it falls into the category of “the car should have known better” for sure. Unclear why the charge rate had dropped, I just unplugged and plugged back in. That worked for a few minutes, then it dropped again. I tried rebooting the car and then again with a different charger. Worked for a bit and then dropped. Eventually I decided we had enough charge to get to the next stop anyways, so we’d just leave.

Pulling onto the onramp, the accelerator didn’t respond but then jerked forward so I just kept going. Within about five minutes on the road, the car entered “turtle mode” (I have learned these are dreaded words in the Rivian world) and dropped speed to 20mph. I crept along to the next exit as speed continued to drop. We parked the car in a safe spot and tried to figure out what was up.

Rebooted the car again and it seemed OK, so back on the highway and nope, same thing within about five minutes. At this point we exited and parked by the side of the road to call Rivian service (shout out to the multiple good folks who stopped ask if we were ok). Service was mostly uniquely unhelpful, although to be fair not sure what they could do — they offered a tow three hours north to the nearest Rivian service center, yeesh.

At the very end of a frustrating call, the rep suggested that battery temp might be an issue and showed us how to look at that … aha. The battery was close to redline at 132F, and surely had been much higher before we sat talking to service for a half hour.

Aware of the issue, we were able to get back on the road, watch the temp around charging stations, and make it down to Ventura without further incident through air temps of 100F and worse. Thanks to the Seattle PNW Rivian Facebook group for offering online therapy while we were in the hotel that night! TLDR:

  • We don’t know why the initial charging speed drop happened.
  • I surely made it much worse by forcing high-speed charging on a hot battery.
  • Bumping up against the 130F mark isn’t unheard of in extreme heat, but it seems likely that there is something suboptimal going on with our heat pump.
  • Limiting fast charges to 85% seems to do the trick; I have no concerns about getting comfortably back to Washington where we’ll have them take a look. Knock on wood.

I’m a Mac, and I’m a PC

At the end of the day, buying a Rivian feels a lot like using Windows instead of a Mac circa the late 90s. The Mac mostly just worked — as long as you used an Apple printer, Apple Mouse, Apple networking, Apple development tools, etc.. The PC had incredible power and a zillion innovative options, but you paid for that in complexity and reliability. Downloading drivers and keeping them up to date was (still is) such a hassle.

Side note: amusingly, in their attempt to capitalize on these real differences, Apple turned John Hodgman (“PC”) into an everyman hero. Who didn’t love that guy?

Boy, does this apply to charging — with the Tesla you mostly stay in your network, and they’ve made the system pretty idiot-proof. This is decaying over time for sure … but the difference is still very much there. From one icon to eight!

The running boards are another example. It’s pretty clear that the car needs them for many folks to comfortably climb into the car. But rather than make them standard, an ecosystem of accessory companies has emerged, each with their own features and quirks (be careful that the Rivian service folks can reach the official jack points!). The community seems to revel in the optionality and DIY of it all, and it is pretty cool — but also complex.

Or take the video streaming. I can watch way more stuff on my Rivian thanks to Google Casting vs the strictly-controlled theater apps on the X. But to use it you need to stream from your phone which is always a bit flaky, and the phone has to be on the car’s hotspot vs. any other Wifi network, and…. you see what I mean.

Early Conclusions

Feeling “stuck” in the middle of a road trip is a deeply crappy experience, and it’s hard not to let that color my first impressions. But I really wasn’t stuck, and in retrospect the issues seem pretty clear and fixable, even before we get it into service.

Taking that for what it is, I really do love the car. I don’t hide my politics and I do feel better knowing that I’m no longer adding to the damage that Elon is doing to my country, but that’s just a part of it. The Tesla is looking more and more like an amazing, ground-breaking, super-awesome first generation of the real EV transition — and that it’s time for the second generation to take the wheel (ha).

Rivian is doing that. They’ve built a sweet product that is fun and comfy to drive, has truly state-of-the-art features, and is positioned to win the day. Both over Tesla and the legacy automakers who, as we all know, really just wish the whole thing would go away. I’m calling it a win.

Woke Not Whiny

~ Sean Nolan

Back in the nineties, recruiting was part of my job as a developer at Microsoft. The bar was high on purpose, and we were proud of it. A tough interview process is good for a lot of (mostly obvious) reasons, but for me the most important was that it created an assumption of competence at the company. If you were in the building, you probably weren’t an idiot.

As a person who collaborates best through, let’s say, “vigorous debate,” this was unbelievably liberating. If an idea was stupid, I could say that — and folks could say the same to me — and except in some very rare cases it didn’t get personal.

People are always surprised when I tell the (admittedly hilarious) story of Bill and the Gunshot Wound, but that was the path to success at Microsoft. Take a position and defend the hell out of it — if you win against all comers, you were probably right. I loved it, and it mostly worked.

And it’s exactly because it worked so well that it took me years to recognize that we (I) could do even better. Microsoft in the 80s and 90s conflated a particular personality type with intelligence and success. It just so happened to be my personality type, and I spent those early years blissfully working with a bunch of copies of myself. Building tools that worked the best for people like me.

Huh.

Now before we go too far here, let me be clear: I don’t abide stupid. Stupid kills and stupid loses. But it turns out that being stupid is mostly a personal choice. A half century of watching people has taught me that “everybody is good at something” is actually true — success comes down to taking ownership for figuring out what that is and working to become great at it. Stupid is just lazy.

Anyways, sitting here I can play back in my head dozens of real, brilliant individuals whose diversity and alternate perspectives made me and my products better — and sadly, a few (especially in my early years) that crashed out because I was unwilling to adjust my own behavior to optimize theirs.

I like to believe that I’ve gotten better at this year over year — but there is one person who probably doesn’t have a clue just how much she changed my life. She is extraordinarily smart, with a remarkable sense of personal responsibility and empathy. We built great things together for years, standing at whiteboards while I pushed and yelled and played every side of an argument to force her to defend her proposals.

Except one afternoon in the middle of a session, just a regular day, she sat down with actual tears in her eyes and asked me: “Why do you DO this?” She was exhausted and it was clear she had lost confidence in herself. I was honestly taken aback — none of this is personal, and if I didn’t believe she were the best person for this job, we wouldn’t have been there in the first place!

And yes, she “knew” this, but it didn’t matter. It turns out she was doing great work in spite of, not because of, my constant barrage of devil’s advocacy. Or more accurately, the way I was presenting my arguments. Some very small adjustments on my part led directly not just to a happier and more confident colleague, but to faster and better results. ***

And that’s the kicker — adjusting my own behavior and perceptions didn’t just help the other person, it got me more of what I wanted too, without costing me anything. You’d have to be crazy not to make that trade, right?  Yet that is exactly what the “anti-woke” crowd pushes every day. Somehow we’ve created a ton of people that are so weak, so scared of anything that challenges themselves or their beliefs, they are actually pushing for the government to protect their fragile egos at their own expense. It’s insane.

The ”alpha male” narrative is not just super-weird, it’s also 100% upside down. What could possibly be weaker than crying “unfair” every time somebody challenges your position? Having a healthy ego by definition should mean you can listen and do a bit of introspection without being butt-hurt. I can be proud of my accomplishments and have a clear-eyed view of my unearned advantages at the same time.

This is true even if, in some incredibly rarified cases, the scales tip a little the other way. If DEI programs make it just a tiny bit harder for a white dude to get into a particular college or job, step up and work a little harder — isn’t that exactly what you’ve been saying to disadvantaged populations for years when the tilt is reversed? Don’t be such a chicken-sh*t.

I’m just really tired of the voice of “personal responsibility” and “strong men” being such a farce. Let’s create a world that works every day to be better and stronger and solve problems — not one that whines and cries and pretends it’s tough by buying a gun and hanging a flag on the truck. Wake up, white guys.

*** I suspect if my spouse reads this she’s going to laugh long and loud, because she taught me this lesson very early in our relationship — yelling in an argument was NOT going to end well. So why did it take so long to translate to work? Human psychology is weird, man.

Work the Problem

~ Sean Nolan

My father-in-law is actually a really great, super-nice guy. But that doesn’t mean he’ll pass up an opportunity to give me a hard time — say, paddling the canoe towards the alligators while I’m in the bow. Or more to the point of today’s post, bringing over the New York Times Crossword puzzle and pretending we’ll do it “together.” For years we’d sit down and he’d basically watch me stare blankly at the clues. Then, on cue every thirty seconds or so, he’d throw one out like he was just guessing: “maybe try OBLIGEE for 20 across?” He always knew the answers, and he knew I knew he knew the answers, and that was the fun part. Infuriating, but I’m just too competitive to not get sucked in again and again.

So when I retired a few years ago, one of the many items on my to-do list was “get good at the NYT crossword.” Since then I’ve done the puzzle pretty much every day, missing maybe a half dozen over three years. And of course, it turns out that solving crosswords is a skill like any other; enough practice and it becomes pretty straightforward. Most interesting to me, it turns out that techniques for solving puzzles overlap a ton with techniques for debugging software.

Now maybe I shouldn’t be surprised that this is the case, because both tasks are about establishing possibilities and fitting them into a set of constraints. Or as Gene Kranz (or Margo Madison) might put it, “working the problem” until you find the answer that was hiding in there all along.

Note: I’m going to be spoiling a few answers for recent NYT puzzles — if you do them in arrears you have been warned!

The Basics

OK, on the surface this seems pretty obvious; we’ve all been doing some form of crossword since elementary school vocabulary worksheets. A bunch of clues to interlocking words, across and down in a grid. It helps to know a lot of trivia, a few foreign words and the names of various artists (modern and not so much). The NYT puzzle goes through a weekly progression where Monday is the easiest and Saturday the hardest. Many folks are surprised to learn that Sunday isn’t the most difficult; it’s just the “biggest,” meant to be done leisurely over your weekend soft boiled egg and tea looking out from your brownstone over Central Park or whatever. Thursday is an interesting day, because it tends to have a “gimmick” — more on that in a bit.

Beyond just “knowing stuff,” the first thing folks learn about puzzles is that there are certain words that repeat all the time. Just a very few:

  • The artists Sia and Ice-T are quite popular with puzzle authors.
  • Their favorite Olympic sport is definitely fencing, specifically épée.
  • Just about every disturbance is an ado, many people are prigs or fops and get into snits.
  • Authors all wish they drove a GTO or at least a car with a T-top.
  • In amour with your amie? Maybe it’s just the romance of été along the Loire.

You get the idea. These common words are usually the quickest way to get started on a puzzle, because they’re easy to pick out. The checklist of things I walk through when faced with a weird debugging problem works much the same way:

  • Off-by-one?
  • Out of memory?
  • Infinite loop or recursion?
  • I/O failure?
  • Unexpected input?
  • Divide by zero?

In both cases, a relatively small number of solutions account for an outsized proportion of problems. And because they happen so frequently, you being to develop a “feel” for their symptoms (or clues) … most of the time running through these common cases will put you on the right path to a complete solution.

Confirmation

Another sort-of obvious technique for puzzle solving is to increase confidence or eliminate possibilities through confirmation. As much as I can, I defer writing in answers until I find and least one intersecting answer that corroborates the first. For example, since I can’t remember what networks show what shows, the four-letter answer for “Several CBS dramas” could be either “CSIS” or “NCIS”. If overlapping clues support a C or N in the first position, or a S or C in the second, that gives me greater confidence in both of them.

One thing that makes a puzzle harder is to provide fewer or more complicated opportunities for easy confirmation. Saturday grids tend to have more “open” sections with longer, multi-word answers placed intersecting and side-by-side. Another approach is to isolate sections of the puzzle from each other, so that solving one doesn’t help you get started on the others.

Confirmation is key to debugging as well. Theories are easy to come by, especially with bugs related to user input or distributed systems. Maybe two users clicked “submit” at the same time. Perhaps one system crashed while another was holding an open network connection. Or it could be something that happens on every 32nd request. Coming up with these ideas is important, but proving or eliminating them is the heart of the work.

Unless you’re able to reproduce the bug on demand, your best friend for confirmation is detailed, timestamped system logs. These let you look back in time and see who was doing what when; the tiny performance hit is always, always, always worth it. Today most enterprises use third-party or cloud-provided distributed logging solutions, but an old-school rotating disk file can do the job just as well. Either way, great engineers learn to slice and dice logs to find evidence that corroborates and confirms their theories — there’s no rush like finding that smoking gun.

Getting Stuck

Sometimes — whether you’re working a puzzle or debugging code or something else — you just get stuck. This is what separates the grownups from the children in the room — do you stick it out and find some way to make progress, or do you just give up, go home, and let it be somebody else’s problem? If you sense a little moral judgment in that phrasing you’d be correct. 😉

While it’s not  the easy answer, years of experience have proven to me that the best thing to do is just keep trying. Read the clues again. Force yourself to look beyond your initial interpretation — a “70s Ford” may not actually be a car! The answers are in there, and if you give your brain enough chances to fire the right neurons, it will happen. And it works for debugging too; a full decade ago and in another life I wrote about it here: The definition of insanity works!

But while giving up is not an option, walking away for a few minutes often is a great idea. It’s quite surprising how often a change of scene will help shake out a solution — answers just pop out where they seemed impossible before. Sticking it out and changing context — it’s all about giving your brain space to make connections.

Patterns and Partial Answers

Something is better than nothing — so when you just can’t figure out that next answer, there are a few ways to keeping pushing forward:

  • Plurals usually end with -S, past tense with -ED and active verbs with -ING. Of course these are often wrong, but they can be super-helpful as confirmation for intersecting clues.
  • Sometimes you can make a guess at part of the answer. For example, it’s a pretty good bet that “Where you’ll find women out to drink” ends with BAR, and “Platonic outing” starts with FRIEND.
  • Even when you’ve got no confirmation at all, it can be helpful to write in guesses that fit. Seeing letters in place is a great way to trigger those stubborn neurons, and I can easily scribble over at least two mistakes before the square becomes illegible!

These techniques all reduce the unknowns. We’re not built to juggle too many things in our head simultaneously, and fewer empty squares makes it easier to focus on the ones that remain. It really doesn’t matter whether the guesses are right or wrong — that will become evident as we make progress on their neighbors.

Once again (and I swear I’m not just forcing these), this same approach helps debug code, where it’s called “divide and conquer.” Split the problem in half and just assume that one half is working correctly. If that’s true, the bug must be in the other half — so find it there. If you can’t, then your assumption must have been wrong. Either way, you’ve made progress.

The Reverse Solve

I mentioned that Thursdays are often a “gimmick” puzzle. Sometimes these are structural — answers extend into the black squares, or multiple letters go into one square. Other times they’re just tricky themes. For example, last week the trick was about missing letters in a set of clues:

  1. First, solve the clue “thrice-remade movie” (A-STAR-IS-BORN)
  2. Next reparse that answer as six words (A-STAR-IS-B-OR-N)
  3. Replace the stars in clues with a B or an N to get the actual clue. For example, replace the star in “*acre on the ocean floor” to get “Nacre on the ocean floor” for the answer “MOTHER-OF-PEARL”

There was no way I was getting this by working “forwards” — I got the movie title, but couldn’t for the life of me figure out how to parse that into six “words.” Instead, I ended up working in reverse. I was able to solve “Acre on the ocean floor” as “mother of pearl” based on confirmation — it had to be correct, but it made no sense. But I knew that M-O-P was another term for NACRE (one of those commonly reused crossword favorites), and eventually it dawned on me that that could fit with “acre” in the clue, and eventually that let me work out the “B or N” part of the primary clue. From there, things fell into place quickly (“*assist in a foursome” was MCCARTNEY, etc.).

And believe it or not, this is yet another way to make your way through a debugging problem — you know the outcome of the bug has to be true because you see it happening. Look at the step right before this — what has to be true for that outcome to happen? And if that’s true, what has to come before that? It’s kind of amazing how often novice debuggers will simply deny what they are seeing right in front of their face. If only I had a nickel for every time I disproved “that just can’t happen” in production code….

… and that’ll do it for yet another wander through “something-that-isn’t-code actually working a lot like code.” It seems clear that over the next few years, AI is going to be writing a lot of software. And that’s OK because progress is progress and there are tons of coding jobs that at their core really should be done by machine. But I do love the craft of it all, and I hope there will remain some space for artisan code, just like we still value artisan woodwork and clever crosswords. I know that sounds a bit funny, but I mean it. We will see!

Nit-Picking my Tesla

~ Sean Nolan

Lara, Copper and I just finished another LA-Seattle run in our 2019 Tesla; just about 23 hours on the road including charging stops. I enjoy the drive — great scenery along most of the route, a good audiobook or two, a lot of snacks, and a quick overnight to explore a new town (OK, usually the same town — there are great donuts in Mount Shasta).

Plus, of course, Autopilot and Dog Mode and optimized routing through the remarkable Supercharger network. Five years in, this stuff just doesn’t get old.

But that doesn’t mean it’s perfect. So from a place of love, and on the slim chance that tweeting non-crazy stuff at Elon still works, a few things that need fixing. From a guy who has spent a lot of highway miles thinking about it.

1. Routing & Charging Stops

Our car gets about 300 miles on a full charge, and there are a ton of Superchargers up and down the West Coast. Getting stranded is just not a real concern, unless you’re dumb — the same kind of dumb that runs out of gas in a “normal” car anyways. Just like in my “normal” car, I don’t like to push my range too low when I’m on a trip. I’m sure that different folks will have different tolerances for this kind of thing.

Tesla has a truly impressive routing system that takes charging into account. For example, when I start in Bellevue and want to drive south to Mount Shasta, it figures out where I’ll need to charge and adds those stops along the way: Kelso, Harrisburg, and Medford. It estimates the remaining charge at each stop, and how long I’ll have to stay before setting off again. It’s smart about charging time too — efficiency slows way down as you approach a full charge, so it usually arranges things so you only need to get to around 80% or even less before continuing.

And because many factors can impact range, it constantly re-assesses these stops as you drive. If the margin to reach a charger gets too slim, it will automatically find a closer option and rejigger the rest of the trip to match. It’s really fun to experience it in action — the kind of innovation that cloud-connected software makes possible.

Anyways, this is all great — BUT. As you can see, Tesla considers it safe to arrive at a stop with 9% charge remaining. At 9% I have about 25 miles left, the gauges are red and the car is disabling AC to conserve power. Sure it’s “fine,” but my blood pressure doesn’t concur.

Ask: Let me set a minimum charge level. I’d love to use 18% as the floor (about 55 miles for my car). Without this setting, I often feel compelled to mess around manually with my route to stop earlier and/or charge longer. Is it a huge deal? No — but it’d be nice to just not think about it. Help me out here.

2. Speed Limit Mistakes

When driving on Autopilot, the car automatically reduces speed when the effective speed limit drops (only slower, never faster). This is a fine feature, and certainly handy when you’re coming into a metro area or whatever and the limit is bopping up and down frequently. The problem is that, especially in the last few FSD iterations, the system gets confused by signs that set alternative limits — e.g., for trucks towing trailers or with more than two axles.

I actually didn’t realize that the car still tried to read signs at all; I just assumed it was using some published data source and GPS. But this happened at least ten times on our last drive; the sudden slowdown to 55 when you’re happily zipping along with traffic at 75 is jarring and annoying. I don’t think it’s strictly “dangerous” since braking is sensitive to traffic to the rear, but it still sucks.

Ask: Do better tracking the current speed limit.

3. Bogarting the Passing Lane

I like to drive consistently about five to eight miles over the speed limit; pretty safe from tickets while still moving along at a reasonable clip. This means I stay mostly in the travel lane, occasionally pulling over to pass folks on the left. Since I’m only going a few clicks faster than the folks I’m passing, maneuvers happen pretty slowly — so I keep an eye out for folks pulling up behind me in the fast lane and stay out of their way as much as possible.

FSD doesn’t deal super-well with this. When it notices that I’m going faster than a car ahead of me — even though that car is quite far ahead and I won’t approach it for some time — it moves into the fast lane. While I don’t love this, it would be OK except that the system is NOT sensitive to folks approaching from behind. So left to its own devices, at my preferred speed I’d be that a**hat squatting in the passing lane blocking people trying to pass.

The system does know generally that riding the passing lane is a bad idea and will sometimes move to the right. But something about this particular scenario screws up that behavior. Ask: Improve awareness of cars approaching from the rear and get out of their way.

4. Semi-Automated Lane Changes

Because of the behavior in #3, I will often put the car into “Minimal Lane Changes” mode. This setting limits automatic lane changes to safety reasons (e.g., to make room during a merge), or to follow an active route (e.g., to take a highway exit). Honestly I don’t mind this at all — it’s a nice compromise between letting the car do the work and being an active driver. But there are a couple of annoying things that need work.

First, the setting doesn’t stick. I cannot for the life of me figure out why they did it this way, but “Minimal Lane Changes” stays active only for the duration of the current drive. They take pains to tell you this every time you turn it on, so it’s not just a bug. I either have to remember myself, or wait for the car to attempt a lane change, cancel it, and catch the popup on the main console before it disappears. This is the only setting on the car that works this way! Ask: Make the “Minimal Lane Changes” setting sticky.

The other challenge here is that, just in the past few software iterations and only sometimes, the car is super-slow to respond to lane change requests. The way this works is the driver uses the turn signal to “request” a lane change, and the car automatically moves over when the target lane is clear. Typically this maneuver is very responsive and happens right away. But every few times now, the car just sits there. The blinker is on, the FSD display shows a clear path to the target lane, but nothing happens for 15-30 seconds. Eventually the car moves over, but only after somebody is annoyed — either me trying to pass, or somebody else trying to pass me!

Ask: Figure out why the car is (sometimes) sluggish to move into a clear lane.

5. Mid Speed Follow Distance

A few software iterations ago, Tesla removed the ability to fine-tune follow distance by number of car lengths. Instead, it’s managed by the FSD “profile” (Chill, Average, or Assertive) without direct control. As with some of the other issues I’ve brought up, “generally” it’s fine, but the inability to change the distance manually is a problem at speeds around 25-40mph, especially on a congested highway. In this range, the gap is too large and invites people to constantly cut in front of the car. Honestly it must be what it feels like to drive a semi.

I suspect this is going to be a tough one to get right automatically, because closing up the distance too much — even though it’s what almost all drivers do — certainly increases the risk of a fender bender. And there a ton of folks out there happy to hold FSD to a much higher standard that our fellow humans. So, my ask: At least for now, give me back fine-grained control of follow distance.

6. “Miles Remaining” Calculation

Maybe this is my blood pressure talking again, but one of the most important stats on the display is “Miles Remaining.” It basically takes the place of the gas gauge, giving me a sense of which destinations are in range and which are not. But unlike gas cars which have relatively consistent MPG, electric miles are super-variable. Elevation, speed, temperature and a host of other factors impact this — coming down out of the Oregon passes even turns the dial significantly backwards, gaining about twenty miles thanks to regenerative braking!

The value reported is clearly based on the last X miles of driving, for some unpublished value of X. That is, it’s pretty accurate if you keep driving in the same conditions. Which is fair if you think about it; the car can’t read your mind to know where you’ll be driving next. Or can it?

From what I can figure out, it seems that the “% remaining at next charge” value we talked about in #1 does predict the future, based on the active route. Those California-Oregon passes are my best recent example. Leaving Corning (CA) the Tesla told me I’d have about 15% charge remaining at my next stop in Medford (OR). By the time I’d climbed to the top of the passes, that expectation was down to 10%, but my “Miles Remaining” value was far, far lower — nowhere near enough to make Medford. But as I said earlier, as we dropped down the other side, “Miles Remaining” actually climbed about 20 miles and we cruised into Medford with just about exactly 10% remaining.

Now I don’t know what the software is really doing here. But somehow, even at the top of the pass, the routing system “knew” that I would use far less energy getting from there to Medford than I had so far. Maybe it did this using data from previous trips (my own and others’), or maybe it actually considers elevation. But one way or the other, the routing system consistently estimates residual charge better than the real-time system does.

Ask: When there is an active route, sync these systems up to show the same information!

There are other issues here and there: the automatic wipers are sadly useless, and I wish I could convince the falcon doors that the posts in my California garage are NOT in their way. But I don’t want to get greedy, and I don’t want to give the impression that these are anything but nits. To be clear, I love my Tesla. I’m turned off by some of Elon’s personal behavior, but I also believe he’s a straight-up genius whose ventures are having a remarkably positive impact on the future of our species. It’s adorable when folks claim (with all the confidence of a college freshman) that “anyone” could have done what he’s done. Spare me, please.

Lastly — I think it’s useful to note that all of the problems I’ve brought up are software issues. I can’t stress enough how important this is — what Tesla is building (and maybe Rivian and a few others) is fundamentally different than what the traditional automakers are building. They’re not just swapping out a gas motor for electric; they’re inventing a whole new kind of vehicle. It’s a lot less of a hassle to deal with a “recall” when it means your car updates overnight in your garage. The car just gets better and better day by day. That’s amazing, and super-fun.

Thoughts from the Hut

~ Sean Nolan

Last week I flew into Colorado and met my brother and cousin for an overnight trip to the Sangree Froelicher “hut” outside of Leadville. It was my second winter backcountry trip, about 3.5 miles into the San Isabel National Forest on an amazing rented pair of dps skis. Truth be told I was not in particularly great shape for this trip, but I managed to weekend-warrior my way through it with no damage beyond an impressively large blister on my little toe.

Sangree’s hut is part of the 10th Mountain Division Hut Association network, more than three dozen chalets tucked into the backcountry around Colorado. It’s hard to describe just how magical the “huts” are — multi-story, multi-room log masterpieces with wood stoves and ovens, bedrooms with mattresses, sinks and cisterns and permanent outhouses, propane burners and solar lights. Arriving at one after a day skinning into the silent, snow-covered forests and hills is like stumbling into some crazy Narnia/Wardrobe situation (minus the White Witch).  

The association is named in honor of the 10th Mountain Division of the US Army, famous for operations in the Italian Alps during WW2, and which dominates the history of skiing in the United States — at least sixty-two modern resorts were founded by folks connected to the 10th, including Aspen, Vail, Sugarbush — even my hometown favorite Crystal Mountain. Sangree Froelicher, for whom our hut this year was named, was killed in action during the Italian campaign. Evenings in the huts lend themselves to a bit of reflection, and to the conflicting emotions of being thankful for the sacrifice of brave people, but also frustrated at the stupidity of those that created the need for it.

Anyway — while the place was amazing, of course there’s no “staff” or anything — so there are plenty of chores to do. The four of us shared our hut with another group of eight up for their own annual trip, and I couldn’t help but notice that the dynamics shared a lot in common with the workplaces I inhabited for so many years. People are people, and it’s always interesting to see what happens when we’re thrown together.

1. Tortoises and Hares

I tend to skin quickly and take frequent breaks. My cousin and her husband are slower but steady. My brother is fast AND steady. The differences caused us to spread out along the trail, but as we all came back together at the hut, it was clear that everybody had had their best day. Being alone in the backcountry is about remembering just how big the world is — and that despite being pretty insignificant at the end of the day, we’re a part of it, and we belong. Sounds corny, but the utter, complete, snow-covered silence is really special.

Travelling at your own pace is also a great exercise in self-motivation. Three and a half miles doesn’t seem like much, but for an old guy shuffling uphill on skis with a pack — it’s plenty.  And there’s something to be said for doing it on your own. Most of our accomplishments in the modern world are parts of a larger whole, which is a-ok. But it’s nice to be able to take end-to-end credit for once! 😉

Most of all, we all made it, in our own way. And in plenty of time for a beer before dinner.

2. Leaders, Followers and Slackers

The huts are amazing — but there’s also a lot to do. Wood needs to be split and carried inside. The stoves need to be tended, and snow brought in to melt and boil for drinking water. Since COVID there are enhanced (and appreciated) cleaning protocols, especially for sleeping and kitchen areas. Snow needs to be shoveled, food prepared, dishes washed and bleached.

But unlike most day-to-day situations, there’s no official hierarchy or assignments, within or between groups. Inevitably, a few folks self-identify (not necessarily explicitly) as “leaders” and start organizing things. Usually this leads to a too-many-cooks situation, as differences in style show themselves or people simply bump into each other trying to do the same job. But as long as nobody is a total a**hole, it usually works out just fine.

Having spent the bulk of my career as a “leader” (oh how I hate that word) — these days I revel in being a follower. I put in my share of work and then some, but much prefer to be labor and let somebody else call the shots. I’m the guy carrying in the wood and rinsing the dishes; somebody else can figure out how to adjust the flue on the wood stove and take the rap when it’s too cold overnight!

Of course, there are always one or two folks who are happy to just skate by without helping at all. And over the course of only one or two nights, it’s pretty easy to “hide” and take advantage — I only tend to notice thanks to years trying to optimize my software teams. But I have learned to delay judgment a bit — there are many legitimate reasons people hold back (lack of confidence, shyness, physical issues, etc.). The ones that are just lazy confuse me; do they really not feel bad? It’s weird.

3. Sharing and Mansplaining

There’s a bunch of technology in the huts — the woodstoves have a million ways to fine-tune them; there’s a big pot for snowmelt; a rooftop cistern and hand-pump at the sink; solar lights with a battery bank; wood-fired ovens and propane burners; you get the idea. There are also plenty of traditions and social expectations about sharing bedrooms and kitchens and drying racks and such. It can actually be a bit intimidating, especially for a first-timer trying to pitch in.

Most folks who come on these trips love to share what they know, and do so generously. But sharing “styles” vary dramatically, I’m sure often with people completely unaware of their effect on others. The women in my life have a lot to say about “mansplaining” — and in this situation there couldn’t be a better word. My personal approach is to just let it ride — if I learn a bit and never see these folks again, I’ll take the net win. But you can really see it eat at folks too, and that’s just a shame.

I love to share things I’ve learned too … it’s why I write this stuff down! And I’m sure that when I get on a roll, I can sometimes forget that other people know things too. Watching the dynamics in the hut is a great reminder: read the room, folks!

4. Us and Them

Segmenting people into groups seems to be wired into our brains. It’s surely a remnant of evolution — always assessing our environment to predict what is helpful, safe, dangerous and unknown. My brother and I were playing a game listening to the radio: how many songs are based on some vague “them” that said “we’d never make it” or whatever? There’s always a bad guy. I mean, this is how right-wing media (and to a much lesser extent all media) keeps our attention: be scared, be afraid, they’re after what is yours!

This even plays out — in a far gentler way — in the hut. There’s a standard set of ice-breaking questions between groups: Your first time here? Where are you from? How’d you like the hike in? Cold enough for you? Sometimes there’s an extrovert that tries to create connections (my friend Jim is a grandmaster at this), but more often we keep largely to ourselves except for shared chores and a few low-stakes exchanges. And people are different, of course. Perhaps one group is overtly religious and the other is not. Or one likes to stay up late with a drink and the other turns in early. And of course there’s politics, although thankfully it seems like the backcountry is understood to be a no-fly zone for that insanity.

In any case, a cozy, warm mountain hut is an excellent place to try to think differently — so I gave it a shot and offered my Chips Ahoy across the aisle. A major accomplishment for introvert Sean!

I really, really don’t miss people management at work. But I do miss the people. Sitting quietly in a corner of Sangree’s hut, watching folks figure out how to live and work together in a shared, special place — it made me a bit nostalgic for the great startup teams my friends and I created out of nothing. Everyone should experience just how awesome that can be. Even with a wicked blister and some dude on the other side of the room mansplaining how to properly store your skins overnight.

SQL Hammer (everything is a nail)

~ Sean Nolan

Update: SQL Hammer has been updated somewhat extensively since this post was written. It’s still a great introduction and contains good detail, but if you just want to use and understand the latest version of the app, start here: https://github.com/seanno/shutdownhook/tree/main/dss#readme.

Everyone loves to talk about code, but it’s really databases that run the world. And not nerd-chic graph databases, or vector databases, or object stores, or whatever hyped-up new hoohah that Meta created this month. I’m talking about old school, relational, SQL databases. Every (every) (every!) company uses them. And for good reason — tables and columns and relationships do an excellent job of modeling things and processes in the real world.

To be fair, there are solid specialized use cases for the hoohas too. But SQL is and will remain ubiquitous because it does its job. I’m actually of the opinion that everyone should learn SQL, not just because it’s a work superpower (I’m looking at you, Tracy) but because normalization is a useful strategy for thinking about all kinds of problems. Actually, that’s another article I should write — but not right now.

The “Direct SQL” Problem

Today, I want to share a little tool that fills a technology gap I’ve found at every company I’ve ever been at — an access-controlled, auditable way to directly execute ad hoc SQL queries for data analysis, troubleshooting, workflow support and data repair.

See, in the “normal” course of business SQL is an infrastructure component — not something folks interact with directly. Enterprise applications provide higher-level end-user and system interfaces that use SQL “under the covers” as a way to store and model data. These applications add important controls on top of the raw data store: making sure that access restrictions are enforced, applying business rules to keep things consistent, generating logs to support auditing requirements, and so on. All good stuff.

But inevitably, something happens that requires people to venture under those covers. Maybe the application just has a bug. Or more commonly, something needs to happen in the real world that the application wasn’t designed to handle — updating the shipping address for an order after it was placed, reprocessing a job after a network interruption, that kind of thing. Perhaps somebody needs metrics for a presentation and the data warehouse doesn’t have the right data. There’s always something.

Easy enough for somebody who knows SQL. Open up a database client or your SQL-aware IDE and you’re good to go. But there’s a problem! Every one of the guardrails provided by application logic are gone. No access control, no auditing, no double-checking that your query doesn’t (oops) accidentally delete a bunch of records. Not only that, but direct database connections typically require access to a machine running directly in a production network — another opportunity for things to go wrong.

It’s easy to say that this “shouldn’t” happen, but that’s just naïve, wishful thinking. Best to just acknowledge reality and at least put some controls in place that minimize risk and maximize capability. That’s exactly what SQL Hammer does.

Try it live!

Before we install anything, let’s take a look at SQL Hammer running on my trusty server at https://shutdownapps.duckdns.org:7083/. I’m assuming you know the basics of SQL syntax. If not, this app probably ain’t much use to you anyways. Log in with any GitHub account and you’ll see two entries in the “Connection” dropdown: “EV Counts in Washington State” and “Scratch DB for Demos etc.” First choose the EV connection, which contains information about EV sales in Washington State (sourced from data.wa.gov) in a single table “evs.”

Running Queries

Your account is granted read-only access to this connection, so you can’t run or save arbitrary SQL statements — you can only run queries that others have created and marked as “shared.” This feature makes direct SQL data safely available to less technical folks in your organization. Click on “EV counts by model year,” then the Run button. Magic!

The “Open as URL” button creates a link that can be bookmarked or shared to provide direct (but still authenticated) access to this query. “Save as CSV” downloads the data for further processing in a spreadsheet app like Microsoft Excel.

Queries can also be more dynamic, requiring user-provided input at runtime. Select and run “EV counts by make in year;” you’ll see that the target model year is listed as a parameter in the results area and applied to the query results. The default is 2023, but you can change this and “Refresh” to look at another year. Parameters are super-powerful, especially for shared queries.

Creating Queries

Now choose the “Scratch DB” connection from the dropdown at the top. In this account you’ve been granted full SQL access, which is a little scary for me, because TTP is a real phenomenon. Please behave yourself! Also, the database resets itself overnight so don’t expect your changes to stick around. Anyhoo.

First click the “New” button, then enter and run a query like “select * from stuff.” Give the query a name and click “Save” so you can use it again later. The query will only be visible to you unless you check the “Shared” box, in which case it can be run (but not edited) by anyone with access to the connection.

Add a parameter to your query by inserting a ? in the query itself, and providing a name in the parameters box — for example, “select * from stuff where label like ?” and a parameter “search”. Run the query again and you’ll be prompted to enter a search string — try something like “yo%” to see wildcard matching live and in color. Woot!

The parameters list should be comma-separated, and each entry in that list must have a corresponding ? marker in the query itself. SQL Hammer doesn’t do a lot of verification on this, so it’s easy to create syntax errors if you’re not careful. A parameters entry can have a default value by adding a colon to the name followed by the value. For example, try “insert into stuff values (?,?)” with a parameter list like “label:mylabel,num:0”.

More than just SELECT

You may have noticed that the query above was actually an insert statement, not a select. This is fine! Assuming you’re configured for write access to a connection, you’re free to execute DML or even DDL statements. You can share these queries too, which can be a great way to capture data from users when combined with parameter lists.

Auditing

With most direct SQL solutions, figuring out who ran what query, when, with what parameters, can be nearly (or actually) impossible. Not only is this just a practical business problem, in many cases it can be a legal one — especially for folks working in regulated environments like healthcare or banking. SQL Hammer to the rescue! If a connection has its “log_queries” column set to 1, every query statement and parameters is logged to disk, together with the user email that executed it. Preserve this logfile and you’ll have no problem satisfying your auditors.

Of course, these logs may now include highly sensitive information — so you’ll have to put processes in place to protect them from prying eyes or tampering. A pain, but well-worth the benefits.

Installing SQL Hammer

SQL Hammer is 100% open source; the code is on GitHub and building it yourself is pretty straightforward. I’ll talk more about that later, but first let’s walk through a binary installation. I’ll be using Linux, but it’s all pure Java so no reason you can’t use Windows or a Mac if you prefer (do let me know if you run into any problems).

First make sure you’ve got a JRE v11 or above. You can check this by running “java -version” at the command line. Next, download sqlhammer-v1.zip from GitHub and unzip it to a directory on a machine that has network access to the database(s) you care about.

Execute “run.sh” in this directory, and point a browser on the same machine at https://localhost:3001. You’ll have to approve the self-signed certificate, but from there you should be prompted for a GitHub login and be on your way! Logs will be written to nohup.out in the same directory; that’s a good place to look for errors if something doesn’t work.

Very cool — but to run “for real” you’ll want to be sure your configuration is correct and secure, so read on. Don’t skip this!

SSL Certificates

The default config.json uses a self-signed localhost certificate for HTTPS. You’ll need to update the SSLCertificateX values to point at your own certificate and key files, using the same same format as Apache’s SSLCertificateFile and SSLCertificateKeyFile. If you really, really want to run without HTTPS you can just delete these values, but that’s probably a bad idea. Remember to set the “Port” value as desired as well.

OAuth2 Provider

As you’ve seen, the default is set up to authenticate users with GitHub accounts, using a stub OAuth2 application registered to my account (seanno). For obvious reasons, you’ll want to pick your own provider and application! If you want to use a social login like Google, Facebook, Amazon or GitHub you’ll find instructions in my recent blog post.

If you’d rather use your enterprise login (likely), you’ll have to figure out how to configure an OAuth2 / OpenID Connection application. Set the “Provider” in config.json to “other,” then provide values for ClientID, ClientSecret, AuthURL, TokenURL and possibly Scope. Most providers make this pretty easy; e.g., for Azure AD / Entra there are instructions here and a config.json fragment will look something like this:

"OAuth2": {
  "Provider": "other",
  "ClientId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
  "ClientSecret": "yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy",
  "AuthURL": "https://login.microsoftonline.com/zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz/oauth2/v2.0/authorize",
  "TokenURL": "https://login.microsoftonline.com/ zzzzzzzz-zzzz-zzzz-zzzz-zzzzzzzzzzzz /oauth2/v2.0/token"
}

If you have trouble with this, first take a closer look at my Social Login post, and feel free to ping me if it’s still not working — happy to help if I can.

An important note on this: When SQL Hammer starts up the first time, the first person to log in is given full write access to the Metadata store. This will have happened when you first logged with GitHub above. If your chosen provider is associated with the same email address you use on GitHub, all good. If not, you’ll want to start with a fresh metadata store — just delete the dss.sql file (by default in /tmp) after configuring your provider, then log in to recreate these tables.

Cookie Encryption

SQL Hammer uses the config.json values under “CookieEncrypt” to protect it’s login cookie. Since the default value is public on GitHub, anybody with access to your server could theoretically whip up a fake cookie to get access to your stuff. Not good.

In your installation directory, run “java -cp dss-server-1.0-SNAPSHOT.jar com.shutdownhook.toolbox.Encrypt keygen”. This will output a randomly-generated key; put into the “CookieEncrypt.Key” field and you’re good to go.

Metadata Store Location

The default setup stores the DSS database in /tmp; edit the Sql.ConnectionString value if you want it somewhere else.

(Maybe) add JDBC drivers

By default, SQL Hammer includes JDBC drivers for SQLite, mySQL and PostgreSQL databases. If you use a database like Microsoft SQL Server, Azure SQL or Oracle, you’ll need to download those driver(s) to the same directory where you installed SQL Hammer. Then load the JAR(s) by adding them to the “cp” argument in run.sh, separated with colons. Here’s an example doing this for Azure SQL (mssql-jdbc-12.4.2.jre11.jar):

nohup java \
  -cp dss-server-1.0-SNAPSHOT.jar:mssql-jdbc-12.4.2.jre11.jar \
  com.shutdownhook.dss.server.App \
  config.json &

Finding the right JDBC driver is pretty easy; just search “XYZ jdbc driver download” on Google and you’ll get there in no time.

99% of JDBC drivers now self-register by including a META-INF/services/java.sql.Driver file in their JAR. If for some crazy reason yours does not, you’ll need to add its fully-qualified class name to config.json under Sql.PreloadDrivers, which should be an array of string values.

Managing Connections and Access

I’m particularly fond of this — admin tasks are performed not through a pre-built UX, but by using SQL itself. The first user to log into an installation is set up as the administrator, with access to the “DSS Metadata Store” connection. A few pre-built queries are added as well, but honestly the SQL is pretty simple; three tables drive it all. Just click the “Schema” button to see how they’re put together.

connections contains one row for each configured database connection. “name” is the primary key, just a short unique label. “description” is what shows up in the dropdown, and setting “log_queries” to 1 will cause query text and parameters to be logged for audit purposes. The star is “connection_string,” which contains everything necessary to connect to and authorize a specific database connection. SQL Hammer connection strings typically contain login information and are very sensitive — be careful to restrict access to the metadata store!

You can add connections using the “connections: add new” query or just with a simple insert statement. The harder part is figuring out a connection string that works; a great place to start is this article at Baeldung. A few examples can also go a long way; so here you go:

  • SQLite: jdbc:sqlite:/PATH/TO/FILE
  • mySQL: jdbc:mysql://SERVER:PORT/DATABASE?user=USER&password=PASSWORD (port is usually 3306)
  • PostgreSQL: jdbc:postgresql://SERVER:PORT/DATABASE?user=USER&password=PASSWORD (port is usually 5432)

access holds (not surprisingly) rules about who can access which connections. The “user” field can either be a specific user (i.e., logged-in email address) or a wildcard pattern matched with the like operator. Some useful practical applications of this are “%” for matching any user, and “%@xyz.com” to match all users with email addresses at xyz.com.

In order to execute or save arbitrary queries, a user must match an access row for the relevant connection in which the “can_create” column has the value 1. Without this flag, users can only run existing queries associated with the connection that are marked “shared” (and can’t see the underlying SQL statements).

The built-in queries “connections: grant access” and “connections: remove access” can be used to manipulate the table; they’re just parameterized versions of insert and delete.

You may find it useful to create multiple “connection” rows that reference the same database, associating different queries with each and granting access to different users. For example, you may have some users that are interested in queries related to financial data, while others are focused on user behavior. By segmenting these query sets by connection, you can reduce confusion and better conform to the principle of minimum access. Since folks only see the connections they have access to, navigation is no problem.

queries holds the actual statements for saved queries, including the “is_shared” flag that marks whether users other than the creator can execute them. Mostly this table is managed through the user interface, but there’s no reason you can’t manipulate it directly as well.

Building from source

The SQL Hammer frontend is an SPA built with React; the backend is a set of Java handlers that run in the context of my custom classes on top of the built-in HttpServer. All of this is packaged into a single uber-JAR that runs as a standalone process starting with com.shutdownhook.dss.server.App.

Building the package requires a git client, Java JDK v11+, maven, node and npm, most of which you probably having hanging around anyways:

git clone https://github.com/seanno/shutdownhook.git
cd shutdownhook/dss
./fullbuild.sh

The fullbuild.sh script first builds the React bundle in the client directory (I had to increase node memory with set NODE_OPTIONS=--max-old-space-size=4096) and copies the resulting files into a zip resource under server/src/main/resources. It then builds and locally installs the utility and server classes under ../toolbox, then the server components, and then finally assembles the uber-JAR which lands in server/target/dss-server-1.0-SNAPSHOT.jar. Use this JAR in place of the one in the release directory to run your own build.

If you want to actively play with the user interface, it’s possible to run the React SPA separately from the server. This is particularly nice because it gives you hot-reload. The details are a bit grotty to include here; drop me a note and I’ll get you started. If there’s any kind of broad interest I’ll write it up; just too lazy to do it now!

Quirks and Futures

This is most definitely a “v1” release. It works and works pretty well, at least for my purposes. But the UX is definitely awkward in places — the interplay between “Run” in the editing pane and “Refresh” in the run pane trips me up regularly. I’d like to have more ready-access to the schema information while running queries. The log file isn’t as configurable as it ought to be. “Setup” is a joke. And on and on.

But you have to start somewhere — and I’m hoping that the app will find its way to a few like-minded folks that have shared my pain and acknowledge the need for something like SQL Hammer. Direct SQL is not just inevitable, it’s super-powerful. It can even be an asset to agility and compliance when managed responsibly. I’d love to hear what you think!

The Elon / Twitter Bummer

~ Sean Nolan

Let’s get this out of the way up front: if you’re here expecting more snarky piling on about how stupid Elon Musk is, you’re going to have to get your schadenfreude fix elsewhere. Frankly, I think he’s a genius. A singular individual of our time that most fairly should be compared with Thomas Edison. But the whole Twitter thing really bums me out, because it makes obvious just how easily an unchecked strength can become a stunning downfall. It’s worth a few words; hopefully ones that will add a little bit of thoughtfulness to a mostly empty public “conversation.” We will see.

First let’s review a couple of the things Elon Musk has contributed to the world.

SpaceX

I’ve long been a believer in space exploration, so it’ll be no surprise that I have followed SpaceX since its earliest days. Back in 2002, Musk made a trip to Russia to acquire rockets at a commercially-reasonable price. The Russians basically told him he was an idiot and that it couldn’t be done. ON THE PLANE HOME, he put together a spreadsheet that showed that he could. His own people though he was nuts at first, but he was right. He surrounded himself with experts ranging from amateur to professional. He seeded money to folks and watched what happened. And most importantly he read, and read, and read. To call out just a few specifics he has cited:

… and then he made it happen. Bigtime. SpaceX is still today*** the only American company that can launch people into space. He puts satellites up for about $1,200 per pound (the Shuttle was $30,000). He has used the capability to launch Starlink, bringing the Internet to people and places previously left behind. The scope of what he has done here is stunning. No gimmicky “tourism.” He has never flown himself. He is simply knocking down real problems, one after another, while most others just second-guess from the sidelines.

Think space doesn’t matter? You’re dead wrong, but OK. How about climate change?

Tesla

It seems I can’t go a day without hearing Fleetwood Mac shill Chevy “EVs for Everyone.” Just like every other car company, Chevy would love you for you to believe that this was all their idea, but in reality they (together with all the usual suspects) have been slow-boating electrics since the 1980s. Not so Elon, who first invested in Tesla in 2004, and launched the Roadster in 2009 as CEO — more than a decade ago. We got our Model X in 2018 and it is straight up the best car I’ve ever owned.

What made the difference with Tesla was not new science, but a willingness to buck conventional wisdom as to what was “production ready.” Rather than whine about a lack of charging infrastructure, they designed the Supercharger and deployed enough of them that I’ve comfortably road-tripped the entire west coast of the USA multiple times. For daily use we haven’t even installed a dedicated charger of our own — we do just fine with a standard wall outlet. Software completes the package: we can safely leave our dog in a climate-controlled car; get automatically-recorded video of accidents or attempted theft; watch Netflix in the ferry line; verify that the doors are locked from our phones; ask it to extract itself from a tight parking space. I’m not allowed to take my hands off the wheel quite yet, but the Tesla drives itself way more than I do — stops at lights, changes lanes, you name it.

And sure they’ve been expensive so far, but at a base price of $47k the Model 3 is within striking distance of “normal” cars. It’s not a stretch to say that the EV industry is at a tipping point today directly because of what Elon has accomplished with Tesla over the past thirteen years.

But wait, there’s more. Tesla has used learning from the cars to become an energy company. One of my son’s best friends is kept busy way more than full-time installing Tesla Solar Roofs across the western half of the country. The Powerwall uses software to optimize power management — even automatically “topping up” the charge when severe weather is in the forecast.

I could keep going like this for a long time. And it’s easy for folks to talk about how nobody should be a billionaire or whatever, but he earned his money creating and selling things people want. His start in business was a $28k loan from his dad — a nice advantage to be sure, but turning $28k into $200B (legally) is a pretty good record and doesn’t happen by accident.

So then WTF happened?

In almost every case, Elon’s success has come down to “just doing” things that conventional wisdom said couldn’t be done. But it’s not a Zuckerberg “move fast and break things” vibe. He really listens to the arguments and the experts and the ideas — he is smart enough to understand what he learns — and only then he makes his call. Educated, but not encumbered, by those that have come before him. It’s just damn impressive.

The thing is, though, a key reason he can ignore the preconceptions of others is that he doesn’t have a ton of empathy for them (clinically it seems). Honestly he said it best himself on SNL:

“To anyone who I’ve offended [with my Twitter posts], I just want to say I reinvented electric cars, and I’m sending people to Mars in a rocket ship. Did you think I was also going to be a chill, normal dude?”

It’s funny because it’s true. The same quality that helps him ignore naysayers also keeps him from understanding the positions of folks that attack him (rightly or wrongly). Especially in the anonymous public sphere. I mean, it’s hard for anybody to turn the other cheek online — throw in some mental instability and it’s just not a shocker when he reacts without thinking.

With Twitter, this all just spun wildly out of control. He’s mad that people are mean to him on Twitter, and because he’s the richest guy in the freaking world his answer is to just buy it and kick off the people he doesn’t like. Obviously he regretted this just days after he set it in motion. But he’d gone too far and was legally required to finish what he started. And the real kicker this time is that — in stark contrast to his other ventures — with Twitter he didn’t do his homework. So in practice he’s just another clueless a$$hat money exec who shows up assuming he knows better. But he doesn’t. And that is exactly what we are seeing play out as he flails and reacts, flails and reacts, flails and reacts.

It just makes me really very sad.

I’m not asking you to feel sorry for the richest man in the world. And I’m not making excuses for this complete sh**show; it could have serious negative implications for all of us. It’s just a bummer to watch it unfold. I hope that history is able to remember both sides of the Elon story, because we’ll all still be benefiting from what he’s built long after people forget what a tweet was.

*** Between the time I wrote the first draft of this piece and published it, NASA finally launched the behemoth that is Artemis on its first trip around the moon. They haven’t put humans in the capsule yet, but it does appear we’re going to have a second option. Awesome, but that program feels a bit like a relic from the 80s. I hope it goes ok!