Even with the COVID-19 endgame in sight, it seems likely that at least partial remote work is here to stay. In many ways, the software world was pretty well set up for this already. Between email, instant messaging and video calls, our technical rhythms didn’t change very much. But just like the rest of the planet, the social rhythms that most of us just took for granted took a big hit. I spent a big chunk of 2020 at Adaptive just trying to find ways to keep physically distant team members connected with each other.
Spotify “collaborative playlists” was one of these — we picked a theme and folks would contribute their favorite songs to the list and play along at home while they worked. The two I see in my Spotify account are (the lists are public; please don’t screw them up, but feel free to contribute!):
These were super fun and prompted some great trash talk on Teams. But at least for me, something was missing on the listening side. We weren’t all hearing the same songs at the same time, in the same order, like happens at a concert or on a roadtrip or at a party.
A couple of weeks ago I decided that it’d be fun to build a little app to try and fill that gap. The result is “Shutdown Radio”, a little web app that allows folks to collaborate on YouTube playlists and listen to them in a synchronized way — everybody hears the cowbell at the same time. It’s kind of neat; let’s see how it works.
Taking it for a Spin
Start by pointing a browser at http://shutdownradio.duckdns.org:7777. Enter your name and a channel name you’ve agreed on with your friends (try “country live” for now). There’s no login per se, which obviously would be needed if somebody were to run this service for real, but wasn’t important for my project. Everybody using the same channel name will experience the same playlist more-or-less synchronized in time.
The next page contains an input box to add videos, an embedded YouTube player, and a list of all the videos in the channel. If the channel already has videos, you’ll jump right in. If not, you can add some by submitting any YouTube video URL into the form.
That’s really it. The app keeps track of what the currently-playing video is and when it started, so when people join they jump into the video right where everybody else is. When it’s time to play the next one, newly-added videos get priority and once they’re all played, it’s just a random pick.
The slightly clever piece here is the code that keeps track of time. Buffering, advertisement breaks, and a host of other factors mean not everyone is exactly in sync. The app tries to strike a balance between the best individual and shared listening experiences — not rocket science, but fun to play with.
Why YouTube? Just because they have a really nice scriptable web player. The old Windows playbook still rings true today: platforms win in large part on developer experience. Just wish they’d remember that at Microsoft…
A Brief Rant, Skip if You Like
Any app like this needs a little server-side code and a basic datastore. I thought, hey, I’ll use Azure since folks might find that interesting. I’d been building Azure functions recently at Adaptive, and a buddy of mine runs the engineering team for Cosmos, so off I went.
It’s pretty much impossible to use this stuff without client libraries, so I fired up the azure-functions-archetype as described in the official quickstart to set up a Maven project, then added the azure-identity and azure-cosmos artifacts to my pom. A quick build to make sure it’s all good and…
… screens and screens of output later, I’d downloaded 1,095 dependent artifacts. A THOUSAND. For “Hello World”! A little shellshocked, I at least took some solace in the fact that the build succeeded. So I added the code to connect to Cosmos, deployed it up to the cloud and….
2021-03-19T21:31:20.736 [Error] Executed 'Functions.next' (Failed, Id=a969286b-91d4-459a-a4a7-88983bc84394, Duration=2772ms)Result: FailureException: NoSuchMethodError: 'reactor.netty.resources.ConnectionProvider reactor.netty.resources.ConnectionProvider.fixed(java.lang.String, int, long, java.time.Duration)'Stack: java.lang.reflect.InvocationTargetExceptionat java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)at ...
Because I promised a brief rant, I will skip over a few hours of soul-sucking “coding by Google” — it turns out that the current Cosmos client library and Azure functions just straight up don’t work together. Color me shocked that an app with more than a thousand dependencies has a conflict between two of them. Supposedly there will be a new Cosmos SDK release by the end of March that fixes this.
For now, let’s just do this the old-fashioned way, shall we?
Let’s Build an App
There are a million opinions on how to design an app. Thirty-four years ago Professor Sewelson taught me about Abstract Data Types using this classic text — and nobody has come up with anything better yet. ADTs (not matter what you decide to call them) are the root of every good design — what are our objects, how do they relate to each other, and what actions do we need to perform using them? For Shutdown Radio, the objects defined in Models.java are quite simple:
- A
Videoreferences a video on YouTube by Id and caches a bit of metadata. - A
Playlistis a list of Video, also with a little metadata. - A
Channelis a Playlist and information about the currently-playing Video.
Our actions in Radio.java are similarly straightforward:
- Fetch the next video to play in a channel.
- Fetch the playlist for a channel.
- Add a video to a channel’s playlist.
If you notice a little nod to actions in the model, good eyes! A more natural Channel object would embed the Playlist, but our pattern of actions, together with our likely store implementation, nudges us to keep them separate. Good engineering is all about balance — model and actions need to be independent from but informed by likely implementation. That knife edge is exactly where our craft lives.
Store
Gotta keep stuff somewhere. All of our actions root at a single channel, so a document store with JSON representation makes good sense. As much as I hate to admit that anything created in the last twenty years has merit, these two technologies used well really have been game changing, both for performance and code clarity.
So wait, does Store.java really just keep these documents in a directory on a file system? Indeed! The granddaddy of document stores still has a lot to offer; just add a little synchronization to ensure integrity and we’re off to the races. The only downside here comes with scale — the implementation effectively limits us to running a single web process, both because our synchronization is in-memory and because network-attached file systems are usually pretty slow. Another learning moment!
Using a simple approach like this is often exactly the right thing to do when you’re early in a project, not just in prototypes but in production too. The code is simple and easy to debug, and will more than handle the task for some time. The trick is to design an abstraction that minimizes switching costs if/when that time comes. (Of course the flip-side trick is to recognize when to go big right away; there’s just no substitute for experience when making these calls.)
Back to JSON for a moment. It is super nice to be able to define our model as simple objects and then easily manage serialization both in the document store and over the network. My favorite library for dealing with this is Google’s well-contained Gson library. Yes, dependencies suck, and it’s weird that Java doesn’t have good native support for something some so common and useful, but it’s hard to throw rocks at Gson as a solid option.
Server
Without the Azure Functions infrastructure, we also need a web server. We could certainly use one of the zillions of web frameworks out there, but who needs that headache? HttpServer is built into the JDK, is quite serviceable and performs well … I’m in.
The basic pattern here is to instantiate an HttpServer, configure it with an ExecutorService that manages requests (remember to shutdown cleanly!), and implement “Handlers” that correspond to incoming URLs. The setup is in Main.java and our Handlers live in Handlers.java. A base Handler implementation wraps up common stuff like exception handling and query string management, making specific Handler implementations pretty straightforward. It would get a little more complicated if our responses were large enough to merit streaming — but they aren’t — so hooray for simplicity.
I wish more people would consider this approach to serving web requests. The frameworks out there are truly impressive, and I would never implement a large-scale app without picking one. But for small, well-contained APIs, there’s a lot to be said for avoiding the sprawl. Just watch out for obvious security holes, like the file handling in Radio.getStaticHtml and Store.getCleanName.
Odds and Ends
Browser-side code is just not my thing. OK I said it. I’ve had the luxury of working with some incredible engineers (yo Ivin) and designers (yo Isabel) that are artists in the space of layout differences and sloppy Javascript frameworks and all that noise. For me, JQuery is about as far as I like to go, and I even still use tables for layout sometimes. So there.
All that notwithstanding, this app needs a little client-side love, which you’ll see in playlist.html. The bulk of this file is just management of the embedded YouTube player and a few async calls back to the server.
YouTube really has done a great job with that player, which makes it all the more annoying that their data API is such a pain. When a user adds a new video, we need to grab its title and duration, which seems like it should be super-easy, but no. It’s not the API itself, it’s the overwrought authorization model behind it. Don’t get me wrong — OAuth is a fantastic technology when it’s appropriate and I wish it had been mainstream when we were building HealthVault — but for simple server-side use cases like this it’s just dumb.
I don’t like dumb, so in YouTube.java I opted to hit the video’s HTML page and pick out what I need from the elements there. This kind of hack-and-grab can really go down the rabbit hole, but I tried to keep it simple using the native HttpUrlConnection and some simple string search stuff … regex might have been marginally more efficient, but down that path lies madness.
The Clever Bits
The only really notable algorithmic stuff in this whole app is the implementation of Radio.updateIfNeeded and the corresponding Javascript. This is where we try to keep the players in sync. In short, updateIfNeeded looks like this:
- If we’re already playing a video and we’re still within its runtime, just return it, all good.
- Otherwise, pick the next video to play.
- If the playlist has any unplayed videos (i.e., recently added), pick the oldest of them.
- Otherwise pick a random video, excluding the last one (so we don’t obviously repeat).
“Still within its runtime” is the first interesting bit. We store the length of the video in seconds, and the Instant that we first picked it, ignoring clock drift because all of the timestamps are allocated at the server. We do trim a couple of seconds from the video length, which is just slop to cover up for the truncation of nanos from the Instant values.
Ultimately the client receives the current video id and the timestamp that it “started” at the server. The second interesting bit is here — the script must decide whether it should start the video from the beginning or jump to the current location.
This is a little trickier than it first appears. Obviously to keep players in sync, we’d always jump to the current location, which would most accurately correct for buffering and other lag issues across the listening population. But that experience would be super-annoying; when I listen to the radio I want to hear whole songs, not weird fragments.
The compromise I chose is this: for the first video, jump to the current location — this mirrors turning on a real-world radio, and doesn’t feel weird. For each subsequent video, start it at the beginning unless we’ve fallen more than 20 seconds behind, in which case we skip forward. In practice this seems to work out quite well, but might benefit from some more tuning.
What Next?
Modulo the annoying Azure side trip, this was a super-fun little app to build, and frankly it seems to work pretty well. I haven’t done loads of testing, but a few days with a bunch of different (muted!) browsers flushed out the worst of the bugs. I would love it if you gave it a shot with your friends or team, and assuming traffic doesn’t get ridiculous will happily leave it running at http://shutdownradio.duckdns.org:7777. (Can I get a w00t for Linux/Java uptime?)
Even better would be for you to fork the code and make it your own! For example, the shared experience would be way better if you could see who else was listening. This presents its own interesting problem vis-a-vis abandoned sessions, but no rocket science there.
For me, I’ll probably come back and reimplement Shutdown Radio on Azure sometime soon, just to close that loop. As much as I love my little self-contained app, serverless options are most definitely the future of API development. I just hope the cloud vendors wake up and remember that developers make their bacon. Until next time!
Added Feb 28, 2022 — finally did the Azure bit — read about it here!