TLDR, drag this link to your bookmarks bar: explain. If you select medical-related text on any page and then click the link, it will open up an “explain” window with an AI-driven translation. Alternatively, you can just visit the site at https://explainmynotes.azurewebsites.net/.
Every day I seem to get just a little bit older. My folks too. Not a bad thing, but it does inevitably mean more time spent trying to navigate the clown show that is American Healthcare. Sometimes things are simple, and sometimes they’re little mystery dramas with House or Doc Martin trying to figure out what’s going on.
By now, most of us have gotten used to using patient portals like MyChart to keep track of our care at various providers. Lab results and clinical notes show up in near real time and — thanks to years of policy pressure — are quite comprehensive. (to wit: when I had appendicitis earlier this year, my wife at home texted me the diagnosis before anybody in the ER came to let me know what was up.)
Access to these primary sources is invaluable. But clinical notes are also full of jargon, shorthand, codes and concepts that very few of us understand. Take for example this short snippet from the surgery notes of my appendicitis visit:
I attempted to bring omentum to sit over the anastomosis, but the omentum was fairly short and there was no easy reach.
I read this shortly after coming out of anesthesia — something she tried to do didn’t work. Is that bad? Should I be concerned? Luckly, Dr. M is pretty awesome; she explained that the omentum is a layer of fat that protects organs in the abdomen, and they like to “drape” it over surgical sites to aid healing by enhancing local blood flow. Somehow despite my notable beer belly I didn’t have enough fat to make this work, but it’s not a big deal. Case closed.
Unfortunately, not every provider is a great communicator like Dr. M. And even when they are, appointments are so short and far-between that there’s rarely a good opportunity for questions like this. That’s why I wrote explain my notes.
Clinical Notes, explained by AI
explain my notes takes advantage of two pretty neat technologies: SMART on FHIR for data access and ChatGPT for helping to interpret the notes. Currently it’s set up to connect to providers using Epic MyChart. In a nutshell, it works like this:
Visit the site, read the terms of use and pick your provider.
Log in at the provider’s patient portal and approve the connection.
Pick an encounter to see a list of associated documents.
Pick a document to view its contents.
Select any text in the document and choose “Explain Selection” to pop up a window that shows the original and “explained” text side by side:
And that’s it! You’ll need to authorize the app each time you use it, because Epic doesn’t permit long-lived tokens for “automatic download” patient applications. Ah well.
Caveat 1: The ChatGPT API isn’t free — if I’m surprised and the app gets a lot of direct use, I may have to figure out how to offset those costs. For now I just hope folks try it and that it’s (a) helpful and (b) inspires others to build on the idea.
I’ve already written a bunch about SMART and why I think it’s so valuable, so I won’t repeat myself here. But this is the first time I’ve written a SMART app for patients, and there were a few interesting nuggets worth a mention:
Standalone Launch
explain my notes uses the “standalone launch” model. With a provider app, a huge part of the benefit comes from living within the context of the EHR — it gets you single sign-on and provider/patient context and feels seamless in an environment where providers are already spending much of their day. It’s not the same for patients; a dedicated site that can explain its function and then “connect to” the portal makes good sense.
Epic Automatic Download
The super-cool thing about patient-facing apps is that you don’t need to “register” them with each individual EHR. Instead, the EHR vendors maintain provider lists and automatically enable connections when authorized by the patient. It’s hard to overestimate just how great this is — back in the day, we had to arrange to connect HealthVault to each and every provider that wanted to work with us.
Careful, though! Automatic download comes with conditions, and they are not immediately obvious (Epic’s conditions are documented behind a free login). “Refresh” tokens aren’t allowed; only certain data types can be accessed; no “write” operations are permitted, etc. My first cut at the app didn’t meet the criteria exactly, and it took me awhile to figure out what was going on.
PDF and CCDA Content
Many notes are stored as HTML or text. Encounter summaries, though, are often stored in “CCDA” format — an old-school XML standard. XML needs to be translated into HTML for display in a browser, and while there is some solid open source code for doing that, the generated HTML doesn’t always display nicely within a larger web page. I was able to tweak it for my purposes; the altered stylesheet is available per the original’s open-source license terms.
PDF content was also a challenge to display so that it both (a) looks correct and (b) makes the selection available for sending to ChatGPT. I ended up doing a server-side translation using pdftohtml, an old standby that still works surprisingly well.
Explaining notes: ChatGPT
I think it’s clear that generative AI is going to be a seriously Big Deal — combustion engine and Internet big. But it’s still very early days, and it’s hard not to be annoyed by the seemingly endless garbage “applications” being churned out by hype-riding VC-funded bros. I get that — but bear with me.
Generative AI (specifically ChatGPT for us) is pretty amazing if you think about it as your well-read, smart, eager-to-please friend without any formal training and a fear of being wrong. People like this are super-useful, because they’ve probably come across information that you haven’t, and can be great “translators” of jargon and other specialty content. You just have to take what they say with a grain of salt — a little fact-checking goes a long way.
The ChatGPT “completions” API is pretty simple — it takes an array of input/questions and returns answers in markdown format. There are a few knobs you can turn, but that’s basically it. “Prompt engineering” is a weird concept, much closer to social engineering than code. The current “setup” prompt for explain my notes is this:
You are a medical professional that explains clinical notes and other medical text using terms and language that an average American adult without medical training will understand. Minimize the use of jargon. Your responses should not be notably longer than the original text. Also please include up to three Google search links targeting the key topics you find.
The “Google search links” part here is the most interesting. I initially asked the system to return “up to five links that would be helpful for further research,” but it turns out that ChatGPT is terrible at this, and is actually known for simply making up gibberish URLs. I’m not sure why this is the case; apologists claim they’re just stale links from old training data, but it’s way more than that. Restricting the links to Google searches seems to work pretty well.
And I guess that’s it for now! Please give the app a try — good test data is hard to come by and so I’d appreciate any and all feedback or bug reports. Until next time…
This is article three of a series of three. The first two are here and here.
Last time here on the big show, we dug into SMART Health Cards — little bundles of health information that can be provably verified and easily shared using files or QR codes. SHCs are great technology and a building block for some fantastic use cases. But we also called out a few limitations, most urgently a ceiling on QR code size that makes it impractical to share anything but pretty basic stuff. Never fear, there’s a related technology that takes care of that, and adds some great additional features at the same time: SMART Health Links. Let’s check them out.
The Big Picture
Just like SMART Health Cards (SHCs) are represented by encoded strings prefixed with shc:/, SMART Health Links (SHLs) are encoded strings prefixed with shlink:/ — but that’s pretty much where the similarity ends. A SHC is health information; a SHL packages health information in a format that can be securely shared. This can be a bit confusing, because often a SHL holds exactly one SHC, so we get sloppy and talk about them interchangeably, but they are very different things.
The encrypted string behind a shlink:/ (the “payload”) is a base64url-encoded JSON object. We’ll dive in way deeper than this, but the view from 10,000 feet is:
The payload contains (a) an HTTPS link to an unencrypted manifest file and (b) a key that will be used later to decrypt stuff.
The manifest contains a list of files that make up the SHL contents. Each file can be a SHC, a FHIR resource, or an access token that can be used to make live FHIR requests. We’ll talk about this last one later, but for now just think of a manifest as a list of files.
Each file can be decrypted using the key from the original SHL payload.
There’s a lot going on here! And this is just the base case; there are a bunch of different options and obligations. But if you remember the basics (shlink:/, payload, manifest, content) you’ll be able to keep your bearings as we get into the details.
Privacy and Security
In that first diagram, nothing limits who can see the manifest and encrypted content — they’re basically open on the web. But all that is basically meaningless without access to the decryption key from the payload, so don’t panic. It just means that, exactly like a SHC, security in the base case is up to the person that’s holding the SHL itself (in the form of a QR Code or whatever). And often that’s perfectly fine.
Except sometimes it’s not, so SHLs support added protection using an optional passcode that gates access to the manifest:
A user receiving a SHL also is given a passcode. The passcode is not found anywhere in the SHL itself (although a “P” flag is added to the payload as a UX hint).
When presenting the SHL, the user also (separately) provides the passcode.
The receiving system sends the passcode along with the manifest request, which succeeds only if the passcode matches correctly.
Simple but effective. It remains to be seen which use cases will rally around a passcode requirement — but it’s a handy arrow to have in the quiver.
The SHL protocol also defines a bunch of additional requirements to help mitigate the risk of all these (albeit encrypted and/or otherwise protected) files floating around:
Manifest URLs are required to include 256 bits of entropy — that is, they can’t be guessable.
Manifests with passcodes are required to maintain and enforce a lifetime cap on the number of times an invalid passcode is provided before the SHL is disabled.
Content URLs are required to expire (at most) one hour after generation.
(Optionally) SHLs can be set to expire, with a hint to this expiration time available in the payload.
These all make sense … but they do make publishing and hosting SHLs kind of complicated. While content files can be served from “simple” services like AWS buckets or Azure containers, manifests really need to be managed dynamically with a stateful store to keep track of things like passcodes and failed attempts. Don’t think this is going to be a one night project!
SMART Health Links in Action
Let’s look at some real code. First we’ll run a quick end-to-end to get the lay of the land. SHLServer is a standalone, Java-based web server that knows how to create SHLs and serve them up. Build and run it yourself like this (you’ll need a system with mvn and a JDK installed):
git clone https://github.com/seanno/shutdownhook.git
cd shutdownhook/toolbox
mvn clean package install
cd ../shl
mvn clean package
cd demo
./run-demo.sh # or use run-demo.cmd on Windows
This will start your server running on https://localhost:7071 … hope it worked! Next open up a new shell in the same directory and run node create-link.js (you’ll want node v18+). You’ll see an annoying cert warning (sorry, the demo is using a self-signed cert) and then a big fat URL. That’s your SHL, woo hoo! Select the whole thing and then paste it into a browser. If you peek into create-link.js you’ll see the parameters we used to create the SHL, including the passcode “fancy-passcode”. Type that into the box that comes up and …. magic! You should see something very much like the image below. The link we created has both a SHC and a raw FHIR bundle; you can flip between them with the dropdown that says “Health Information”.
So what happened here? When we ran create-link.js, it posted a JSON body to the server’s /createLink endpoint. The JSON set a passcode and an expiration time for the link, and most importantly included our SHC and FHIR files as base64url-encoded strings. SHLServer generated an encryption key, encrypted the files, stored a bunch of metadata in a SQLite database, and generated a SHL “payload” — which looks something like this:
(You can make one of these for yourself by running create.js rather than create-link.js.) Finally, that JSON is encoded with base64url, the shlink:/ protocol tag is added to the front, and then a configured “viewer URL” is added to the front of that.
The viewer URL is optional — apps that know what SHLs are will work correctly with just the shlink:/… part, but by adding that prefix anybody can simply click the link to get a default browser experience. In our case we’ve configured it with https://shcwork.z22.web.core.windows.net/shlink.html, which opens up a generic viewer we’re building at TCP. That URL is just my development server, so handy for demo purposes, but please don’t use it for anything in production!
Anyways, whichever viewer receives the SHL, it decodes the payload back to JSON, issues a POST to fetch the manifest URL it finds inside, pulls the file contents out of that response either directly (.embedded) or indirectly (.location), decrypts it using the key from the payload, and renders the final results. You can see all of this at work in the TCP viewer app. Woot!
A Quick Tour of SHLServer
OK, time for some code. SHLServeris actually a pretty complete implementation of the specification, and could probably even perform pretty reasonably at scale. It’s MIT-licensed code, so feel free to take it and use it as-is or as part of your own solutions however you like, no attribution required. But I really wrote it to help folks understand the nuances of the spec, so let’s take a quick tour.
Because the manifest format doesn’t include a way to identify specific files, the admin methods expect the caller to provide a “manifestUniqueName” for each one. This can be used later to delete or update files — as the name implies, they only need to be unique within each SHL instance, not globally.
The last interesting feature of the class is that it can operate in either “trusted” or “untrusted” mode. That is, the caller can either provide the files as cleartext and ask the server to allocate a key and encrypt them, or it can pre-encrypt them prior to upload. Using the second option means that the server never has access to keys or personal information, which has obvious benefits. But it does mean the caller has to know how to encrypt stuff and “fix up” the payloads it gets back from the server.
The bottom layer of code is SHLStore.java, which just ferries data in semi-ORM style between a Sqlite database and file store. Not much exciting there, although I do have a soft spot for Sqlite and the functional interface I built a year or so ago in SqlStore.java. Enough said.
Anatomy of a Payload
OK, let’s look a little more closely at the payload format that is base64url-encoded to make up the shlink:/ itself. As always it’s just a bit of JSON, with the following fields:
urlidentifies the manifest URL which holds the list of SHL files. Because they’re burned into the payload, manifest URLs are expected to be stable, but include some randomness to prevent them from being guessable. Our server implements a “makeId” function for this that we use in a few different places.
keyis the shared symmetric key used to encrypt and decrypt the content files listed in the manifest. The same key is used for every file in the SHL.
labelis a short string that describes the contents of the SHL at a high level. This is just a UX hint as well.
v is a version number, assumed to be “1” if not present.
flagsis a string of optional upper-case characters that define additional behavior:
“P” indicates that access to the SHL requires a passcode. The passcode itself is kept with the SHL hoster, not the SHL itself. It is communicated to the SHL holder and from the holder to a recipient out of band (e.g., verbally). The flag itself is just another UX hint; the SHL hoster is responsible for enforcement.
“L” indicates that this SHL is intended for long-term use, and the contents of the files inside of it may change over time. For example, a SHL that represents a vaccination history might use this flag and update the contents each time a new vaccine is administered. The flag indicates that it’s acceptable to poll for new data periodically; the spec describes use of the Retry-After header to help in this back-and-forth.
One last flag (“U”) supports the narrow but common use case in which a single file (typically a SHC) is being transferred without a passcode, but the data itself is too large for a usable QR code. In this case the url field is interpreted not as a manifest file but as a single encrypted content file. This option simplifies hosting — the encrypted files can be served by any open, static web server with no dynamic manifest code involved. The TCP viewer supports the U flag, but SHLServer doesn’t generate them.
Note that if you’re paying attention, you’ll see that SHLServer returns another field in the payload: _manifestId. This is not part of the spec, but it’s legal because the spec requires consumers to expect and ignore fields they do not understand. Adding it to the payload simply makes it easier for users of the administration API to refer to the new manifest later (e.g., in a call to upsertFile).
Working with the Manifest
After a viewer decodes the payload, the next step is to issue a POST request for the URL found inside. POST is used so that additional data can be sent without leaking information into server logs:
recipientis a string representing the viewer making the call. For example, this might be something like “Overlake Hospital, Bellevue WA, registration desk.” It is required, but need not be machine-understandable. Just something that can be logged to get a sense of where SHLs are being used.
passcode is (if the P flag is present) the passcode as received out-of-band from the SHL holder.
embeddedLengthMaxis an optional value indicating the maximum size a file can be for direct inclusion in the manifest. More on this in a second.
The SHL hoster uses the incoming manifest request URL to find the appropriate manifest (e.g., in our case https://localhost:7071/manifest/XruV__8k1Zn68NK1lsLH05ZmONtaUC85jmAW4zEHoTA), then puts together a JSON object listing the content files that make up the SHL. The object contains a single “files” array, each element of which contains:
contentType, typically one of application/smart-health-card for a SHC or application/fhir+json for a FHIR resource (I promise we’ll cover application/smart-api-access before we’re done).
Directly, using an embedded field within the manifest JSON.
Indirectly, as referenced by a location field within the manifest JSON.
This is where embeddedLinkMax comes into play. It’s kind of a hassle and I’m not sure it’s worth it, but not my call. Basically, if embeddedLengthMax is not present OR if the size of a file is <= its value, the embedded option may be used. Otherwise, a new, short-lived, unprotected URL representing the content should be allocated and placed into location. Location URLs must expire after no more than one hour, and may be disabled after a single fetch. The intended end result is that the manifest and its files are considered a single unit, even if they’re downloaded independently. All good, but it does make for some non-trivial implementation complexity (SHLServer uses a “urls” table to keep track; cloud-native implementations can use pre-signed URLs with expiration timestamps).
* Note I have run into compatibility issues with encryption/decryption. In particular the specification requires direct encryption using A256GCM, which seems simple enough. But A256GCM requires a 12-byte initialization vector, and there are libraries (like python-jose at the time of this writing) that mistakenly use 16. Which might seem ok because it “works”, but some compliant libraries (like javascript jose) error out when they see the longer IV and won’t proceed. Ah, compatibility.
SMART API Access
OK I’ve put this off long enough — it’s a super-cool feature, but messes with my narrative a bit, so I’ve saved it for its own section.
In addition to static or periodically-updated data files, SHLs support the ability to share “live” authenticated FHIR connections. For example, say I’m travelling to an out-of-state hospital for a procedure, and my primary care provider wants to monitor my recovery. The hospital could issue me a SHL that permits the bearer to make live queries into my record. There are of course other ways to do this, but the convenience of sharing access using a simple link or QR code might be super-handy.
A SHL supports this by including an encrypted file with the content type application/smart-api-access. The file itself is a SMART Access Token Response with an additional aud element that identifies the FHIR endpoint (and possibly some hints about useful / authorized queries). No muss, no fuss.
The spec talks about some other types of “dynamic” exchange using SHLs as well. They’re all credible and potentially useful, but frankly a bit speculative. IMNSHO, let’s lock down the more simple file-sharing scenarios before we get too far out over our skis here.
And that’s it!
OK, that’s a wrap on our little journey through the emerging world of SMART Health Cards and Links. I hope it’s been useful — please take the code, make it your own, and let me know if (when) you find bugs or have ideas to make it better. Maybe this time we’ll actually make a dent in the health information exchange clown show!
