Z-What? Rescuing my Z-Wave smart lights.

We’re definitely in the “chaotic innovation” period of smart home stuff. Embedded technology and cloud connectivity are advanced enough to support actually useful smart lights, vacuums, thermostats, cameras, washers, dryers, and a ton of other devices. And we’re starting to see an early veneer of consistency, as most can forge some kind of a connection with Alexa or Google Home or whatever. But under the covers, things are pretty insane. Companies enter and leave the market constantly, and each seems to require a new account with a new password, maybe another hardware gateway, and probably a recurring subscription fee.

The pace of innovation is actually super-exciting and fun … it just isn’t tidy, as evidenced by the growing stack of dead-end devices in my closet. Mostly that’s just part of the process. But what really kills me is companies that pull the bait-and-switch, like Wink did last year when they abruptly starting charging monthly for functionality that had been sold as a one-time purchase. This is the height of shoddy business — your inability to do basic math on costs is not an excuse to renege on promises made. Especially when those promises were instrumental in growing your user base against other (more honest) companies. Not that I’m bitter or anything.

In any case, that move left me with a house full of smart lights that, just like the humans and dogs that live here, refused to listen to me. Which became the impetus for a deep dive into smart home technology that turned out to be a ton of fun and generated some useful and/or interesting code. Let’s check it out.

First, the Network

It seems inevitable that when all this settles out, all of our smart devices are just going to connect via wifi like everything else. This is already pretty much the case for “high end” stuff like washers and cameras, but wifi has mostly been considered too expensive for low-margin devices like lightbulbs, and too power-hungry for battery-powered stuff. While both are really non-issues at this point, it takes a while for the consumer device manufacturers to catch up.

So at least as of today, there are tons of devices out in the wild that are not wifi, so what do they use? One option is Bluetooth, but that’s pretty rare and generally sucks just like Bluetooth always sucks. I have a BT-based August door lock and it’s just dumb. You can add a wifi gateway (and I have) but that just adds more money and more complexity to a unit that eats batteries like Cookie Monster eats cookies anyways. Nope.

This leaves two technologies that were designed ground-up for smart home use: Zigbee and Z-Wave. This is very much a Coke vs. Pepsi kind of thing — there are a few differences, but for the most part they’re the same thing solving the same problems:

• Both require a “gateway” to coordinate communication on the network. This is typically a dedicated hardware unit that keeps an inventory of devices, sends them commands, and receives status updates. The gateway is also the face of the Z* network to the outside world — usually over wifi, either using an embedded webserver or a direct connection to a cloud service, or both.
• Both are “mesh networks,” which is quite handy in the home environment. Z* signals can only travel about 20-30 meters, but each node acts as a “relay” to pass messages along. So if device “A” is 20 meters from your gateway, and device “B” is another 20 meters farther out, “A” can serve as an intermediary to relay messages between the gateway and device “B”. Lighting devices are particularly well-suited to this kind of network, since bulbs tend to be evenly distributed around the house.

The differences aren’t super-important. Zigbee enables more hops between nodes and more devices on the network; Z-Wave has a longer reach between nodes and is cheaper. Amazon’s Echo Plus has a built-in Zigbee gateway, so that’s nice. But either one is perfectly serviceable; just make sure you’re buying devices that match the technology in your gateway! There are a few gateways that contains chips for both protocols if you really really want to run both.

Z-Wave it is … hello RaZberry!

I didn’t really know any of this when Wink dropped the subscription bomb, but I figured there had to be a simple way to make my lights start listening again. My first step was to get out the ladder and look at the lights to figure out who made them and what was inside. It turns out that most of my lights are made by GoControl — pretty neat units that slide right into an existing 6-inch ceiling can. Most importantly, that little Z-Wave logo told me where to start!

A couple of hours of research made it clear that (a) Aeotec probably makes the best ready-to-go Z-Wave compatible hub available at the moment, but also that (b) the market is super-unsettled with companies entering and leaving almost every day. Not a lot of stability. I also didn’t see any control software that I wanted to get invested in — just a bunch of complex user interfaces designed by engineers.

Plan B — Could I take control of my destiny by getting a little closer to the metal and building something myself? It turns out that the answer is yes; the RaZberry daughter board for Raspberry Pi delivers excellent Z-Wave capability and is fully programmable. Shiny! Now, full disclosure for those of us that worry about certain state actors: RaZberry is manufactured by Z-Wave.me, a company out of Russia funded by the non-profit Skolkovo Foundation. From everything I see, they have been extraordinarily open and explicit about how the hardware works and what it does / does not share with its cloud service (for starters see their privacy policy). I was able to get comfortable here; your mileage may vary.

The RaZberry also comes with a license for Z-Way, their software stack that includes everything from a low-level C API up through a user-facing home control cloud service. Getting this all built out was my first order of business and was pretty straightforward:

1. Set up the Pi. I used a 3B, but the RaZberry is compatible with all models that have the header block. I splurged on an official Pi3 case; the daughter board fits inside just fine.
2. I used Pi Imager to set up the SD card as per the usual, but picked up two nice tricks (hat tip) that meant I could do the setup completely headless. First, add an empty file named “ssh” to the root of the SD card after using the imager — this tells the OS to start the SSH daemon by default. Second, pre-configure your wifi details by adding a text file “wpa_supplicant.conf” to the root of the card as well; this link describes how to set up the contents for your network. SO much better than having to hook up a monitor and keyboard.
3. Install the Z-Way software by downloading and running the install script: “wget -q -O - https://storage.z-wave.me/RaspbianInstall | sudo bash” (details here).
4. Find the Pi’s IP address with “ifconfig | grep inet” and browse on over to http://IPADDRESS:8083 on your local network to set up passwords and such. That’s it! Your Pi is now running as the gateway node of a brand new Z-Wave network.

Set up devices with the Z-Way interface

My snide comments about engineer-designed interfaces aside, the web-based “Z-Way Smart Home Interface” really does provide a ton of functionality: device management, diagnostics, device control, automation of complex scenarios, integration with Alexa, and a ton more. You could 100% stop reading this article now and just use Z-Way as your smart home ux and be pretty happy.

If you want to do this and also access your gateway from outside of your local network, you’ll need to use the Z-Way cloud service at https://find.z-wave.me. Log on to the local browser interface and navigate to the Settings / Remote Access section. Make sure that the “Enable Remote Access” checkbox is checked and take note of your “Access ID” number. When you log in at https://find.z-wave.me, you’ll enter your login as this Access ID, a slash, and then your user name; e.g., “123456/admin”. This public URL just serves as a proxy to your local gateway to provide access from anywhere. As I mentioned above, you’ll have to decide for yourself how comfortable you are with using the Z-Wave.me cloud services. In addition to remote access to your gateway, there are features to enable remote support and backup of your configuration to the cloud. All of this can be disabled if you want to keep your data entirely local.

All well and good, but the point of this whole exercise was to avoid being dependent on a third-party service that could go away at any time. So I’m avoiding the cloud and only using the Z-Way app locally for network administration and troubleshooting. Most importantly I use it to “include” devices in the network.

This part is kind of like Bluetooth pairing. You tell the gateway to enter “inclusion” mode, and then tell your device to do the same. When they each notice each other, an association is formed, and the device is registered as part of the network.

1. Make sure the device is not already part of another network. If you just bought it, all good. If not, dig out the manual and figure out how to “reset” it. For my GoControl devices, you do this by turning the power on and off four times — the lights blink twice to show they’ve been reset.
2. Choose “Devices” from the Z-Way top-right menu, then “Add new” next to Z-Wave in the top row, then “Add new Z-Wave Device and identity it automatically” at the top, and finally “Start” to put the gateway into inclusion mode.
3. Put the device into inclusion mode, again according to the manufacturer’s instructions. For my lights that just meant turning them off and on again. If all goes well, the gateway and the device will see each other and live happily ever after. My lights acknowledge this by blinking twice.

Two little gotchas in this process. First, each time you run the process above, only one device will be included. If you have a single switch that controls multiple bulbs, it’s a little messy. Step #1 will reset all the bulbs at once, but you’ll have to repeat steps 2 and 3 multiple times until all the bulbs in the group have been included. Don’t worry, we’ll set things up so you can easily turn the whole group on or off together later on. If you’re starting from scratch, though, using a Z-Wave switch (many options, I’m using this one) and regular bulbs may be simpler.

Also, be aware that inclusion (and exclusion) can be quite sensitive to proximity. The “mesh” part of the network doesn’t seem to apply during this process, so the gateway needs to be quite close to the device you want to include. If you go through the steps above and nothing happens, try moving your gateway temporarily closer to the device. When I’m adding devices, I actually plug the gateway into a portable power station so I can easily carry it around the house. (We bought the power station for power outages, not Z-Wave maintenance, but it’s a nice bonus!)

APIs Everywhere

OK, first let me say that there is a LOT going on in this protocol, and about a million ways to dig in. My development scenario is really straightforward — manage a bunch of dimmable lights — so I’ve chosen a pretty simple, high-level approach. My bet is that for most folks using the RaZberry, starting from what I’ve done here is going to be the least insane choice, but just for the record:

1. Z-Wave is currently (as of 2018) owned by Silicon Labs, which publishes an official Z-Wave SDK. It’s primarily meant for device implementers, but there’s a bunch of other stuff in there too.
2. OpenZWave is a fully open-source C++/.NET library that is the basis for a bunch of other stuff.
3. Z-Wave JS is another open source set of libraries and apps, including zwavejs2mqtt which seems quite popular.
4. Z-Way itself has a bazillion different ways to work with it. I tried to summarize them all, but gave up — read all about it in their full documentation.

Some of the cacophony here is just history; the IP rights for Z-Wave have changed hands a million times and it shows. But more than that, the protocol is just complicated. Commands sent to the gateway go onto a queue and are marshalled out onto the wire when bandwidth is available. Responses, if any, come back from nodes asynchronously and without any strong binding to the original command. Early on there was some patent issue that meant nodes couldn’t proactively update the gateway with status changes, putting the onus on developers to figure out if data they received was up-to-date or stale. And all this on top of a protocol that is just trying to encompass a TON of diveres device types. It’s a lot.

Anyways, with my simple scenario I was hoping to find a simple API to work with. It turns out that the Z-Way “VDev” (virtual device) API fits that bill perfectly by (a) providing a normalized and simplified view of the device set; and (b) exposing a small, standard set of REST commands across device types:

1. /devices/ID – returns the device status in a consistent JSON format.
2. /devices/ID/update – tells the device to report its current status back to the gateway. There is a little complexity here to avoid stale data, but nothing too awful.
3. /devices/ID/on – tells the device to turn on (for some device types this may mean “do your thing” e.g., as the command to press a toggle button).
4. /devices/ID/off – tells the device to turn off.
5. /devices/ID/exact?level=# – tells the device to apply an integer value from a range, if the device supports it (e.g., 0-100 for a dimmer switch, or a thermostat setting, etc.)

Finally, some code!

With a Pi set up with the RaZberry board, Z-Way installed, devices added and an API selected, we are finally ready to write some code. You can actually download and build this right on your Pi or anywhere that has maven, git and java installed:

sudo apt install git maven default-jdk # if needed
git clone https://github.com/seanno/shutdownhook.git
cd shutdownhook/toolbox
mvn clean package install
cd ../zwave
mvn clean package

Next you’ll need a configuration file in json format that looks like this:

{ "Login": "LOGIN", "Password": "PASSWORD", "BaseUrl": "http://localhost:8083" }

Assuming you’ll run this code on the same Pi that is running Z-Way, just replace LOGIN and PASSWORD with the credentials you used when setting up the Z-Way interface. If you want to run from another machine on your local network, also replace “localhost” with the IP address of your Z-Way Pi. Finally, if you want to run the code from anywhere in the world, set the BaseUrl value to be “https://find.z-wave.me” and for LOGIN use your Access ID + slash + login, just as we talked about earlier when setting up devices.

Next, verify your build and configuration by running (from within the zwave directory):

java -cp target/zwave-1.0-SNAPSHOT-jar-with-dependencies.jar \
    -Dconfig=PATHTOCONFIG \
    com.shutdownhook.zwave.App \
    devices

If all goes well, you’ll see a list of all the virtual devices attached to your Z-Wave network (name, type, and id). Woo hoo! A command like this will set a dimmable light named “Pantry” to 50%:

java -cp target/zwave-1.0-SNAPSHOT-jar-with-dependencies.jar \
    -Dconfig=PATHTOCONFIG \
    com.shutdownhook.zwave.App \
    Pantry exact 50

Command-line options corresponding to each of the VDev APIs can be found in App.java.

Talkin’ Z-Way

The code that communicates with the Z-Way gateway lives in ZWay.java, ready to use standalone in your own projects. In general it’s pretty simple: instantiate the object providing a ZWay.Config; use getDevices to enumerate the network; get status with getLevel; send commands with turnOn / turnoff / setLevel; remember to call close when you’re finished.

Of course, there are always some fun details under the covers. You can connect either to your local web endpoint or the cloud-based one (remember to add your Access ID to the “Login” configuration if you do this). Cookie-based authorization is supported on both versions of the endpoint, so we use that. The class is a little lazy about authorization timeouts — tokens expire in a week, so we re-fetch them after six days (or each time a new object is created). This interval isn’t guaranteed, so it’s conceivable the strategy could fail at some point, but that seems unlikely. If it happens, sorry, my bad. Do remember to call “close” on the object when you’re shutting down — Z-Way remembers these tokens persistently, so if you forget you’ll end up with a ton of orphan tokens clogging up the works.

Stale device data presents another wrinkle. Remember that sending a Z-Wave command is basically fire-and-forget; some devices send back updated status, but many do not. And the ones that do, do so asynchronously. If you set a device value and then immediately query the gateway, almost certainly the data you get back will be stale. I tried to balance performance and hassle by addressing this in two ways:

1. The configuration UpdateOnCommand (default true) causes every command to be followed by an explicit “update” to the affected device. If you don’t need to reflect command changes immediately, this tends to keep the gateway values up-to-date pretty well with minimal chatter. Setting this configuration value to false makes the set operation a little more performant, but at the cost of more uncertainty about gateway values.
2. The status methods all take a refresh boolean parameter. If this value is true, you will (almost) always receive up-to-date values, but the call will be a little slower and result in a minimum of three network requests. In refresh mode we ask for the value (noting the update timestamp), request an explicit update, and then re-fetch the value until the update timestamp changes or we give up (“give up” settings can be configured using the MaximumUpdateRefreshIterations and UpdateRefreshIntervalMilliseconds values).

In most home scenarios, none of this is going to matter that much, you can refresh at will, and the default values will probably work a-ok. But it’s always helpful to know what’s going on below the waterline, so there you go.

A Handy Web UX

My goal was to control my lights in two ways — with my phone, and with Alexa. This article is getting pretty long, so I’m going to cover Alexa in the next one (you can get a sneak peek in Queue.java). For the phone, I chose a simple, bare-bones HTML approach. Since it’s running entirely on my local network, I’ve ignored login and wire encryption, although neither would be super-complicated to add.

Server.java implements this web interface using three core concepts:

1. A “Screen” is a logical collection of devices and could have reasonably been called a “Room” or “Location.” Each screen is displayed on its own web page and is associated with VLights and Settings.
2. A “VLight” is a collection of Z-Wave devices that are addressed together. For example, there are four smart ceiling lights in the family room that are controlled by a single switch and should always be on/off/dimmed together — these are collected into a single VLight.
3. A “Setting” is a list of VLights and values that together put lights into a useful configuration. The “Movie” setting dims the lights in the family room and turns off all of the lights on the periphery, while “Cooking” turns all the lights in the kitchen to their brightest levels.

All of this is described in a JSON configuration file defined by the Server.Config class. You can see a sample configuration in example-server.json that exposes the service on port 7071. Using the same binaries you built earlier, fire up the server with the following command, which starts it up in the background and saves any log output to PATHTOLOG:

nohup java \
    -cp target/zwave-1.0-SNAPSHOT-jar-with-dependencies.jar \
    com.shutdownhook.zwave.Server PATHTOCONFIG >> PATHTOLOG &

Each screen displays its Settings (plus “on” and “off” which do the obvious) as pushbuttons, and each VLight as a slider. Pressing a Settings button sets all of its VLights to the appropriate levels; sliding a slider sets the brightness of all the devices within that VLight (including 0 which turns the device off). No muss, no fuss, but extremely usable for my purposes. Voila! (Remember that in my scenario all the devices are dimmable lights; I likely will add some non-dimmable switches into the mix soon and will have to tweak things a bit when that happens.)

The guts of this are all things I’ve discussed before, primarily WebServer.java, Template.java and WebRequests.java. These workhorses continue to show up quite nicely; I particularly love the interplay between the template and code in screen.html.tmpl and registerScreenHandler. Ooh, who else felt that little code reuse dopamine hit?

What Next?

With this web app pinned to my phone’s home screen, my lights are finally back in business. They still aren’t voice-controlled, but this article has gotten way too long so I’ll pick up that task next time. It turns out that poking around at Alexa skills is pretty interesting, so check back or follow or whatever so you don’t miss out. Until then — I hope your lights do what you tell them! And let me know if you find a bug or if I can help you work through your own Z-Wave adventure.

OK, just a little more miscellany

A few last tidbits and quirks that I had to figure out the hard way; hope they save you a little frustration:

• Z-Wave network lag can be super annoying. Commands seem to go into a black hole, only to execute a couple of minutes later. This happens when the job queue gets backed up; your new command just has to wait its turn. There can be a number of reasons for this, but for me it usually happens when devices registered on the gateway are offline, for example when somebody turns off the wall switch controlling a smart bulb. When the node is absent, cached route maps can fail and force a bunch of retries that slow things down. Lag can also come from too much background noise on the unlicensed radio band — check out the “Analytics” tab on the Z-Way Expert Interface to dig into this.
• Some Z-Wave devices are security-enabled and will prompt you for a PIN (usually found on a sticker on the device and/or packaging) during the inclusion process. You can bypass the PIN if necessary, but in my experience that is a super-bad idea. When you do, part of the “interview” between the gateway and device fails forever and seems to create confusion (i.e., extra chatter) on the network. It’s anecdotal, but save yourself some hassle; look up the PIN ahead of time and have it ready.
• Lastly, getting devices OFF of your network can be a big hassle. The process here is exclusion (the obvious opposite of inclusion); in order to work cleanly the device must be available and responsive on the network during the exclusion process. While it’s possible to recover and force a device off of the network, it’s messy at best — try to think ahead so you don’t end up with a bunch of zombie nodes on your network.
• Whew, I think that’s it. Maybe. For now.