Tag: z-wave

Alexa, ask Bellevue House…

~ Sean Nolan

This is part two of my smart light adventure; read part one here.

At the end of the last post, my orphaned Z-Wave lights (thanks again Wink) were working again under the control of a little web app I’d written for my phone. Individual devices were grouped into “virtual lights” that could then be switched on/off or dimmed, either individually or through coordinated “Settings” designed for a particular purpose like watching TV or taking a nap.

Handy, but still missing a key bit of functionality — I’ve gotten used to asking Alexa to turn on the lights in the morning and turn them off at night. Yes I’m that lazy. So my next challenge was to figure out how to wire up my bespoke solution to an Alexa skill. It turns out that Alexa has a whole API dedicated to smart home stuff, which seemed the obvious place to start, but I was quickly overwhelmed with the complexity and ran away screaming. Don’t get me wrong, it’s super-powerful and I appreciate how it enforces a consistent interface to a ton of diverse devices and system. It’s just that my scenario is super-simple; I just want to trigger pre-configured settings with statements like “Alexa, ask Bellevue House to set family room to nap.” For a problem this constrained, the most basic of skills did the trick.

Creating the Skill

Anyone can build Alexa skills and deploy them to their own devices for free. Getting them into the Alexa Skills Directory is more complicated, but not required for our scenario. Just be sure to sign up for the Alexa Developer Program using the same Amazon account that your devices are registered to. There is no cost to develop a skill or use it this way.

The first step is just to create the skill; below are the steps I used. You’ll see throughout that I was extremely lazy; this default template includes a bunch of “hello world” interactions that I haven’t removed — perhaps I’ll come back someday and clean all that up, but probably not.

  1. Log into the Alexa developer console (remember to use the same Amazon account used to register your Alexa devices).
  2. Click “Create Skill”.
  3. Give the skill a name, then choose the “Custom” model and the “Alexa-hosted (Node.js)” hosting option. Then click “Create skill” at the top right.
  4. Choose “Start from Scratch” and click “Continue with template.” This will take a minute or two to complete.

The Interaction Model

Next up is the “interaction model.” This is the grammar and request/response framework for the conversation with Alexa. Mine is very simple: “Alexa, ask Bellevue House to set family room to nap.” Even for this, there are a bunch of components at work:

  • “Alexa, ask Bellevue House to…” is the standard way Alexa interprets which skill you want to invoke. “Bellevue House” is the skill invocation name.
  • Each skill can include one or more intent. My skill only supports one intent, invoking a Setting on a Screen. I creatively named this “ScreenSettingIntent”.
  • “set family room to nap” is an utterance associated with the intent. Because there can be many ways to express the same intent, you can configure multiple utterances. In my case, I also added “turn kitchen off”.
  • Those bold phrases (“family room / kitchen” and “nap / off”) are specially tagged as slots within the utterance. These are the variables that tell the skill what to do. Each slot is assigned a slot type which describes what kind of content is likely to appear in the slot. There are tons of built-in slot types, or you can create your own. An important feature of slot types is that they are not closed vocabularies; sample values help train the model, but Alexa will attach whatever she hears even if it deviates from that training set.

At the end of the day, a successful invocation of the skill identifies the intent, attaches values to each of its slots, and triggers code to actually do whatever should be done. Before we look at that code, a quick cheat sheet for setting all of the values in our interaction model:

  1. Set the skill invocation name under Build / Invocations / Skill Invocation Name. (I used “Bellevue House” but that probably doesn’t work for you!)
  2. Under Slot Types choose Add Slot Type, set the name to SCREEN_NAME, click Next and add some sample values.
  3. Repeat this process for a new Slot Type called SETTING_NAME with appropriate samples.
  4. Under Intents choose Add Intent, set the name to “ScreenSettingIntent” and click Create Custom Intent.
  5. Scroll down to Intent Slots, and add two slots:
    1. “screenName” with the type “SCREEN_NAME”
    1. “settingName” with the type “SETTING_NAME”
  6. Scroll back up and add utterances with slot placeholders (the curly-braces mark the slots):
    1. “set {screenName} to {settingName}”
    1. “turn {screenName} {settingName}”
  7. Use the buttons at the top to “Save Model” and “Build Model”.

Handle the Intent

Each time Alexa needs to do something related to a skill, she makes a call to an HTTPS URL configured for that skill, passing request details as a JSON-formatted POST. This URL can live anywhere and be written in any language. When we created our skill, we chose the “Alexa-hosted (Node.js)” hosting option — under this model, Alexa allocates and hosts an AWS Lambda function for us for free, which seems awfully generous. We can edit the code for this function using the “Code” tab at the top of the Alexa developer console.

Remember I’m being lazy here; there is a bunch of boilerplate code auto-generated for us, and I’m just letting it all be. The important stuff for us is in index.js, a copy of which I’ve stashed for reference up at the ShutdownHook github. This file defines a few global utility objects at the top, then a whole bunch of handler functions, and then at the bottom wires the handlers up as “exports.handler”. Each time Alexa makes a request, this list of exports is scanned for a matching handler until one is found and executed.

The code I’ve added to handle our intent is in the function ScreenSettingIntentHandler, defined at line 10 and inserted into the handlers list at line 208. Don’t worry about the details quite yet — first we have to figure out how we’re going to communicate from this Alexa-hosted Lambda function all the way down to the home control web server we built last time.

Bringing the Outside Inside

We’re getting there, but there’s a networking problem we still need to solve. One that comes up a lot when building solutions that integrate Internet services with code running on a home network. Our personal routers are really good at getting us OUT onto the Internet, but they frown upon the Internet getting back IN to initiate communication with devices inside our houses. This is a very good thing of course — Internet security is scary enough without inviting hackers inside our private networks for drinks and conversation.

So hooray for routers and personal firewalls. But when you DO want an event on the outside to trigger something on the inside (say, for example, a notification from Alexa to turn on the lights), it can be a little challenging. It’s absolutely possible to create the necessary routes: tell your DHCP server to assign a static internal IP address to the device you care about; configure your router to pass through traffic to that address; and set up something like https://www.duckdns.org/ (a primo service btw) to assign a name that outside devices can use to find it. It’s certainly not the end of the world, but it is complicated, and it’s very easy to get wrong in a way that makes you vulnerable to the bad guys out there. I don’t recommend it.

A better solution uses an intermediary to serve as a go between. In this model Alexa drops a message somewhere on the Internet saying “please turn on the lights,” and the inside device reaches OUT to the intermediary to pick up the message. Message queues that work this way are used all over the place, and certainly will do the trick for our use case. Two design decisions will help us pick what flavor of queue we’ll use:

  1. How quickly do incoming messages need to be acted on? In our case, we don’t need millisecond responsiveness, but it can’t take more than a second or two — the lights really need to come on when I ask them to, not five minutes later.
  2. Do message senders require a synchronous response to their messages? That is, does the sender need to wait until the message has been picked up and handled before they can move on? This seems like a nice-but-not-essential feature for our scenario, since it’s pretty obvious if lights come on or not.

Setting up the Queue with SQS

There really are tons of very reasonable, lightweight ways we could go about this, but for now I just went with Amazon’s Simple Queue Service, a well-supported workhorse service with good Java support and easy integration into the Alexa side of things. SQS does require the client to poll for messages, but minimizes the performance penalty by supporting “long polling.” Each polling request stays active for up to twenty seconds, returning immediately if a message arrives. The worst case then is three messages per minute; 180 per hour; 4,320 per day — basically nothing as far as any modern CPU and network is concerned. And in return we get more or less instantaneous delivery. Cool beans!

In order to use SQS you need an account with Amazon Web Services. Note this is distinct from the Alexa developer account, although you can use the same email. There is a remarkable amount you can do within the AWS free tier, including (at least as of this writing) making up to a million SQS requests per month. I’m pretty sure I’m going to stay under a million requests per month turning my lights on and off.

So the big picture is that we’ll set up a Queue in our AWS account and drop a message into it each time our Alexa handler is called. We’ll then add a thread to our home control web server that long-polls the queue for messages and executes the requested screen/setting behaviors on our Z-Wave network. No sweat!

The most challenging part about all of this is authorization; we need to give our Alexa-hosted Lambda function the rights to post messages to the queue in our AWS account. This can get a little hairy, so buckle up and bear with me (general instructions can be found in the Alexa docs under “Use Personal AWS Resources”). The nut of it all is that we’re creating a role in our AWS account that has rights to post messages to a new queue, and then giving the Alexa role the rights to “assume” the AWS role when the skill is invoked:

  1. In the Alexa developer console, find the ARN of your Alexa-hosted Lambda role. Click the Code tab, then “Integrate” in the toolbar. Copy the “arn:” value there and tuck it away. We’ll refer to this as the Alexa Role ARN going forward. Make a note of the account number in this ARN, e.g., the account number for the ARN “arn:aws:iam::866314627097:role/AlexaHostedSkillLambdaRole” is 866314627097. We will refer to this as the Alexa Role Account.
  2. In the AWS management console, choose “Simple Queue Service” under the huge “Services” dropdown at the top-left.
  3. Click “Create Queue” and then the following options:
    1. A “Standard” queue is fine, although it will require a bit of message de-deduplication logic we’ll see later.
    1. Any name is fine.  
    1. Under “Access Policy”, choose “Basic” and then select the radio button “Only the Specified AWS accounts, IAM users and roles” under “Define who can send messages to the queue.” In the edit box that appears, enter the Alexa Role ARN.
    1. The rest of the default configuration settings are fine. You may optionally choose to configure a “dead letter” queue that will receive failed messages; details are here and you can add the option later if you choose.
    1. Click “create queue” to confirm the operation.
  4. On the queue information page that appears, copy the “arn:” value for the queue and tuck it away. We’ll refer to this as the Queue ARN going forward. Also copy the URL value which we’ll refer to as (perhaps not surprisingly) the Queue URL.
  5. Choose “IAM” under the huge “Services” dropdown at the top-left.
  6. Click “Roles” and then “Create Role”.
  7. Under “Select type of trusted entity”, choose “Another AWS account,” enter the Alexa Role Account you saved in step 1, and click “Next.”
  8. On the “Attach permissions policies” page just click “Next: Tags”, then “Next: Review”.
  9. Give the Role a name and then click “Create role.”
  10. Click your newly-created role in the list, and then “Add inline policy” under “Permissions”. Use these settings for the policy:
    1. Under “service”, search for and add SQS.
    1. Under “actions”, search for and add SQS:SendMessage.
    1. Under “resources”, choose “Add ARN” and enter the Queue ARN.
    1. Click “Review Policy” and then provide a name for the policy and click “Create policy.”

Whew, almost done! That takes care of creating the queue and setting up the Alexa role to be able to send message to it. The last bit of authorization required is a user that we will use from the home control web server to poll for messages. We need an access key and secret for that user.

If you’re really really lazy and are logged into the AWS management console as the root user, you can just choose “My Security Credentials” from the top-right account menu, allocate a key and secret, and use those. But those credentials have an insane level of access. Much better to go into the breach one more time and create a user just for accessing the queue:

  1. Logged into the AWS management console, choose “IAM” under the huge “Services” dropdown at the top-left.
  2. Click “Policies” and “Create Policy”.
  3. As with step 10 above, create a policy with access to the Queue ARN, but in this case under actions choose “All SQS actions”.
  4. Click “Users” and “Add Users”.
  5. Provide a user name (I used “homepi”; if I end up using other AWS services from the home control system I’ll reuse this user).
  6. Check “Access key – Programmatic access” and then “Next: Permissions”.
  7. Choose “attach existing policies directly”, then search for and check the box for the policy created in step 3.
  8. Click “Next: Tags” and then “Next: Review”.
  9. Name the user and click “Create User”.
  10. Copy the Access Key ID and Secret access key shown on the confirmation page.

Wow, you made it! Now go have a beer and then come back for the rest. It’s all downhill from here!

Sending to the Queue

OK, we’re ready to look at the code that sends the message from Alexa into the queue we created. This is pretty simple, although figuring the authentication most definitely was not. First, add a couple of global service clients at the top after the Alexa object is allocated (lines 7-8 of my sample index.js):

const AWS = require('aws-sdk');
const STS = new AWS.STS({ apiVersion: '2011-06-15' });

Next, add the handler itself (lines 10-66), which breaks down like this:

  • Lines 11-14 tell Alexa that this routine handles the “ScreenSettingIntent” intent that we built so long ago.
  • Lines 16-18 pull the “screenName” and “settingName” slot values out of the request, as heard by Alexa.
  • Lines 20-31 build the JSON content of the message we’ll put into the queue. The queue will take messages in any format. Be sure to use your own Queue URL at line 30!
  • Line 33 constructs the response that Alexa will speak back to the user. This is where having a synchronous response from our home server would be nice, because Alexa doesn’t really know if the setting was applied successfully or not.
  • Lines 35-51 creates a SQS client operating with the assumed Role we created that has rights to send messages to the queue.
  • Lines 53-60 actually, finally, send the message!
  • Lines 62-64 send the response back to Alexa.

Finally, we register the handler by adding the handler to the list at line 208:

exports.handler = Alexa.SkillBuilders.custom()
    .addRequestHandlers(
        LaunchRequestHandler,
        ScreenSettingIntentHandler,
        HelloWorldIntentHandler,
        HelpIntentHandler,
        CancelAndStopIntentHandler,
        FallbackIntentHandler,
        SessionEndedRequestHandler,
        IntentReflectorHandler)
    .addErrorHandlers(
        ErrorHandler)
    .withCustomUserAgent('sample/hello-world/v1.2')
    .lambda();

With all this in place, use the “Save” and “Deploy” buttons at the top of the code editor to push out your code.

On the “Test” tab in the code editor, you can test the skill right from your browser — just hold down the mic button and try out one of your utterances. You’ll see the response from Alexa, including logs and message details. Over at the AWS management console, you can peek at the contents of your queue to see if the message has actually made it. Don’t be discouraged if you get some auth errors on the first try; there is so much to configure here it’s hard to get it all right.

The really cool thing is that the skill is also now available on all of your Alexa devices. Honest! Skills in test are pushed to all of the Alexa devices registered to the account, so there’s no need to worry about publication or certification. Unless of course you want to allow random folks to turn your lights on and off from their own homes? Weird.

Reading from the Queue

We really are at the last mile now — all that’s left is to read messages out of the queue and take action based on the slot values we receive. For this we’re back to our (well at least my) happy place … Java that runs on my machine. We already have a process running on the Pi that implements the home control web server. We’ll just add a thread that long-polls the queue for messages.

Most of the implementation here is in Queue.java. We also need to add a reference to the AWS SQS Java SDK in our pom.xml. It’s basically obscene how much dependency this drags into our project, but signing AWS access tokens is a huge hassle — so I’m holding my nose and just living with it:

<dependency>
  <groupId>com.amazonaws</groupId>
  <artifactId>aws-java-sdk-sqs</artifactId>
  <version>1.12.94</version>
</dependency>

The reader thread is built on Worker.java … if you’re in the mood for more judgy commentary about managing backgound threads, feel free to check out this article, one of the first I wrote for the blog earlier this year. The queue thread itself is pretty straightforward — allocate the client object on creation, enter a long-polling loop that calls receiveMessage and sends received content to a handler interface and clean up the messages when that’s done. Just a few notes:

  • SQS supports two queue models: “FIFO” queues preserve ordering and guarantee exactly-once delivery; “Standard” queues are more efficient, but do not guarantee ordering and may deliver the same message more than once. I used a “standard” queue, so added a bit of code to protect against duplicates. It only works if two duplicates arrive in sequence, but for our low-volume use case that seems to be the norm.
  • The SQS retrieval pattern is standard for queue technology — when a client receives a message, it becomes “invisible” for a short window (default 2 minutes). If the client successfully processes the message, it confirms this by deleting the message. If that delete does not happen for some reason, the message reappears for another client to retrieve. This serves as a great retry model for transient failures; our Queue expects the handler to throw an exception on failure to support the pattern. It can be problematic for “poison” messages that can never be processed — this is where a dead letter queue comes into play.

The handler code in Server.java does the actual work of turning the lights on and off according to the screen and setting values. Which honestly seems a little anti-climactic after all the minutia we fought to get here … but we made it. I love the end result, and hope all of those lists and steps will save you some time, but the ever-spiraling complexity at AWS (and the other cloud providers) really is unfortunate.

Pushing my luck

Now I can manage my lights from my phone and with my voice. There’s just one last thing I’d love them to do: turn on and off automatically when I get up during the night. I found these pretty cool PIR motion sensors at Amazon, so let’s see if I can get that hooked up. Next time!

Z-What? Rescuing my Z-Wave smart lights.

~ Sean Nolan

We’re definitely in the “chaotic innovation” period of smart home stuff. Embedded technology and cloud connectivity are advanced enough to support actually useful smart lights, vacuums, thermostats, cameras, washers, dryers, and a ton of other devices. And we’re starting to see an early veneer of consistency, as most can forge some kind of a connection with Alexa or Google Home or whatever. But under the covers, things are pretty insane. Companies enter and leave the market constantly, and each seems to require a new account with a new password, maybe another hardware gateway, and probably a recurring subscription fee.

The pace of innovation is actually super-exciting and fun … it just isn’t tidy, as evidenced by the growing stack of dead-end devices in my closet. Mostly that’s just part of the process. But what really kills me is companies that pull the bait-and-switch, like Wink did last year when they abruptly starting charging monthly for functionality that had been sold as a one-time purchase. This is the height of shoddy business — your inability to do basic math on costs is not an excuse to renege on promises made. Especially when those promises were instrumental in growing your user base against other (more honest) companies. Not that I’m bitter or anything.

In any case, that move left me with a house full of smart lights that, just like the humans and dogs that live here, refused to listen to me. Which became the impetus for a deep dive into smart home technology that turned out to be a ton of fun and generated some useful and/or interesting code. Let’s check it out.

First, the Network

It seems inevitable that when all this settles out, all of our smart devices are just going to connect via wifi like everything else. This is already pretty much the case for “high end” stuff like washers and cameras, but wifi has mostly been considered too expensive for low-margin devices like lightbulbs, and too power-hungry for battery-powered stuff. While both are really non-issues at this point, it takes a while for the consumer device manufacturers to catch up.

So at least as of today, there are tons of devices out in the wild that are not wifi, so what do they use? One option is Bluetooth, but that’s pretty rare and generally sucks just like Bluetooth always sucks. I have a BT-based August door lock and it’s just dumb. You can add a wifi gateway (and I have) but that just adds more money and more complexity to a unit that eats batteries like Cookie Monster eats cookies anyways. Nope.

This leaves two technologies that were designed ground-up for smart home use: Zigbee and Z-Wave. This is very much a Coke vs. Pepsi kind of thing — there are a few differences, but for the most part they’re the same thing solving the same problems:

  • Both require a “gateway” to coordinate communication on the network. This is typically a dedicated hardware unit that keeps an inventory of devices, sends them commands, and receives status updates. The gateway is also the face of the Z* network to the outside world — usually over wifi, either using an embedded webserver or a direct connection to a cloud service, or both.
  • Both are “mesh networks,” which is quite handy in the home environment. Z* signals can only travel about 20-30 meters, but each node acts as a “relay” to pass messages along. So if device “A” is 20 meters from your gateway, and device “B” is another 20 meters farther out, “A” can serve as an intermediary to relay messages between the gateway and device “B”. Lighting devices are particularly well-suited to this kind of network, since bulbs tend to be evenly distributed around the house.

The differences aren’t super-important. Zigbee enables more hops between nodes and more devices on the network; Z-Wave has a longer reach between nodes and is cheaper. Amazon’s Echo Plus has a built-in Zigbee gateway, so that’s nice. But either one is perfectly serviceable; just make sure you’re buying devices that match the technology in your gateway! There are a few gateways that contains chips for both protocols if you really really want to run both.

Z-Wave it is … hello RaZberry!

I didn’t really know any of this when Wink dropped the subscription bomb, but I figured there had to be a simple way to make my lights start listening again. My first step was to get out the ladder and look at the lights to figure out who made them and what was inside. It turns out that most of my lights are made by GoControl — pretty neat units that slide right into an existing 6-inch ceiling can. Most importantly, that little Z-Wave logo told me where to start!

A couple of hours of research made it clear that (a) Aeotec probably makes the best ready-to-go Z-Wave compatible hub available at the moment, but also that (b) the market is super-unsettled with companies entering and leaving almost every day. Not a lot of stability. I also didn’t see any control software that I wanted to get invested in — just a bunch of complex user interfaces designed by engineers.

Plan B — Could I take control of my destiny by getting a little closer to the metal and building something myself? It turns out that the answer is yes; the RaZberry daughter board for Raspberry Pi delivers excellent Z-Wave capability and is fully programmable. Shiny! Now, full disclosure for those of us that worry about certain state actors: RaZberry is manufactured by Z-Wave.me, a company out of Russia funded by the non-profit Skolkovo Foundation. From everything I see, they have been extraordinarily open and explicit about how the hardware works and what it does / does not share with its cloud service (for starters see their privacy policy). I was able to get comfortable here; your mileage may vary.

The RaZberry also comes with a license for Z-Way, their software stack that includes everything from a low-level C API up through a user-facing home control cloud service. Getting this all built out was my first order of business and was pretty straightforward:

  1. Set up the Pi. I used a 3B, but the RaZberry is compatible with all models that have the header block. I splurged on an official Pi3 case; the daughter board fits inside just fine.
  2. I used Pi Imager to set up the SD card as per the usual, but picked up two nice tricks (hat tip) that meant I could do the setup completely headless. First, add an empty file named “ssh” to the root of the SD card after using the imager — this tells the OS to start the SSH daemon by default. Second, pre-configure your wifi details by adding a text file “wpa_supplicant.conf” to the root of the card as well; this link describes how to set up the contents for your network. SO much better than having to hook up a monitor and keyboard.
  3. Install the Z-Way software by downloading and running the install script: “wget -q -O - https://storage.z-wave.me/RaspbianInstall | sudo bash” (details here).
  4. Find the Pi’s IP address with “ifconfig | grep inet” and browse on over to http://IPADDRESS:8083 on your local network to set up passwords and such. That’s it! Your Pi is now running as the gateway node of a brand new Z-Wave network.

Set up devices with the Z-Way interface

My snide comments about engineer-designed interfaces aside, the web-based “Z-Way Smart Home Interface” really does provide a ton of functionality: device management, diagnostics, device control, automation of complex scenarios, integration with Alexa, and a ton more. You could 100% stop reading this article now and just use Z-Way as your smart home ux and be pretty happy.

If you want to do this and also access your gateway from outside of your local network, you’ll need to use the Z-Way cloud service at https://find.z-wave.me. Log on to the local browser interface and navigate to the Settings / Remote Access section. Make sure that the “Enable Remote Access” checkbox is checked and take note of your “Access ID” number. When you log in at https://find.z-wave.me, you’ll enter your login as this Access ID, a slash, and then your user name; e.g., “123456/admin”. This public URL just serves as a proxy to your local gateway to provide access from anywhere. As I mentioned above, you’ll have to decide for yourself how comfortable you are with using the Z-Wave.me cloud services. In addition to remote access to your gateway, there are features to enable remote support and backup of your configuration to the cloud. All of this can be disabled if you want to keep your data entirely local.

All well and good, but the point of this whole exercise was to avoid being dependent on a third-party service that could go away at any time. So I’m avoiding the cloud and only using the Z-Way app locally for network administration and troubleshooting. Most importantly I use it to “include” devices in the network.

This part is kind of like Bluetooth pairing. You tell the gateway to enter “inclusion” mode, and then tell your device to do the same. When they each notice each other, an association is formed, and the device is registered as part of the network.

  1. Make sure the device is not already part of another network. If you just bought it, all good. If not, dig out the manual and figure out how to “reset” it. For my GoControl devices, you do this by turning the power on and off four times — the lights blink twice to show they’ve been reset.
  2. Choose “Devices” from the Z-Way top-right menu, then “Add new” next to Z-Wave in the top row, then “Add new Z-Wave Device and identity it automatically” at the top, and finally “Start” to put the gateway into inclusion mode.
  3. Put the device into inclusion mode, again according to the manufacturer’s instructions. For my lights that just meant turning them off and on again. If all goes well, the gateway and the device will see each other and live happily ever after. My lights acknowledge this by blinking twice.

Two little gotchas in this process. First, each time you run the process above, only one device will be included. If you have a single switch that controls multiple bulbs, it’s a little messy. Step #1 will reset all the bulbs at once, but you’ll have to repeat steps 2 and 3 multiple times until all the bulbs in the group have been included. Don’t worry, we’ll set things up so you can easily turn the whole group on or off together later on. If you’re starting from scratch, though, using a Z-Wave switch (many options, I’m using this one) and regular bulbs may be simpler.

Also, be aware that inclusion (and exclusion) can be quite sensitive to proximity. The “mesh” part of the network doesn’t seem to apply during this process, so the gateway needs to be quite close to the device you want to include. If you go through the steps above and nothing happens, try moving your gateway temporarily closer to the device. When I’m adding devices, I actually plug the gateway into a portable power station so I can easily carry it around the house. (We bought the power station for power outages, not Z-Wave maintenance, but it’s a nice bonus!)

APIs Everywhere

OK, first let me say that there is a LOT going on in this protocol, and about a million ways to dig in. My development scenario is really straightforward — manage a bunch of dimmable lights — so I’ve chosen a pretty simple, high-level approach. My bet is that for most folks using the RaZberry, starting from what I’ve done here is going to be the least insane choice, but just for the record:

  1. Z-Wave is currently (as of 2018) owned by Silicon Labs, which publishes an official Z-Wave SDK. It’s primarily meant for device implementers, but there’s a bunch of other stuff in there too.
  2. OpenZWave is a fully open-source C++/.NET library that is the basis for a bunch of other stuff.
  3. Z-Wave JS is another open source set of libraries and apps, including zwavejs2mqtt which seems quite popular.
  4. Z-Way itself has a bazillion different ways to work with it. I tried to summarize them all, but gave up — read all about it in their full documentation.

Some of the cacophony here is just history; the IP rights for Z-Wave have changed hands a million times and it shows. But more than that, the protocol is just complicated. Commands sent to the gateway go onto a queue and are marshalled out onto the wire when bandwidth is available. Responses, if any, come back from nodes asynchronously and without any strong binding to the original command. Early on there was some patent issue that meant nodes couldn’t proactively update the gateway with status changes, putting the onus on developers to figure out if data they received was up-to-date or stale. And all this on top of a protocol that is just trying to encompass a TON of diveres device types. It’s a lot.

Anyways, with my simple scenario I was hoping to find a simple API to work with. It turns out that the Z-Way “VDev” (virtual device) API fits that bill perfectly by (a) providing a normalized and simplified view of the device set; and (b) exposing a small, standard set of REST commands across device types:

  1. /devices/ID – returns the device status in a consistent JSON format.
  2. /devices/ID/update – tells the device to report its current status back to the gateway. There is a little complexity here to avoid stale data, but nothing too awful.
  3. /devices/ID/on – tells the device to turn on (for some device types this may mean “do your thing” e.g., as the command to press a toggle button).
  4. /devices/ID/off – tells the device to turn off.
  5. /devices/ID/exact?level=# – tells the device to apply an integer value from a range, if the device supports it (e.g., 0-100 for a dimmer switch, or a thermostat setting, etc.)

Finally, some code!

With a Pi set up with the RaZberry board, Z-Way installed, devices added and an API selected, we are finally ready to write some code. You can actually download and build this right on your Pi or anywhere that has maven, git and java installed:

sudo apt install git maven default-jdk # if needed
git clone https://github.com/seanno/shutdownhook.git
cd shutdownhook/toolbox
mvn clean package install
cd ../zwave
mvn clean package

Next you’ll need a configuration file in json format that looks like this:

{ "Login": "LOGIN", "Password": "PASSWORD", "BaseUrl": "http://localhost:8083" }

Assuming you’ll run this code on the same Pi that is running Z-Way, just replace LOGIN and PASSWORD with the credentials you used when setting up the Z-Way interface. If you want to run from another machine on your local network, also replace “localhost” with the IP address of your Z-Way Pi. Finally, if you want to run the code from anywhere in the world, set the BaseUrl value to be “https://find.z-wave.me” and for LOGIN use your Access ID + slash + login, just as we talked about earlier when setting up devices.

Next, verify your build and configuration by running (from within the zwave directory):

java -cp target/zwave-1.0-SNAPSHOT-jar-with-dependencies.jar \
    -Dconfig=PATHTOCONFIG \
    com.shutdownhook.zwave.App \
    devices

If all goes well, you’ll see a list of all the virtual devices attached to your Z-Wave network (name, type, and id). Woo hoo! A command like this will set a dimmable light named “Pantry” to 50%:

java -cp target/zwave-1.0-SNAPSHOT-jar-with-dependencies.jar \
    -Dconfig=PATHTOCONFIG \
    com.shutdownhook.zwave.App \
    Pantry exact 50

Command-line options corresponding to each of the VDev APIs can be found in App.java.

Talkin’ Z-Way

The code that communicates with the Z-Way gateway lives in ZWay.java, ready to use standalone in your own projects. In general it’s pretty simple: instantiate the object providing a ZWay.Config; use getDevices to enumerate the network; get status with getLevel; send commands with turnOn / turnoff / setLevel; remember to call close when you’re finished.

Of course, there are always some fun details under the covers. You can connect either to your local web endpoint or the cloud-based one (remember to add your Access ID to the “Login” configuration if you do this). Cookie-based authorization is supported on both versions of the endpoint, so we use that. The class is a little lazy about authorization timeouts — tokens expire in a week, so we re-fetch them after six days (or each time a new object is created). This interval isn’t guaranteed, so it’s conceivable the strategy could fail at some point, but that seems unlikely. If it happens, sorry, my bad. Do remember to call “close” on the object when you’re shutting down — Z-Way remembers these tokens persistently, so if you forget you’ll end up with a ton of orphan tokens clogging up the works.

Stale device data presents another wrinkle. Remember that sending a Z-Wave command is basically fire-and-forget; some devices send back updated status, but many do not. And the ones that do, do so asynchronously. If you set a device value and then immediately query the gateway, almost certainly the data you get back will be stale. I tried to balance performance and hassle by addressing this in two ways:

  1. The configuration UpdateOnCommand (default true) causes every command to be followed by an explicit “update” to the affected device. If you don’t need to reflect command changes immediately, this tends to keep the gateway values up-to-date pretty well with minimal chatter. Setting this configuration value to false makes the set operation a little more performant, but at the cost of more uncertainty about gateway values.
  2. The status methods all take a refresh boolean parameter. If this value is true, you will (almost) always receive up-to-date values, but the call will be a little slower and result in a minimum of three network requests. In refresh mode we ask for the value (noting the update timestamp), request an explicit update, and then re-fetch the value until the update timestamp changes or we give up (“give up” settings can be configured using the MaximumUpdateRefreshIterations and UpdateRefreshIntervalMilliseconds values).

In most home scenarios, none of this is going to matter that much, you can refresh at will, and the default values will probably work a-ok. But it’s always helpful to know what’s going on below the waterline, so there you go.

A Handy Web UX

My goal was to control my lights in two ways — with my phone, and with Alexa. This article is getting pretty long, so I’m going to cover Alexa in the next one (you can get a sneak peek in Queue.java). For the phone, I chose a simple, bare-bones HTML approach. Since it’s running entirely on my local network, I’ve ignored login and wire encryption, although neither would be super-complicated to add.

Server.java implements this web interface using three core concepts:

  1. A “Screen” is a logical collection of devices and could have reasonably been called a “Room” or “Location.” Each screen is displayed on its own web page and is associated with VLights and Settings.
  2. A “VLight” is a collection of Z-Wave devices that are addressed together. For example, there are four smart ceiling lights in the family room that are controlled by a single switch and should always be on/off/dimmed together — these are collected into a single VLight.
  3. A “Setting” is a list of VLights and values that together put lights into a useful configuration. The “Movie” setting dims the lights in the family room and turns off all of the lights on the periphery, while “Cooking” turns all the lights in the kitchen to their brightest levels.

All of this is described in a JSON configuration file defined by the Server.Config class. You can see a sample configuration in example-server.json that exposes the service on port 7071. Using the same binaries you built earlier, fire up the server with the following command, which starts it up in the background and saves any log output to PATHTOLOG:

nohup java \
    -cp target/zwave-1.0-SNAPSHOT-jar-with-dependencies.jar \
    com.shutdownhook.zwave.Server PATHTOCONFIG >> PATHTOLOG &

Each screen displays its Settings (plus “on” and “off” which do the obvious) as pushbuttons, and each VLight as a slider. Pressing a Settings button sets all of its VLights to the appropriate levels; sliding a slider sets the brightness of all the devices within that VLight (including 0 which turns the device off). No muss, no fuss, but extremely usable for my purposes. Voila! (Remember that in my scenario all the devices are dimmable lights; I likely will add some non-dimmable switches into the mix soon and will have to tweak things a bit when that happens.)

The guts of this are all things I’ve discussed before, primarily WebServer.java, Template.java and WebRequests.java. These workhorses continue to show up quite nicely; I particularly love the interplay between the template and code in screen.html.tmpl and registerScreenHandler. Ooh, who else felt that little code reuse dopamine hit?

What Next?

With this web app pinned to my phone’s home screen, my lights are finally back in business. They still aren’t voice-controlled, but this article has gotten way too long so I’ll pick up that task next time. It turns out that poking around at Alexa skills is pretty interesting, so check back or follow or whatever so you don’t miss out. Until then — I hope your lights do what you tell them! And let me know if you find a bug or if I can help you work through your own Z-Wave adventure.

OK, just a little more miscellany

A few last tidbits and quirks that I had to figure out the hard way; hope they save you a little frustration:

  • Z-Wave network lag can be super annoying. Commands seem to go into a black hole, only to execute a couple of minutes later. This happens when the job queue gets backed up; your new command just has to wait its turn. There can be a number of reasons for this, but for me it usually happens when devices registered on the gateway are offline, for example when somebody turns off the wall switch controlling a smart bulb. When the node is absent, cached route maps can fail and force a bunch of retries that slow things down. Lag can also come from too much background noise on the unlicensed radio band — check out the “Analytics” tab on the Z-Way Expert Interface to dig into this.
  • Some Z-Wave devices are security-enabled and will prompt you for a PIN (usually found on a sticker on the device and/or packaging) during the inclusion process. You can bypass the PIN if necessary, but in my experience that is a super-bad idea. When you do, part of the “interview” between the gateway and device fails forever and seems to create confusion (i.e., extra chatter) on the network. It’s anecdotal, but save yourself some hassle; look up the PIN ahead of time and have it ready.
  • Lastly, getting devices OFF of your network can be a big hassle. The process here is exclusion (the obvious opposite of inclusion); in order to work cleanly the device must be available and responsive on the network during the exclusion process. While it’s possible to recover and force a device off of the network, it’s messy at best — try to think ahead so you don’t end up with a bunch of zombie nodes on your network.
  • Whew, I think that’s it. Maybe. For now.