Jekyll2020-07-16T05:58:33+00:00https://blog.hmac.io/feed.xmlhmac.io blogWriting about software engineering, hardware engineering, and other things.How I Built an Escape Room Magic Mirror2020-07-15T22:30:00+00:002020-07-15T22:30:00+00:00https://blog.hmac.io/2020/07/15/how-i-built-an-escape-room-magic-mirror<p><img src="/assets/images/building-an-escape-room-magic-mirror/header.jpg" alt="Title Image" /></p>
<p>This post is part of a series on an escape room I designed for my wife. In fact, the whole idea for the escape room started while I was making a Smart Mirror for her. It was just meant to be a simple project; a small mirror that shows a new poem every day. Yet like many ideas in my life the spark lit a fire and I found myself flung into three weeks of feverishly building an escape room…</p>
<h2 id="the-idea">The Idea</h2>
<p>As part of the escape room the Smart Mirror became a Magic Mirror and served as the main character’s, Mabel’s, way of speaking to the player. Early on in the game when a mysterious sheet of paper is held up it triggers a video of Mabel talking to the player through the mirror. Later, once the player has beaten the escape room, the mirror is triggered again to show Mabel congratulating them. Overall I was satisfied with how the mirror worked in the game, and I think it was delightfully surprising to the player when the mirror suddenly sprung to life and played a message. So here’s how everything was built.</p>
<h2 id="hardware-build">Hardware Build</h2>
<p>The hardware build wasn’t terribly unique in the Smart Mirror world. Like all Smart Mirrors it involves a digital display behind a one-way mirror. They look and work like regular mirrors, but can also superimpose text and simple graphics on the reflected image. But besides text and simple graphics, they can also show full images and video. People don’t tend to use them that way since it’s harder to see your reflection when a full-screen image is showing, and the quality of the image through the mirror is not very good. For a talking magical mirror, however, the washed out haziness of full-screen images works to its advantage.</p>
<p><img src="/assets/images/building-an-escape-room-magic-mirror/tablet-installation.jpg" alt="Tablet" /></p>
<p>Most Smart Mirrors involve a Raspberry Pi and either a repurposed monitor or TV for the screen. However I was only building a small mirror (8”x10”), so a monitor or TV was out. RasPi is nice, but would be a “wart” on the back of picture frame. Finally, I wanted to keep cost and complexity down, since I would be spending money and time elsewhere on the escape room. To that end I decided to go with a cheap tablet; namely a FireHD 8. I was able to get an older model off Amazon Warehouse for $50, far cheaper than the all-in cost of a RasPi+accessories+screen. Plus it gives me a camera, microphone, and a thin form factor that could fit into a picture frame. Most Warehouse deals are brand-new items that have just been opened and returned. Amazon cannot sell them as new, so they flip them on Warehouse. More importantly though, it let me get an older model FireHD which could be easily rooted.</p>
<p>I rooted the tablet in the usual way and put LineageOS on it. LineageOS makes it more responsive, uses less power, and doesn’t come with any of the default cruft which would detract from a seamless Magic Mirror experience. But it’s not without its faults. Only older releases of LineageOS were well supported, so I was stuck on LOS 14.1 (Android Nougat). And being non-standard, some things were a bit buggy (this will come back to haunt me later in the story…).</p>
<p>For the frame, I used an <a href="https://amzn.to/3ewrUOg">8x10 picture frame</a>. 8x10 is larger than the tablet, but I couldn’t find a better fitting smaller frame given the tablet’s awkward dimensions. A hand-built frame would be perfect, but was decided against since I was going to spend time building and coding a <em>lot</em> of other things for the escape room. Simple, ready-made was king. In the end, for the mirror’s main purpose of displaying poems in the corner, the mismatched size is fine.</p>
<p>The glass of the picture frame was replaced with a sheet of <a href="https://amzn.to/3946Y03">one-way acrylic mirror</a>. Compared to glass, acrylic mirrors are cheaper, easier to cut to size, and don’t run the risk of shattering into flesh seeking knives. They just suffer from being very easy to scratch and slightly lower optical quality. Cost being king, I went with acrylic. The sheet I got was 12x12. To cut it to size, most people use those special acrylic tools to score the sheet several times and then “snap” the sheet. Luckily, I have a table saw! I used a medium-toothed blade (fine would be better; I didn’t have one on hand), and fed it through on my sled at a slow-medium feed rate. Worked great.</p>
<p><img src="/assets/images/building-an-escape-room-magic-mirror/back-of-frame.jpg" alt="Frame's extended back" /></p>
<p>All-in, everything only cost $100. To assemble it, I couldn’t just throw the tablet behind the mirror and call it a day; tablets tend to be a bit thicker than a photos… So I whipped up some parts in Fusion 360 and 3D printed them to extend the back of the picture frame enough for the tablet to fit. I also added some 3D printed filler to take up the extra space and secure the tablet in place. It includes a channel to route the USB power through. Again, a hand-built picture frame would have been nicer and neater, but this was good enough.</p>
<h2 id="poetry-app">Poetry App</h2>
<p><img src="/assets/images/building-an-escape-room-magic-mirror/poem-mirror.jpg" alt="Poetry in Mirror" /></p>
<p>With the magic mirror built, it was time to build the software for it. The daily poem app is straight-forward; just a simple Android app with a text element and some timer logic in the background for updating it at night. I used the lovely <a href="https://poetrydb.org">PoetryDB</a> as the source of the poems. I only had to do a little filtering to skip poems that would be too long for the screen. And of course the usual Android stuff for keeping the screen always on.</p>
<h2 id="escape-room-app">Escape Room App</h2>
<p>For the escape room, I built a second app that would run instead. It had to handle two things: detect when the player holds a specific note up to the mirror and play the first video; and waiting for a trigger signal from the Magic Pedestal to play the final video. The Magic Pedestal will be covered in another post, along with the design for the overall escape room. For now it’s enough to say that when the player has collected three magic books and placed them on the Magic Pedestal, the pedestal sends a signal to the Magic Mirror.</p>
<div class="post-gif"><video preload="auto" autoplay="autoplay" muted="muted" loop="loop" webkit-playsinline=""><source src="/assets/images/building-an-escape-room-magic-mirror/demo.mp4" type="video/mp4" /></video></div>
<p>The note that “unlocks” the first video is a QR code with the words “Take a moment to reflect” written backwards (so it’d look correct when read in a mirror). Originally I had planned for the QR code to be hidden within a complex design so it wasn’t obvious and would seem more magical. But after testing I noticed that the escape room’s lighting (candle light) made it difficult enough as it was to recognize the QR code.</p>
<p>I used Google’s ML Kit to handle QR decoding, as it seemed like the easiest to get working. As usual, Google’s documentation is atrocious and outdated, but with a bit of effort and piecing together four or five different example codebases that all had some kind of deprecated APIs in them, I was able to get it to work. Once the correct QR code is detected and decoded, it plays the relevant video in a VideoView.</p>
<p>The other function of the app, triggering a video when the Magic Pedestal says so, was supposed to be the easiest part. I figured I could just listen on a TCP port in the app and wait for the Magic Pedestal to connect and send a specific packet. Alas, this is where LineageOS came back to haunt me. No matter how much I tried it simply <em>refused</em> to accept connections. There’s no firewall that I could find, no privacy setting, no set of permissions. Nothing. Not even root and SSHd could accept connections. So I decided on a more hacky approach that actually killed two birds with one stone. Plugging the tablet’s USB into the Magic Pedestal’s Raspberry Pi. That handles both powering the tablet during the escape room, and gives the pedestal access to adb. With adb I could have the pedestal simulate a keystroke on the tablet, which the escape room app would recognize as the signal to trigger the final video. Is that convoluted and stupid? You bet! But it works.</p>
<p>That’s about it. In Mabel’s final, congratulatory video message, she tells the player that they can keep the magic mirror as a gift, and that she would cast a spell on it so it would show a poem for them every day. That was my way of revealing my wife’s gift. But it also served a useful purpose, because after the final video played the escape room app “magically” launched the poetry app. That way, after the escape room was done, the tablet was already setup and ready to be used for its original purpose.</p>
<h2 id="conclusion">Conclusion</h2>
<p>Overall I’m pleased with how the Magic Mirror turned out. Using a tablet made a lot of things easier, and opened up the possibility of using its camera as part of the escape room puzzles. It definitely wasn’t without its caveats though. RasPis are far easier to setup and less buggy compared to a rooted, bottom barrel tablet. Hard to beat that price though.</p>
<p>I hope this post has been fun, and maybe even inspires other escape room artists to build something neat.</p>Writing New Image Hashing Algorithms to Help a Yearbook Teacher2020-06-10T23:24:00+00:002020-06-10T23:24:00+00:00https://blog.hmac.io/2020/06/10/writing-new-image-hashing-algorithms-to-help-a-yearbook-teacher<p><img src="/assets/images/teacher-hash/header.jpg" alt="Title Image" /></p>
<p>A friend of mine came to me with a seemingly innocuous problem. They run the school’s yearbook class, and the students will often accidentally reuse the same photo in different places. A lot of time gets spent throughout the year tediously combing through and looking for these duplicates. Imagine having to memorize and identify all the different photos used throughout a 200 page book! So I agreed to help out and write a program to do the work instead. It seemed easy at first; extract images from a bunch of PDFs and throw some image hashing at the problem. How hard could it be? Oh how wrong I was…</p>
<p>There are many ways to find duplicates images, but the most robust methods involve perceptual hashes. These algorithms “hash” the image in such a way that an image will match even if it has been scaled, cropped, rotated, etc. Average Hash, for example, does this by scaling the input image down to 8x8 and outputting a 1 or a 0 bit for all 64 pixels based on whether that pixel is greater than or less than the average. Changing the image’s resolution, brightness, color, etc won’t affect such a hash much (if at all). For the most part, all perceptual image hashing algorithms work similarly.</p>
<p><img src="/assets/images/teacher-hash/average-hash.jpg" alt="Average Hash" /></p>
<p>Perceptual hashes are simple, powerful algorithms that get used in things like reverse image search engines. Yet they have a certain weakness that made them all unsuitable for the task at hand. The photos used in the yearbook were often <em>heavily</em> cropped and rotated. Consider this example:</p>
<p><img src="/assets/images/teacher-hash/extreme-crop.jpg" alt="Extreme Crop" /></p>
<p>In that example there are two images, each cropped from the same source image. They are duplicates, but they would not be detected by existing perceptual hashes because only <em>part</em> of the images match. Popular perceptual hashes also cannot handle large rotations. Even delving into recent perceptual hashing research, the algorithms just aren’t designed to handle extreme cases like these. Luckily reading those research papers inspired an idea for a potential solution.</p>
<p>The first and most important piece of the puzzle involves feature extraction. There is a field of computer vision based around extracting salient “features” from an image. Before the dawn of deep neural networks, these feature based algorithms were used for things like object detection. Given an image they spit out a set of landmarks, or “keypoints” as they’re called, in the image, and a “description” (feature vector) for each of those keypoints.</p>
<p><img src="/assets/images/teacher-hash/matches.jpg" alt="Feature Matching" /></p>
<p>The most important part is that the features these algorithms find are stable under various transformations. You can scale, rotate, translate, and otherwise manipulate the image and the same features would tend to be found. Perfect for our task! Looking again at the cropping example above, we would hope that any features it detects in the matching halves of the images will be exact matches. Perhaps we can find duplicates by looking for any images where at least some of the detected features match.</p>
<p>So now there’s a possible algorithm. Run feature detection to get a set of features for each image. Compare all possible pairs of images. In any given pair, if some amount of their features “match” then they might be a duplicate image. To keep things simple I ignored the locations of the features and only match based on their descriptions. So that gives us two bags of feature vectors that can be brute force compared. With a few experiments I was able to confirm that this algorithm works! When two images weren’t duplicates their features almost never matched up. When they were duplicates, at least a few features were near identical.</p>
<p><em>It’s true that the location of the features could be important. More correctly, the relative locations of matching features. It’s likely that considering this information would make a better matching algorithm, but it turns out to be good enough without that specificity.</em></p>
<p>Now I had an algorithm that <em>worked</em> where all other perceptual hashes failed. It wasn’t time for champagne just yet, though. There remained one more difficult challenge: search. These feature detection algorithms can spit out <em>thousands</em> of features per image. And I was working with a set of tens of thousands of images to compare. A naive search would involve searching every combination of every image (with n=10000 that would be 50 million comparisons). Every comparison involves yet another naive search to compare all the features (with just n=1024 features that’s half a million comparisons). That amounts to 25 trillion feature vector comparisons. Yikes.</p>
<p>To make this tractable the search needs to be less naive. There are many different tree and index based approaches that could be used. Locality Sensitive Hashing proved to be the winner. LSH is kind of like a HashMap. With a HashMap if two keys are exactly the same they’ll end up in the same bucket. So HashMaps can be used to quickly find exact matches. LSH can be used to find things that are <em>near</em> matches. Imagine being able to “look up” each feature vector and find other feature vectors that are similar to it. Perfect for our problem.</p>
<p>The version of LSH I used is quite clever and worth elucidating. The feature vectors themselves are 512-bit vectors where the similarity between two features is measured by the number of differing bits (the hamming distance). To index these using an LSH, these 512-bit vectors first get permuted (mix the bits up in a predetermined way) and then split into thirty two 16-bit pieces. Those 16-bit pieces are actually 32 different “hashes”, from which we build 32 different indexes. The result of this is that if two features differ by 31 bits or less they are guaranteed to end up in at least one bucket in the index together.</p>
<p>Why? Consider two features that differ by 1 bit. In that case, given the above procedure, it’s clear that those two features will still share 31 hashes. Hence they will end up sharing a bucket together. Similarly if they differ by 2 bits they will share at least 30 hashes. Even if two features differ by 31 bits they will still have at least one hash in common.</p>
<p><img src="/assets/images/teacher-hash/lsh-example.png" alt="Matching LSH Hashes" /></p>
<p>With this technique all features could be indexed, and the buckets of those indexes could be searched for matches. Because our distance threshold for matching features is less than 31, this method is guaranteed to find all matches. Most importantly this algorithm runs 100x faster than a naive search!</p>
<p>With this and liberal application of Rust, optimization, and multithreading the program went from spending several <em>days</em> of runtime to a couple <em>minutes</em>. It’s amazing what a little computer science can do, huh?</p>
<p>Those were the hardest pieces of this program. I also spent some time running experiments on a test dataset to pick the best feature extraction algorithm. There are many, but of all the ones in OpenCV, BRISK performed best in this application. The dataset also let me calibrate the permutation used in the LSH hashing, which turned out to be important since the bits in the feature vector are highly correlated. And finally the program has two tunable hyperparameters: Distance Threshold and Score Threshold. Distance Threshold is the threshold used for determining if two feature vectors are “similar”. Score Threshold is used for comparing potentially matching images. A score is calculated by dividing the number of matching features by the total number of features. If it’s greater than the threshold, the images match. By calibrating these hyperparameters the algorithm was tuned such that it caught all but one of the most pathological duplication tests, and let only a rare few false positives through.</p>
<p>The final program had a quick GUI thrown on it for selecting the set of PDFs to be processed, and to view and filter the results. My friend ended up using the app for this year’s yearbook and it performed splendidly. Not only did it catch obvious duplicates, it also caught a few really hard to find duplicates that a human would have likely missed. Nothing like inventing cutting edge computer vision algorithms to fix a school’s yearbook.</p>A Daydreaming AI for my Desk2020-06-08T16:11:00+00:002020-06-08T16:11:00+00:00https://blog.hmac.io/2020/06/08/a-daydreaming-ai-for-my-desk<p><img src="/assets/images/daydreaming-ai/dreamer.jpg" alt="Daydreaming AI" /></p>
<p>Using a Raspberry Pi and a 7” screen, I built a little AI that sits on my desk and daydreams to entertain and inspire me. Here’s how I did that.</p>
<p>This project is based on BigGAN, a neural network that is able to generate entirely made up, but realistic looking photos. One of the most interesting things about BigGAN, besides its impressive ability to generate convincing photos, is how it responds when it is “abused”. Normally BigGAN is fed random noise and a specific class of image that you want. But if you feed it a mixture of classes then it will find weird and wild ways to blend those mixtures of classes. What would the combination of a person and a broom look like? BigGAN will attempt to explain it to you, among other weird and bizarre combinations in the examples below:</p>
<p><img src="/assets/images/daydreaming-ai/weird-gan.jpg" alt="BigGAN Samples" /></p>
<p>I find these class mixtures to be deeply compelling. They reveal the creative side of the neural network, because it was never explictly trained to generate images like these. They are artifacts of its process, and those artifacts constantly surprise and inspire me. And so I set out to build a small picture frame that could sit on my desk and show a stream of these creative pictures. A daydreaming AI.</p>
<p>Normally, advanced neural networks like these require a big computer with a GPU, due to their memory footprint and computational requirements. But I would only be running infererence and I wouldn’t need to generate the photos quickly. There’s nothing special about a GPU; a neural network can just as happily run on a CPU. And the more recent Raspberry Pi 4 comes in 4GB versions with quad cores, which might <em>just</em> be enough to run the network. So, I gave it a try.</p>
<p>To my great surprise, the network runs in a reasonable amount of time, about 3 seconds to generate a 128x128 image. It should also technically have enough RAM to generate 256x256 images, but as far as I could tell the current Raspbian OS limits process memory to 2 or 3GB (the newer 64-bit Raspberry OS will supposedly raise this limit).</p>
<p><img src="/assets/images/daydreaming-ai/assembly.jpg" alt="Test Assembly" /></p>
<p>Here are the more technical details. For the screen I used the official Raspberry Pi 7” touchscreen, installed into a nice case. The BigGan-deep-128 version of the network is used, <a href="https://tfhub.dev/deepmind/biggan-deep-128/1">available on TensorFlow Hub</a>. Tensorflow installed on the Pi with no problems, although only older versions are available (in my case it installed 1.14). One major problem with running this network on the Pi is that it takes a few minutes just to load the model into memory (even when it’s cached locally). So to make it easy to mess around and tweak things I ran jupyter notebook on the Pi and wrote the code as a notebook.</p>
<p><a href="https://gist.github.com/fpgaminer/95f8df092c1d0154df5d970a2f82b07f">Here’s a gist with the final notebook</a>. It’s more-or-less the BigGAN example code, with the addition of a main loop for continously generating images. To keep things simple I don’t even run an X server on the Pi; the generated images are instead written directly to the framebuffer (see the <code class="language-plaintext highlighter-rouge">display_image_to_framebuffer</code> function). PIL is used to resize the images from 128x128 to 480x480, then numpy is used to pad that out to the native 800x480 resolution, and the bytes are reordered for the framebuffer’s native byteorder (which I think was ABGR). At that point the raw numpy array can be written directly to <code class="language-plaintext highlighter-rouge">/dev/fb0</code>.</p>
<p>The final piece is getting the notebook to run on startup. As far as I could find, jupyter notebook does not provide an easy way to start and run a notebook from the command line or programmatically (I guess you’d have to manually poke the notebook server API). So instead I have a simple script that runs on startup. First it runs: <code class="language-plaintext highlighter-rouge">venv/bin/jupyter nbconvert --to script Dream.ipynb</code> to convert the notebook to a normal Python script. Then it just runs the output: <code class="language-plaintext highlighter-rouge">venv/bin/python -u Dream.py</code>.</p>
<p><img src="/assets/images/daydreaming-ai/pickle.jpg" alt="Pickle Pepper" /></p>
<p>Overall this was a simple project without too many surprises (a rarity in computer science). I’ve had the little guy sitting on my desk for a few days now and I’m constantly surprised by the weird and wonderful things it dreams up. It’s nice to look over at it when I’m thinking about some challenging code.</p>Making Dumb Fans Smart Using Software Defined Radio2019-10-25T20:21:00+00:002019-10-25T20:21:00+00:00https://blog.hmac.io/2019/10/25/making-dumb-fans-smart-using-software-defined-radio<p><img src="/assets/images/smart-fan/fan.jpg" alt="Dumb Fan" /></p>
<p>This is the story of how I spent far too much time and effort making dumb ceiling fans smart using Software Defined Radio (SDR). It’s a longer, slightly technical post covering the RX and TX side of SDR, ancient signaling protocols, pain, agony, buggy firmware, and temperamental hardware. Hopefully the details of this journey will be useful to others getting into SDR, or at least entertaining to those with schadenfreude.</p>
<p>Let’s set the stage. Our house has a number of remote controlled ceiling fans, and I want to be able to control these fans from Alexa and HomeKit (Siri). Since they all have remotes, it should be possible to use SDR to emulate the signals the remotes would normally send and then I can wire up the SDR to a server for integrating into Alexa and HomeKit. Easy! …</p>
<p>First things first, let’s take a look at the remotes:</p>
<h2 id="the-remotes">The Remotes</h2>
<p><img src="/assets/images/smart-fan/remote.jpg" alt="Dumb Remote" /></p>
<p>The best way to learn more about any remote control is to check the FCC ID. All RF emitting devices in the U.S. have an FCC ID printed somewhere on them, and this ID is tied to an FCC report which will contain lots of juicy details about the device. My remotes have the ID: CHQ7083T. While there is lots of information in the report, the only thing we care about is the frequency the remote control is operating on: <code class="language-plaintext highlighter-rouge">303.217 MHz</code>.</p>
<p>Once I knew what frequency the remotes talk on, I could spy on it and begin decoding the protocol it uses to talk to the fans.</p>
<h2 id="decoding-the-signal">Decoding The Signal</h2>
<p><img src="/assets/images/smart-fan/rtl-sdr.jpg" alt="RTL-SDR" /></p>
<p>For this job I used the venerable <a href="https://www.amazon.com/gp/product/B00P2UOU72/ref=as_li_qf_asin_il_tl?ie=UTF8&tag=progranism-20&creative=9325&linkCode=as2&creativeASIN=B00P2UOU72&linkId=23356b068e7b514ec43607abc7e6afe3">RTL-SDR dongle</a>. These devices are plentiful, cheap, and versatile. I recommend picking one up even if you don’t plan on doing something crazy like me, since they’re so cheap and have such a wide variety of SDR uses (who doesn’t want to receive satellite images!?). Just make sure, when working with any SDR project, your hardware of choice can handle the frequencies you’re targeting (in this case 300MHz is well within range).</p>
<p>As for software I used CubicSDR. I found it to be an easy way to quickly scrub around and get decent recordings of the remote’s signal. There’s probably better options, but CubicSDR sufficed for this simple job.</p>
<p><img src="/assets/images/smart-fan/cubicsdr.jpg" alt="CubicSDR" /></p>
<p>Within CubicSDR I set the center frequency to 303.217 MHz and watched the plot as I pressed buttons on the remote. Sure enough there were short signals bursts, though slightly off from 303.217 MHz (which is to be expected, nothing is perfect, let alone the bottom dollar radio hardware in these remotes).</p>
<p>Figuring that the remotes used a simple protocol I configured CubicSDR for AM decoding to get a look at just the amplitudes of the signal. (Note: Don’t use the screenshot above as a reference, I adjusted bandwidth and set a manual gain for my actual recordings.) Taking a quick recording of the signal and bringing it into Audacity allowed me to begin figuring out what the remote was sending.</p>
<p><img src="/assets/images/smart-fan/audacity.jpg" alt="Audacity" /></p>
<p>The signal is based on an old IR remote protocol. Everything is composed of “pulses” (carrier on) and “spaces” (carrier off). The signal always begins with what’s called a “header” and “leading pulse”. In this case all of that is covered by a simple 400μs pulse, 300μs space, and finally a 700μs pulse. After that it sends the data payload. A 1 bit is sent by transmitting a 300μs space followed by a 700μs pulse. 0 bits are the opposite; a 700μs space followed by a 300μs pulse. Finally there is a long gap between repeated commands, about a 12,000μs gap. The remotes repeat the command that they’re sending as long as the button is held down, so in practice they get repeated 20 to 40 times.</p>
<p>As for the data payload itself, I took recordings of several different button presses to puzzle it out. If you open the battery compartment there is a set of four small switches that are used to program the remote to different fans. Because of this I knew four bits of the payload were likely a verbatim copy of these switches; a four bit address. This turned out to be true, which left 7 bits for a “command”. By trying all the different buttons each unique command was easily decoded.</p>
<p>Before I make this seem like it was easy, know that I spent the better part of a day getting the RTL-SDR and its drivers set up, figuring out CubicSDR, and pondering over the remote’s signals. The irregular timing of the header pulse was especially confusing at first.</p>
<h2 id="transmitting">Transmitting</h2>
<p><img src="/assets/images/smart-fan/ys1.jpg" alt="YARD Stick One" /></p>
<p>With the remote control protocol decoded, it was time to move on to the real meat of this project: emulating the remotes. For that I would need new SDR hardware. The RTL-SDR cannot transmit signals, so I grabbed a <a href="https://greatscottgadgets.com/yardstickone/">YARD Stick One</a>. It’s a bit limited, but good enough for this application and one of the cheapest options for SDRs with TX capability. I also bought the recommended telescoping antenna to go with it.</p>
<h4 id="warning"><strong>WARNING:</strong></h4>
<p><em>Transmitting SDRs are no joke. You <strong>must</strong> follow all relevant local and international regulations for the frequency bands you plan to transmit in. <strong>Seriously.</strong></em></p>
<p>It didn’t take long to get the YARD Stick One running. It uses <a href="https://github.com/atlas0fd00m/rfcat">rfcat</a> for its firmware and software, so you basically just plug it into a Linux machine and bring up the rfcat Python REPL to begin poking it. Stupidly easy. I configured it for OOK modulation, the right frequency, and 10000 baudrate. This puts the YS1 in a mode where we can send it raw bits to turn the carrier signal on and off at a granularity of 100us. Given the protocol we decoded it’s straightforward to generate a bitstream to emulate what we want. Here’s my ugly code:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">build_bits_hampton</span><span class="p">(</span><span class="n">data</span><span class="p">):</span>
<span class="c1"># First we build a bit string of the bits we want to send to the modem
</span> <span class="n">result</span> <span class="o">=</span> <span class="s">""</span>
<span class="c1"># header (400 us pulse, 300 us space)
</span> <span class="n">result</span> <span class="o">+=</span> <span class="s">"1"</span> <span class="o">*</span> <span class="mi">4</span> <span class="o">+</span> <span class="s">"0"</span> <span class="o">*</span> <span class="mi">3</span>
<span class="c1"># plead (700 us pulse)
</span> <span class="n">result</span> <span class="o">+=</span> <span class="s">"1"</span> <span class="o">*</span> <span class="mi">7</span>
<span class="c1"># data
</span> <span class="k">for</span> <span class="n">x</span> <span class="ow">in</span> <span class="n">data</span><span class="p">:</span>
<span class="k">if</span> <span class="n">x</span> <span class="o">==</span> <span class="s">"1"</span><span class="p">:</span>
<span class="n">result</span> <span class="o">+=</span> <span class="s">"0"</span> <span class="o">*</span> <span class="mi">3</span> <span class="o">+</span> <span class="s">"1"</span> <span class="o">*</span> <span class="mi">7</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">result</span> <span class="o">+=</span> <span class="s">"0"</span> <span class="o">*</span> <span class="mi">7</span> <span class="o">+</span> <span class="s">"1"</span> <span class="o">*</span> <span class="mi">3</span>
<span class="c1"># gap
</span> <span class="n">result</span> <span class="o">+=</span> <span class="s">"0"</span> <span class="o">*</span> <span class="mi">120</span>
<span class="c1"># Pad to a multiple of 8 so we can build bytes
</span> <span class="k">while</span> <span class="p">(</span><span class="nb">len</span><span class="p">(</span><span class="n">result</span><span class="p">)</span> <span class="o">%</span> <span class="mi">8</span><span class="p">)</span> <span class="o">!=</span> <span class="mi">0</span><span class="p">:</span>
<span class="n">result</span> <span class="o">+=</span> <span class="s">"0"</span>
<span class="n">bytecount</span> <span class="o">=</span> <span class="nb">len</span><span class="p">(</span><span class="n">result</span><span class="p">)</span> <span class="o">//</span> <span class="mi">8</span>
<span class="c1"># Now convert into a string of bytes
</span> <span class="n">final</span> <span class="o">=</span> <span class="s">''</span>
<span class="k">for</span> <span class="n">i</span> <span class="ow">in</span> <span class="nb">range</span><span class="p">(</span><span class="n">bytecount</span><span class="p">):</span>
<span class="n">bits</span> <span class="o">=</span> <span class="n">result</span><span class="p">[</span><span class="n">i</span><span class="o">*</span><span class="mi">8</span><span class="p">:(</span><span class="n">i</span><span class="o">+</span><span class="mi">1</span><span class="p">)</span><span class="o">*</span><span class="mi">8</span><span class="p">]</span>
<span class="n">value</span> <span class="o">=</span> <span class="nb">int</span><span class="p">(</span><span class="n">bits</span><span class="p">,</span> <span class="mi">2</span><span class="p">)</span>
<span class="n">final</span> <span class="o">+=</span> <span class="nb">chr</span><span class="p">(</span><span class="n">value</span><span class="p">)</span>
<span class="k">return</span> <span class="n">final</span></code></pre></figure>
<p>I wrote a quick Python script to automate all this and send whatever command I wanted to any particular fan, began testing, and everything worked great. Project done! End of post…</p>
<h2 id="debugging-firmware">Debugging Firmware</h2>
<p><img src="/assets/images/smart-fan/foobar.jpg" alt="Car on fire" /></p>
<p>Well it <em>would</em> be, in some fantasy land where all hardware and software is perfect. Unfortunately, the YS1 is a “plug-and-pain” device. Works out of the box for delivering endless amounts of pain.</p>
<p>My script would work once or twice, kind of, but never consistently and the YS1 would frequently lock up and stop responding. Since rfcat is sparsely documented I figured I was just using the library wrong and spent the next few hours toying with different combinations of API calls, debugging the USB stack in case something was wrong with my particular Linux drivers, etc, etc. All roads led to the same chaos. I also discovered, using the RTL-SDR to check what the YS1 was actually sending out, that often it would sometimes go into a mode where it would repeat the same, old command no matter what data I would tell it to send. Complete, utter chaos.</p>
<p>After much debugging (and even dropping the project for a few months) I determined that the YS1 only freaked out when large packets of data were given to it. The ceiling fans need commands repeated 20 to 40 times in a row to ensure that they reliably receive the signal. That meant sending large packets of data to the YS1 which, for whatever reason, causes it to go crazy or crash.</p>
<p>There was no way around having to repeat the commands; the fans wouldn’t reliably respond otherwise. And I couldn’t use a Python loop to do it because Python was too slow and inconsistent. Too long a gap between repeated commands and the fans can become confused.</p>
<p>So what to do? Turns out the <code class="language-plaintext highlighter-rouge">RFxmit</code> function, which is used to tell rfcat to transmit data, has a <code class="language-plaintext highlighter-rouge">repeat</code> argument. I could give it the small, 31 byte payload and have the YS1 itself repeat it 40 times. Because of the small payload size this didn’t cause the YS1 to crash or behave oddly. Yay! …Only problem now was that the <code class="language-plaintext highlighter-rouge">repeat</code> argument didn’t actually do anything! The YS1 just wouldn’t repeat the command more than once no matter what.</p>
<p>Luckily, the discovery of <code class="language-plaintext highlighter-rouge">repeat</code> led me to this <a href="https://github.com/atlas0fd00m/rfcat/issues/2">open issue on the rfcat Github repo</a>. At this point I had already invested too much time into my crazy fan project … so why not dig deep into the internals of the firmware!?</p>
<p>According to the Github issue there are really two problems here. First, the firmware ignores the <code class="language-plaintext highlighter-rouge">repeat</code> argument entirely: <code class="language-plaintext highlighter-rouge">transmit(&buf[6], len, 0, offset)</code>. Second, if one tries to patch that “silly” mistake, ala <code class="language-plaintext highlighter-rouge">transmit(&buf[6], len, repeat, offset)</code>, they discover the firmware starts behaving weirdly and crashing (surprise surprise).</p>
<p>With some effort I got my machine set up for firmware development (rfcat requires an ancient version of sdcc), and spent a few hours to learn the code’s layout. Once all the relevant pieces to the issue were found I began to strip the firmware down. All code that wasn’t strictly necessary to reproduce the bug was removed by hand, reducing the firmware to something simpler that could be experimented on. I spent a lot of time digging around in the TX loops where the firmware interacts with the radio module, figuring the bug must be in there, but never found anything. Instead, by complete accident I stumbled onto the real cause. It was back at the line referenced in the Github issue:</p>
<figure class="highlight"><pre><code class="language-c" data-lang="c"><span class="n">len</span> <span class="o">=</span> <span class="n">buf</span><span class="p">[</span><span class="mi">0</span><span class="p">];</span>
<span class="n">len</span> <span class="o">+=</span> <span class="n">buf</span><span class="p">[</span><span class="mi">1</span><span class="p">]</span> <span class="o"><<</span> <span class="mi">8</span><span class="p">;</span>
<span class="n">repeat</span> <span class="o">=</span> <span class="n">buf</span><span class="p">[</span><span class="mi">2</span><span class="p">];</span>
<span class="n">repeat</span> <span class="o">+=</span> <span class="n">buf</span><span class="p">[</span><span class="mi">3</span><span class="p">]</span> <span class="o"><<</span> <span class="mi">8</span><span class="p">;</span>
<span class="n">offset</span> <span class="o">=</span> <span class="n">buf</span><span class="p">[</span><span class="mi">4</span><span class="p">];</span>
<span class="n">offset</span> <span class="o">+=</span> <span class="n">buf</span><span class="p">[</span><span class="mi">5</span><span class="p">]</span> <span class="o"><<</span> <span class="mi">8</span><span class="p">;</span>
<span class="n">txTotal</span><span class="o">=</span> <span class="mi">0</span><span class="p">;</span>
<span class="n">buf</span><span class="p">[</span><span class="mi">0</span><span class="p">]</span> <span class="o">=</span> <span class="n">transmit</span><span class="p">(</span><span class="o">&</span><span class="n">buf</span><span class="p">[</span><span class="mi">6</span><span class="p">],</span> <span class="n">len</span><span class="p">,</span> <span class="mi">0</span><span class="p">,</span> <span class="n">offset</span><span class="p">);</span></code></pre></figure>
<p>Replace <code class="language-plaintext highlighter-rouge">0</code> with <code class="language-plaintext highlighter-rouge">repeat</code> and the firmware explodes. Replace the call with something stupid like <code class="language-plaintext highlighter-rouge">transmit(&buf[6], len, (buf[3] << 8) | buf[2], offset);</code> and … everything works fine. <strong>!?!</strong> I reverted my working copy of rfcat, applied that single line change, and confirmed that the problem was still “fixed”.</p>
<p>Such a change to the firmware <em>shouldn’t</em> have any impact. However it’s possible that rfcat is suffering from either a compiler bug, or a deepseeded stack corruption bug that gets avoided by whatever slightly different code the compiler generates in my modified version. Actually fixing the bug would require more knowledge of the rfcat project than I had any desire to have, and being already several evenings deep into this torturous bug, I decided it was best to take the workaround and move on.</p>
<p>On the brightside, while digging through the rfcat firmware I also found that I was using the API a bit wrong. Rfcat is sparsely documented, and with plenty of bad example code around it wasn’t too surprising. First off, it seems you are supposed to put the radio into IDLE mode before configuring it, which I wasn’t doing. Second, I was erroneously calling <code class="language-plaintext highlighter-rouge">setModeTX</code> before my calls to <code class="language-plaintext highlighter-rouge">RFxmit</code>. It seemed like the right thing to do, given the function’s name. But what that function actually does is tell the rfcat firmware what state to put the radio into <em>after</em> <code class="language-plaintext highlighter-rouge">RFxmit</code> (or other radio calls). Oops? The rfcat firmware automatically puts the modem into TX mode when you call <code class="language-plaintext highlighter-rouge">RFxmit</code>. You don’t need to do anything special. And without that silly call to <code class="language-plaintext highlighter-rouge">setModeTX</code> it happily puts the radio back to IDLE afterwards, as intended.</p>
<p>So this was my eventual setup code:</p>
<figure class="highlight"><pre><code class="language-python" data-lang="python"><span class="k">def</span> <span class="nf">configure_radio</span><span class="p">(</span><span class="n">d</span><span class="p">,</span> <span class="n">fan_type</span><span class="p">):</span>
<span class="c1"># Don't know if this is necessary, but some documentation seems to indicate that the radio should be IDLE before configuring it
</span> <span class="n">d</span><span class="p">.</span><span class="n">setModeIDLE</span><span class="p">()</span>
<span class="k">while</span> <span class="p">(</span><span class="n">d</span><span class="p">.</span><span class="n">getMARCSTATE</span><span class="p">()[</span><span class="mi">1</span><span class="p">]</span> <span class="ow">not</span> <span class="ow">in</span> <span class="p">(</span><span class="n">MARC_STATE_IDLE</span><span class="p">,)):</span>
<span class="k">pass</span>
<span class="k">if</span> <span class="n">fan_type</span> <span class="o">==</span> <span class="s">'hampton'</span><span class="p">:</span>
<span class="n">d</span><span class="p">.</span><span class="n">setFreq</span><span class="p">(</span><span class="mi">303217000</span><span class="p">)</span> <span class="c1"># This is the frequency the fans should be using, based on the FCC ID
</span> <span class="c1">#d.setFreq(303682000) # This is closer to the real-world frequency
</span> <span class="k">elif</span> <span class="n">fan_type</span> <span class="o">==</span> <span class="s">'hunter'</span><span class="p">:</span>
<span class="n">d</span><span class="p">.</span><span class="n">setFreq</span><span class="p">(</span><span class="mi">434000000</span><span class="p">)</span>
<span class="n">d</span><span class="p">.</span><span class="n">setMdmModulation</span><span class="p">(</span><span class="n">MOD_ASK_OOK</span><span class="p">)</span>
<span class="n">d</span><span class="p">.</span><span class="n">setMdmDRate</span><span class="p">(</span><span class="mi">10000</span><span class="p">)</span> <span class="c1"># 100 microseconds per bit
</span> <span class="c1">#d.makePktFLEN(len(final)) # As far as I know, this isn't needed because PKTLEN gets set and handled during RFxmit in the firmware
</span> <span class="n">d</span><span class="p">.</span><span class="n">setMdmSyncMode</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="c1"># Tell firmware not to transmit its own sync words
</span> <span class="n">d</span><span class="p">.</span><span class="n">setPower</span><span class="p">(</span><span class="mi">255</span><span class="p">)</span>
<span class="n">d</span><span class="p">.</span><span class="n">setMdmChanSpc</span><span class="p">(</span><span class="mi">100000</span><span class="p">)</span>
<span class="n">d</span><span class="p">.</span><span class="n">setMdmChanBW</span><span class="p">(</span><span class="mi">100000</span><span class="p">)</span>
<span class="n">d</span><span class="p">.</span><span class="n">setChannel</span><span class="p">(</span><span class="mi">0</span><span class="p">)</span> <span class="c1"># I don't think this is necessary, but just in case...</span></code></pre></figure>
<p><em>Note that you’ll need to adjust <code class="language-plaintext highlighter-rouge">d.setPower(255)</code> for your application/region; I calibrated it lower later.</em></p>
<p>With the patched firmware and my improved script, everything <em>finally</em> worked well. The YS1 was no longer hardlocking and I could control the fans reliably.</p>
<h2 id="integrating-into-the-smarthome">Integrating Into The Smarthome</h2>
<p><img src="/assets/images/smart-fan/completed.jpg" alt="Completed Project" /></p>
<p>At this point the SDR portion of the project was working reliably. It was time to hook up to the smart ecosystem. I moved the YS1 to a Raspberry Pi 3 and the hacky Python script was refactored into a web server that presented a simple API for the smart home integrations to talk to. <a href="https://pypi.org/project/fauxmo/">fauxmo</a> and <a href="https://github.com/nfarina/homebridge">Homebridge</a> were set up on the same Pi. Fauxmo provides the (somewhat hacky) integration to Alexa. Homebridge provides the HomeKit integration (I used the HttpAdvancedAccessory to get it to talk to the fan control server). (<strong>Security Note:</strong> I keep the fan control server isolated to the Pi. Homebridge is authenticated and reasonably secure. But Fauxmo doesn’t have any authentication and exposes itself to the local network. Be wary!).</p>
<p>I can, somewhat proudly, say that this concoction of hardware and software has been solid since putting it together. It is certainly not without its flaws, though. The software has no way of knowing the state of the fans, which both Alexa and HomeKit would prefer to know (so, you know, you can see if your fans are on…). For now my server just always reports back “off”.</p>
<p>There’s no good way to get this information. You can store local state in the server, but if someone operates the fans manually then it goes out of sync with the real world. There are commercial solutions that do more-or-less everything I’ve done here, and they solve this problem by operating an RX that listens in case someone manually operates a fan.</p>
<p>As mentioned at the beginning of this post, there are many <em>better</em> ways to accomplish everything I’ve done here. Still there is something special about telling Siri to turn on the fans, knowing that it’s your magic in the background making things work.</p>
<p>I would like to end this post with huge thanks to the engineers and developers behind RTL-SDR, CubicSDR, YARD Stick One, rfcat, fauxmo, and Homebridge. Nothing is without flaws, but this project would not have been possible without the work of those people. I only hope that my journey can contribute to the knowledge of others.</p>
<p>Happy hacking.</p>