David's Blog

Stripping Exif tags, magick-rust, and nix-built docker images

My site now supports stripping Exif tags (and others) on media when uploaded via the Micropub media endpoint I've implemented. This was important to me to ensure that, when possible, identifying information like location is removed from any images I upload. This is doable before uploading the images but integrating it into the web endpoint makes it much more likely that it'll happen (especially if I ever get the micro pub implementation up to the point where something like a mobile upload can happen).

The code itself to do so within the upload handler was pretty straightforward so the more interesting bit is how I got to the point of being able to write that code and also packaging it up once it compiled.

Challenge 1: Stripping tags in Rust

I did a fair amount of looking around for solutions to this problem. Since my server is implemented in Rust, I was looking for Rust bindings to strip tags as reliably as possible (and didn't want to have to go down the rabbit hole of implementing it myself, for now).

There were some pure-Rust image crates but none seemed to have support for what I wanted to do. There was also `magick-rust` which provides Rust bindings for the ImageMagick MagickWand library. I would have preferred bindings to other portions of the ImageMagick API as the MagickWand stripping bindings seem to do a bit more other stuff than I really wanted but I still wasn't quite willing to build bindings to the rest of the API. The goal was stripping tags without implementing a whole lot of stuff in Rust in order to do so.

One problem was that the strip functionality provided in this API wasn't yet surfaced in the `magick-rust` crate. This seemed like something I would be able to add without going down a huge rabbit hole so I proceeded with this approach (the alternative at this point, given no other Rust options, was to shell out to ImageMagick itself so this was preferable). It also gave me a chance to learn about `rust-bindgen` as that is how this crate was built. A small PR later, the crate would support stripping tags! Excitingly, that PR was quickly accepted and merged.

Along the way, I also learned that default `rust-bindgen` cargo features involve building `clap`, a CLI parsing crate that's somewhat large (but also quite handy). This was surprising to me and stood out as my server doesn't currently have that crate as a dependency, so I dug in there. I learned that the default build of rust-bindgen also builds it as a CLI which requires that crate. I haven't made a PR out of this change yet but it seemed fairly easy to turn off that extra functionality, so I did so in a branch.

Challenge 2: Building a functioning Docker image

The other issue I ran into with this "small" bit of functionality was that I needed to be able to have the proper version of ImageMagick available at runtime so that my server could actually do the stripping (this dependency is not statically linked). This lead to a host problems.

First, the apt-based repositories the existing image build had access to were still on a 6.x version of ImageMagick. The `magick-rust` crate requires a build in the 7.0.x range or later. At first, I tried to work around this by building ImageMagick from source during the image build. This didn't seem too bad initially, I didn't have a whole lot of trouble getting the source downloaded, built, and installed within my Dockerfile. The trouble was, even once that was happening, the server was erroring when the image tag stripping code was called.

Looking into this some more, it turned out that a bunch of "optional" ImageMagick dependencies were not available (and had been mentioned as such during the `configure` script) and this meant that it didn't know how to read a bunch of common image formats. I had a really rough time trying to make all those deps available to the build. I tried to list them all out in an `apt-get` command, did things like installing the 6.x version to make sure its deps were available, and more, all to no avail.

Separately, I've been experimenting with Nix package manager (and NixOS) recently and had no problems building and running the server from within a `nix-shell` environment. Nix had a new enough ImageMagick and making the symbols available to the server at runtime wasn't really much of a problem. I knew that there was a way to build Docker images with nix and so, failing all of my more traditional attempts at packaging my server into a container image with its new ImageMagick dependency, I decided to look at packaging the server in a Docker image with nix.

This first involved being able to build my micropub server via `nix-build`. I'd written some other package derivations at this point so it seemed feasible but I hadn't done any Rust packages yet. It turned out not to be too bad, I ended up primarily following the `naersk` readme (and transitively, `niv`'s too). That paired with some additional suggestions from the NixOS wiki and Xe's blog (which has also been quite helpful with Nix in general) got me going. Once I had the project being built via `nix-build`, adding the Docker bits was relatively straightforward! (I leaned on essentially the same resources there, with the addition of the official manual).

Once all that was said and done, I had a minimal Docker image that could run my site and (finally) strip Exif tags from images that I upload!

I have more thoughts about Nix package management that I'm working to write up. It appeals to me in quite a few ways but some of its requirements also make me question whether I want to stick with it. In a lot of ways, the problems are endemic to package management or configuration management tools in general and so I might just have to decide to live with this one given all the benefits it brings. More on that soon, I hope.