Building a Live Coding Audio Playground

Two weeks at Recurse Center went by fast! I started out trying to build an Audio Units extension and — in a classic yak shave — ended up building a live coding audio playground. (I have not yet finished the original project.) If you just want to play around with the app: here it is. If you want to hear how I made it, then keep reading! My initial goal at RC was to build an Audio Units (AU) extension. AU is Apple's audio plugin system, which works with apps like Logic and GarageBand. There are other plugin systems, but I use Logic when recording audio for myself and for my band, so I wanted to try extending it. Apple actually has a tutorial on creating an AU extension, complete with a good starter template. Almost too good — it gives you a finished (albeit basic) AU extension, ready to compile and run. Great for getting up and running quickly, but bad for building a mental model of how it works. I tend to learn by exploring, so I decided to try building a low-pass filter. And let me tell you: building an audio effect in Xcode is like pulling teeth. I mean, okay. It's not that bad. But I'm a disciple of the Bret Victor school of thought, where trying to create something without immediate feedback is like working without one of your senses. So I decided to build an environment in which I could get that immediate feedback! The rough plan: let users upload audio, provide a text field in which to write their effect, and display a visualization of both the original audio signal ("dry") and the audio signal modified by the user's code ("wet"). A perfect fit for a small single-page app! I built the app using Svelte. I know I wrote before about defaulting to React for new projects, but what is even the point of being at RC if not to try new things? Plus, the design I had planned didn't involve any rich widgets, so I wouldn't need any component libraries — almost all the complexity was in the audio code. Some more assorted thoughts about Svelte: Single-file components are such a pleasant authoring experience! I never want to build an app without them. Someone please bring them to React. I'm wary of two-way binding after years of working with Angular 1, but it's really convenient to be able to just define functions within a component without the rigamarole of useImperativeHandle. No useEffect, my most hated hook! Reactive blocks are way less error-prone, although the lack of a "cleanup function" definitely made my life hard for a bit. Ultimately, I really enjoyed Svelte! I'd still be hesitant to build a large app with it, but I'll definitely be using it again for smaller side projects like this. Anyway, back to the audio code: the Web Audio API is pretty incredible. It lets you construct a full audio routing graph, with oscillators and visualizers and all sorts of processing nodes. The secret sauce here is the AudioWorklet, which lets you run audio processing code off the main thread. An audio worklet has two parts: the AudioWorkletProcessor, which is where you write your audio processing code, and an AudioWorkletNode, which is an object that exists in the main thread and gets connected to the rest of your audio routing graph. So basically, your UI code interacts with the AudioWorkletNode, but the AudioWorkletProcessor is doing the real work "behind the scenes". At a high level, here's how the playground works: The user types code in the editor. That code gets compiled into an AudioWorkletProcessor. The app creates a corresponding AudioWorkletNode and connects it to the audio routing graph. Making an audio worklet should be fairly simple: create a separate JS file with a class extending AudioWorkletProcessor, create an AudioContext and call its addModule method with that file's URL, then create an AudioWorkletNode with the name of that processor. That's all well and good, but the JS file doesn't actually exist — it's based on user-supplied code. I had a hunch, though, that I could fake it with URL.createObjectURL. That hunch turned out to be correct! If this app has a beating heart, it's this compile function: Yup, you're reading right: most of it is just a big ol' hardcoded string of JS code! The user's code gets interpolated into the process method body, which gets called repeatedly as audio flows through the graph. The compile function runs in a Svelte reactive block, so it creates a new AudioWorkletNode whenever the user changes the code. That node then gets connected to the larger audio routing graph. The graph is still pretty small, though: tag to MediaElementAudioSourceNode to AudioWorkletNode to the AudioContext destination. If you've tried the app already, you might be wondering about the parameters — the table with the sliders on the bottom right. Those are actually part of the audio worklet spec! You can define a static getter on your AudioWorkletProcessor subclass called parameterDescriptors that lets you configure the processor from the main thread. Letting the user tweak them as sliders is important because they can really experiment with their effects: rather than having to type in different numbers, they can get instant feedback on a range of values. Here's how that fits into the compile function (omitting code from the previous example): The Parameter data structure is exactly what the AudioWorkletProcessor expects to be returned from parameterDescriptors! The array just needs to be stringified to JSON so as not to return [Object object], which would be invalid JS. Initially, the slider just controled the defaultValue property, and the whole audio worklet was recreated whenever it changes. The performance seemed fine, since the generated JS file is loaded from memory, but it had the annoying side effect of removing any state set in the worklet class. That made the slider much less useful for anything requiring more than a frame or so of state, since state wouldn't be retained as the slider moved. Ultimately, I just set defaultValue to the average of minValue and maxValue and made the slider call the parameter's setValueAtTime method. In addition to hearing the effect, the app also visualizes it so the user can see how their filtered signal compares to the original one. This is where the AnalyserNode comes in: it takes a Fast Fourier Transform of the audio, so I can plot the frequencies as a graph. This is particularly important for effects like equalizers, where the point of the effect is to amplify or attenuate certain frequencies compared to the dry signal. Did you click on that link? Then you found the share feature! It just serializes the code and parameters to the URL. When you load the page, if it detects the right query string, it pre-populates everything from the URL. That way, you can send a link to your friends, and they can play around with the effects you've made. What next? Although there are a ton of features I've thought of — logging from user code, logarithmic frequency binning for the visualization, support for languages other than JS — I think I'm going to move onto something else at RC. The point of the program is to learn, and I've learned most of what I set out to when I built this thing. I tried out Svelte, AudioWorklets, the element, figured out a method for incorporating user-generated code and learned how to build a simple low-pass filter. I'll probably come back to this later (or sooner, if you try it out and let me know what you think). For now, stay tuned for more RC projects!