{
  "$type": "site.standard.document",
  "content": "---\ntitle: \"On livecoding annotations and visualisations\"\ndescription: \"A response to Charlie Roberts' essay on code annotations, exploring the\n  tension between visualising program state and editing live code.\"\ntags:\n  - livecoding\n---\n\nI've just finished reading Charlie Roberts'\n[interactive web essay on annotations and visualisations for live code](https://charlieroberts.github.io/annotationsAndVisualizations/).\nIf you haven't read it, go do that now---and make sure you have some\nspeakers/headphones to listen to the code examples, because it would be a real\nshame to read it without watching/listening to his ideas in action. All the code\nsamples (with the visual annotations) are live-editable, both running \"whole\nblocks\" and re-evaluating individual expressions with `ctrl+enter`.\n\nThe essay lays out some guiding principles:\n\n> There are three principles that guide the design of the presented annotations\n> and visualizations.\n>\n> 1. If a value changes over time (typically related to musical progression or\n>    signal processing), display its current state, either by adding a\n>    visualization / annotation or by modifying the source code to reflect the\n>    current value.\n> 2. Make annotations and visualizations as proximal as possible to the code\n>    fragment responsible for generating the data they are representing.\n> 3. In addition to displaying when values change, whenever possible also\n>    provide some indication of when a value is being read, in particular when\n>    there is a direct affect on musical / sonic output.\n\nIt then goes on to show some example visualisation in\n[gibberwocky](https://gibberwocky.cc/), both in an \"all in\" demo and then in\nbite-sized listings which show off the different specific ideas. In general, I\nreally like the ideas, and it's something Andrew Sorensen and I have written\nabout before in our paper\n[Visual Code Annotations for Cyberphysical Programming](/research/#swiftVisualCodeAnnotations2013)\n(2013) in the _1st International Workshop on Live Programming (LIVE)_ (it's\nreference #4 in the essay). I'll refer to this a bit in this post, so let's call\nit \"the LIVE paper\".\n\nI also think that the three points listed above are pretty solid, especially in\na multimedia livecoding context (maybe even in a broader context). One thing I\nlike about the visual annotations provided is that they're mostly ASCII (or\nASCII-ish). This is not so important when deploying them in the web browser\n(since you can do so much fancy styling stuff with CSS & js these days) but it's\nreally important when dealing with... _ahem_, more venerable editors. I ended up\nhaving to use some unholy Emacs hacks with\n[overlays](https://www.gnu.org/software/emacs/manual/html_node/elisp/Overlays.html)\nto get the original annotations discussed in the LIVE paper working.\n\nI think that displaying hidden state in comments is a good compromise, and\navoids the need for fancy \"extra-textual\" overlays. Not that overlays aren't\nsometimes useful, but there's a lot you can show with inline `text-decoration`\nhacks and adding a few comments to provide ascii text to decorate (when it isn't\nexplicitly represented in the first place).\n\n**But**, (you can probably tell that there was going to be a \"but\" somewhere)\nmany of the cool annotations displayed don't work while the code is being\nedited[^broken]. In some cases they actually break the code (try editing one of\nthe `'x*ox*xo--x*x*o'` patterns while the code's running---you'll end up with\nnew stuff in your pattern that you didn't put there).\n\n[^broken]: Charlie, if I'm doing it wrong, please let me know :)\n\nThe problem isn't so obvious when all the code listings are fully-formed on page\nload, and you can just play them as is. But when you try and mess with the code\nthen you'll see what I mean (again, I really suggest that you try it---it's\nsuper-cool being able to mess around with the live code in the browser).\n\nI feel this particularly keenly because I'm a clean-slate livecoder (as is\nCharlie), so I'm _always_ moving through an incomplete code state until I get\nsomething which will even run. This isn't just a problem for clean-slate\nlivecoding, though---even tweaks to existing code which introduce \"bad\" code\nstates (from the visualisation's perspective) will cause these issues.\n\nWe talked about this in the LIVE paper---the fact that there's a distinction\nbetween the \"state of the world\" vs \"state of the code\". This is a fundamental\nchallenge for the sorts of inline code visualisation/annotations shown in the\nessay, because it's using the **code** as the \"raw material\" for displaying\ninformation about the **world** (beat/timing and other \"hidden variables\", audio\nengine state and output, etc.).\n\nThere are a few different ways to tackle this problem:\n\n1. mark some annotations as \"safe\" for code being edited, and some \"unsafe\"\n2. when code is being edited, turn off all annotations (or at least for that\n   expression)\n3. some sort of \"grand unified theory\" of the delta between the current code\n   state and the current execution state, and reconcile these to provide a\n   maximal set of acceptable visualisations (this approach includes a\n   [coq](https://coq.inria.fr/) program to formally verify that you're doing\n   everything right)\n\nI think the pragmatic choice is #2, as long as the \"disabled edits\" section is\nkept as small as possible. It's still going to be a pain, though, because when\nI'm livecoding I'm tweaking stuff all the time, so it's likely that a lot of the\ncode will spend a lot of the time with the visualisations disabled.\n\nI'll leave #3 up to people smarter than me 😁 (although perhaps there are\nheuristics which could do a decent job).\n\nFinally, I can identify with Charlie when he writes:\n\n> But, in the end, the feedback provided by these annotations and visualizations\n> have become a critical part of my live coding practice.\n\nI get this---in my experience (when I've had even more limited annotations than\nthe ones he shows in the essay) are just as much for my benefit as the audience.\nLivecoding is hard, and any extra information you can get about what's going on\nwith your code is super helpful.\n\nIn principle visual code annotations can be even more useful to the livecoder\nbecause they allow her to \"audition\" algorithmic changes to the code\nwithout[^feedback] actually eval-ing the code and changing the music. Here's an\nexample from the essay:\n\n```js\nEuclid(9, 16); /* 1010110101011010 */\n```\n\nI might not always know what the `9,6` euclidean rhythm is, but I'm by now\nfairly used to looking at `10100010` sequences and \"hearing\" the rhythm in my\nhead. I could poke around with the parameters in a live set, exploring the\nparameter space (and thinking through the effect it'll have on the music) and\nthen only evaluating the code when I'm satisfied. That's super powerful---the\nequivalent of the DJ cueing the next track with one can on their ear, and one\near in the club---something which I don't have currently in my livecoding setup\n(although others might).\n\n[^feedback]:\n    Yes, I know that the real-time feedback of hearing the sound is crucial, and\n    I'm not for a second saying that we do away with it, but there are some\n    situations where I want to check what the result of an algorithmic/parameter\n    change might be without inflicting it on the audience.\n\nYou can probably tell that I think there's a productive research agenda\nhere---and I hope Charlie continues with it. I hope to help out myself, too. I\nguess my main point is just to shout from the rooftops:\n\n> any code visualisation/annotation techniques must be robust for **code which\n> is currently being edited**\n\nI'm not just talking about technical issues, either; obviously any\ndemo/prototype is going to have those, but they're fixable. I think there are\ndeeper issues with trying to use _live_ text as both the description of program\nbehaviour and as a \"view\" on the hidden state of the program.\n\nAnyway, this is just a blog post, so I'm off the hook with regard to rigour,\naccountability and just general good scholarship, right 😜\n\nAs an addendum, a few thoughts on web publishing. I love that this\nessay/paper is published online---the interactive examples are\ncrucial to getting the point across. I know that some conferences & journals\nthese days allow html submission (nicer for reading on mobile, anyway) and other\nmultimedia artefacts (audio/video recordings) but it's still hard to get\ntraction for this sort of rich, interactive in-browser work. The fact that at\nthe end Charlie has to say:\n\n> If you're going to cite this website in an academic paper, please consider\n> also citing either reference #1 or reference #7 given above; citations of such\n> papers count more in academia than citations of a website. Plus, there's\n> further information in them not covered in this essay. Thank you!\n\nOh well. Mad props to Charlie for putting this out there for comment and\ndiscussion, and hopefully there are a new generation of publications[^distill]\nwhere this stuff can be front-and-centre, not just a weird \"supplemental web\nmaterials\" section to a traditional pdf.\n\n[^distill]:\n    [Distill](https://distill.pub/) is great, but it's pretty DL/AI-focussed.\n    The livecoding community needs something similar (although it does privilege\n    livecoding environments which work in the brower, so that's not ideal\n    either. Hmm.)\n",
  "createdAt": "2026-05-13T23:15:00.916Z",
  "description": "A response to Charlie Roberts' essay on code annotations, exploring the tension between visualising program state and editing live code.",
  "path": "/blog/2019/01/31/on-livecoding-annotations-and-visualisations",
  "publishedAt": "2019-01-31T00:00:00.000Z",
  "site": "at://did:plc:tevykrhi4kibtsipzci76d76/site.standard.publication/self",
  "tags": [
    "livecoding"
  ],
  "textContent": "A response to Charlie Roberts' essay on code annotations, exploring the tension between visualising program state and editing live code.",
  "title": "On livecoding annotations and visualisations"
}