A first look at token efficiency

Needn’t be a new language. You could do grammar prompting for example if your problem is sufficiently constrained.

people.csail.mit.edu

Lecture 12

I’ve been reading a fair bit about how the classical synthesis community has adapted to LLMs and most of the literature is around correctness (rightfully so) but not much about “search” efficiency. The measures you suggest make sense. Hard thing would be defining a sufficiently representative set of tasks for a benchmark like that.