Fast structured output could be a genuinely new computational primitive for intelligence. So far we've built a lot of programs on top of autoregressive token generation, and they all look like fancy chatbots. Leaning hard into structured output might conceivably unlock a bunch of non-chatbot use cases for AI.
[...]
If you want fast, parallelized structured output against limited choices, you don't strictly need to do autoregressive generation at all. You can simply prefill the response with "choice": " and generate one token, restricted to the user-provided choices. [...]