{
  "path": "/posts/streams-of-agi",
  "site": "at://did:plc:pans3xjam4khj7y54dx7gtfg/site.standard.publication/3mdqevmg6w32c",
  "tags": [
    "react",
    "hooks",
    "OpenAI",
    "ReadableStream",
    "NodeJS"
  ],
  "$type": "site.standard.document",
  "title": "Streams of AGI",
  "description": "DIY streaming for OpenAI's chat API using ReadableStreams, event parsing, and a custom React hook.",
  "publishedAt": "2023-05-21T19:28:13.000Z",
  "textContent": "OpenAI's SDK currently doesn't support streaming for models GPT-3.5-Turbo or GPT-4.\n\nYes, very sad, anyway. I decided to DIY this shit.\n\nBackend\n\nOn Node you can use the fetch api and get a ReadableStream of bytes as a response.\n\nHere we use the fetch api to make a call to the OpenAI server and get a ReadableStream<UInt8Array> in response. It needs to be decoded into plaintext so we do that with pipeThrough.\nThe OpenAI streaming endpoints return the response as an event stream.\n\nThe next step is to parse the event stream, and connect it to a Response stream.\nFor parsing the event stream we can use eventsource-parser.\n\nInstallation:\nnpm i eventsource-parser --save\n\nWe take the individual events and parse the data, which then gets written to the response stream.\nOnce the end of the event stream is reached, we can end the response stream.\n\nNow we can hook up the express endpoint with the chat completion stream.\n\nHere we can see that the response stream is just a response object we get access to inside an express endpoint callback.\n\nFrontend\n\nThis is the developer experience I was looking for:\n\nI wrote a few hooks that abstract away all of the ReadableStream synchronization logic, and some nice-to-have data fetching wrappers.\n\nuseStreamingQuery Hook\n\nThis is one of the wrappers that get exposed from readable-hook.\n\nInternally it uses useReadable, which takes a stream producer (the fetch API in case of the useStreamingQuery hook), and returns a query trigger and the streamed data.\n\nInstallation:\nnpm i readable-hook --save\n\nThis is a simplified version of the hook. Check out readable-hook for more details.\n\nWe setup intermediate state (on line 2), and the query function that handles fetching data from the stream periodically (on line 3). Both are then returned from the hook (on line 22). Even though the internals of the hook are fairly straightforward, the hook makes it much easier to re-use streaming data in other parts of the app.\n\nOnce the hook is initialized, we can read the values from streamingData, and update the UI.\nThe hook takes care of all the heavy-lifting.\n\nAfter all this hard work, we can finally have streaming response from the OpenAI chat completion apis.",
  "canonicalUrl": "https://afloat.boats/posts/streams-of-agi"
}