Streams of AGI

OpenAI's SDK currently doesn't support streaming for models GPT-3.5-Turbo or GPT-4. Yes, very sad, anyway. I decided to DIY this shit. Backend On Node you can use the fetch api and get a ReadableStream of bytes as a response. Here we use the fetch api to make a call to the OpenAI server and get a ReadableStream in response. It needs to be decoded into plaintext so we do that with pipeThrough. The OpenAI streaming endpoints return the response as an event stream. The next step is to parse the event stream, and connect it to a Response stream. For parsing the event stream we can use eventsource-parser. Installation: npm i eventsource-parser --save We take the individual events and parse the data, which then gets written to the response stream. Once the end of the event stream is reached, we can end the response stream. Now we can hook up the express endpoint with the chat completion stream. Here we can see that the response stream is just a response object we get access to inside an express endpoint callback. Frontend This is the developer experience I was looking for: I wrote a few hooks that abstract away all of the ReadableStream synchronization logic, and some nice-to-have data fetching wrappers. useStreamingQuery Hook This is one of the wrappers that get exposed from readable-hook. Internally it uses useReadable, which takes a stream producer (the fetch API in case of the useStreamingQuery hook), and returns a query trigger and the streamed data. Installation: npm i readable-hook --save This is a simplified version of the hook. Check out readable-hook for more details. We setup intermediate state (on line 2), and the query function that handles fetching data from the stream periodically (on line 3). Both are then returned from the hook (on line 22). Even though the internals of the hook are fairly straightforward, the hook makes it much easier to re-use streaming data in other parts of the app. Once the hook is initialized, we can read the values from streamingData, and update the UI. The hook takes care of all the heavy-lifting. After all this hard work, we can finally have streaming response from the OpenAI chat completion apis.