How do you reduce latency in real-time AI applications?

We are dealing with similar issues at my company. We need near-realtime audio processing - transcribe and then analyze the text in various ways. A delay of 1 or 2 minutes is fine for us but the pipeline we need to run is quite long. Since we are using Gemini, We are currently experimenting with provisioned throughtput to see if it helps at least stabilize the response times. For some parts of the pipeline we are actually moving back to smaller local models, not necessarily generative.