Reduce in-app AI cost without reducing quality

Divide and conquer… I would definitely question whyyou need the huge payloads like this. Just hearing the size and problem description, I feel like the underlying process is not defined enough to make it simpler, more deterministic, and potentially on some steps, executable by cheaper models or even good old code. Here is an example: From what I read you use full potential model with all schemas for all tools in loop which basically drains your tokens. But if you look closer at that process, here is what you may see: 1. You bring some input in 2. Based on that input something decides what tool to use 3. Once the tool is selected something four months the tool request 4. The request is sent for tool execution 5. The response is collected 6. The response is verified for conformity 7. The sanitized response is passed further down the pipe. So instead of sending all contacts plus all tools to highly expensive model, 1. you extract sub process of selecting the tool by describing your tools in plain language like tool name and short description which you present with your input to a cheaper model which will make the only decision about what tool to use (this may be even fine tuned, but I don’t think the task is complicated enough for that). 2. Once you have the tool name you take the same input and call the tool formatting model (again it can be way cheaper because you don’t need a huge model for that) that would return you the exact call to make. 3. Value rumble tool and use code to validate the response against the schema. 4. If the validation fails you retry (potentially switching the model to a more capable one) 5. Once you have the clean response, you pass it further down the pipe (the step 7 from above). Weight less tokens, cheaper models faster responses, if the quality falls on cheaper models find tuning will save you. Then, you just do your math.