I want to build an AI/LLM app that preserves privacy. Please roast my idea :)

This can be built in a matter of days in reality. Given the low cost barrier to entry, are you sure this hasn’t been done before? In fact if all you’re doing is just acting as a web wrapper around an existing SLM, I’m 99% sure it must already exist.

It's FOSS – 5 Jun 24

12 Tools to Provide a Web UI for Ollama

Don't want to use the CLI for Ollama for interacting with AI models? Fret not, we have some neat Web UI tools that you can use to make it easy!

As for remote computing, that’s essentially opening yourself to tons of compute cost for other people. Your service will be abused. If you plan on charging for this, then at that point I’d rather pay for a state of the art model from Anthropic then for a poor performing SLM on a remote server. Otherwise Lumo from Proton somewhat meets this criteria already, just not as FOSS, and I get it for free as part of my subscription to Proton.

MrCakeBoss:

Background is that we like local AI

If you are saying we as in a business partner, do market research. Is there a demand for people wanting SLMs on remote machines that also aren’t apt enough to self host it or simply utilize existing services?

If it’s we as in PG, I’d say there is likely already an existing good enough FOSS solution. If you want to make a new one, ask yourself if you can meaningfully contribute to an existing project or ask yourself if what you have is truly a differentiator.