AI as a centralizing force

Sraars:

We won’t be able to afford them.

NPU is literally in every new processor. As time advances they will get more powerful and be able to handle a lot of the simple tasks people throw at them.

Homeprivacy:

Absolutely. I think Locally hosted LLM with Cloud PII redaction for any inference that has to hit a frontier model is not only prudent but necessary for those that want to maintain privacy going forward. The tradeoff of privacy for productivity doesn’t have to happen, but most people doin’t know that.

There’s always going to be room for both. Some things a small model can do quite easily, like semantic searching eg “find me the invoice where I bought a shoe” should never leave the system. You might not have the word “shoe” in the file, but maybe “sneaker” instead. A small language model is perfect for this. You can use a sliding window that clears as it goes since it doesn’t need to reference those documents again. This can all be done easily on a small NPU.

There are plenty of other examples where you need huge context windows, especially when dealing with massive datasets. In those cases, it makes more sense to use the cloud, especially for data that isn’t sensitive.

Contrary to popular belief, public LLMs such as ChatGPT and Gemini do not train on active conversations in real-time. Training on this type of data would lead to model collapse. For example, a model trained on low-quality data from someone who thinks that an LLM is intelligent and now their boyfriend would not be useful to anyone.