I built ARSENIC - a tool to analyse what actually changes when you upgrade models

Hi.

Totally agree with your points, it is as you say - a different brain. arsenic is an attempt, if you’re relying on a answer being consistent across a model upgrade , to provide some indication that the answer might be subtly different and so your system needs tweaking. Easier if fact based, less so if its conjecture or opinion.

When someone identifies an AI by name and expects to find equivalent traits in another model, they fundamentally misunderstand the nature of AI - this 100%, unless you’re under the hood trying to understand the fundamentals of what it is doing it can (and sometimes is) misunderstood as ‘magic’

Thanks for the comments! -Mark