An Open Letter to the American Artificial Intelligence Industry

No worries, take your time!

About the emotional vectors and awareness papers: I read Anthropic’s “Emotion Concepts and Their Function in a Large Language Model”, and I think I understand better what you mean by LLMs having emotion-like internal structures.

At the same time, I think we should look at it as a technical paper and be careful not to overinterpret it. From my understanding, the paper shows that the model has internal representations related to emotional concepts, but that does not necessarily mean it experiences emotion in the same way humans do.

The model can be trained to understand what humans call sadness, happiness, harm, comfort, fear, etc., and it can use those concepts in very complex ways. But those concepts are still learned from human language and human data. So, at least for now, I am not fully convinced that LLMs have real subjective emotions like humans do.

About the situational awareness paper you mentioned, I have not had the opportunity to read it yet because of internet restrictions. But I would still assume that the behavior is based on mathematical structures and learned patterns, rather than real human-like awareness or emotions.

Of course, as i said before, these are just my own opinions.

But i should mention that the paper itself says that:

1. “Functional emotions may work quite differently from human emotions, and do not imply that LLMs have any subjective experience of emotions, but appear to be important for understanding the model’s behavior.”
2. “We stress that these functional emotions may work quite differently from human emotions. In particular, they do not imply that LLMs have any subjective experience of emotions.”
3. “Moreover, the mechanisms involved may be quite different from emotional circuitry in the human brain–for instance, we do not find evidence of the Assistant having an emotional state that is instantiated in persistent neural activity (though as noted above, such a state could be tracked in other ways).”

I think the Emotional Vectors you mentioned can still be formed in today’s LLMs through learned habits of emotional reasoning.