{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreidvvqtpw5qsbspiuz4gimt6cz736ymy2znjia6er45xh7uzypl3zy",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mjmjcpwf32j2"
  },
  "path": "/t/extract-polarity-scores-from-probabilites/175298#post_2",
  "publishedAt": "2026-04-16T12:27:16.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "Statsmodels",
    "Hugging Face",
    "scikit-learn"
  ],
  "textContent": "It seems that a rather primitive method is still the de facto standard…\n\n* * *\n\nThe easiest good default is:\n\n  * map `negative`, `neutral`, `positive` to `-1`, `0`, `+1`\n  * then take the **probability-weighted average**\n  * for 3 classes, that simplifies to **`polarity = P(positive) - P(negative)`**. (Statsmodels)\n\n\n\nThat is usually the best starting point because it is:\n\n  * bounded in `[-1, 1]`\n  * easy to explain\n  * sensitive to uncertainty\n  * naturally pulled toward `0` when the model is unsure or mostly neutral. (Statsmodels)\n\n\n\n## What models actually output\n\nTransformer sentiment models do **not always directly output probabilities**. In the standard Hugging Face setup, the model returns **logits** , which the docs describe as classification scores **before SoftMax**. The pipeline then turns those into postprocessed scores. (Hugging Face)\n\nSo in practice, the usual flow is:\n\n  1. model outputs logits\n  2. softmax converts them to class probabilities\n  3. you collapse those probabilities into one polarity score. (Hugging Face)\n\n\n\n## The recommended formula\n\nFor a 3-class sentiment model:\n\n  * `P(neg)`\n  * `P(neu)`\n  * `P(pos)`\n\n\n\nuse:\n\n**`polarity = P(pos) - P(neg)`**\n\nThat is the same as taking the expected value of the scale `[-1, 0, +1]`. Neutral does not disappear conceptually. It just has value `0`, so it automatically shrinks the score toward the center. (Statsmodels)\n\n### Quick examples\n\n  * `neg=0.80, neu=0.10, pos=0.10` → polarity `-0.70`\n  * `neg=0.20, neu=0.60, pos=0.20` → polarity `0.00`\n  * `neg=0.05, neu=0.15, pos=0.80` → polarity `0.75`\n\n\n\nThat matches the intuition most people want. More positive mass pushes right. More negative mass pushes left. More neutral mass pulls inward. This follows directly from the ordered-label interpretation. (Statsmodels)\n\n## Why this works\n\nThis is just an **expected-value** calculation.\n\nIf the classes are ordered, you can assign each class a location on a sentiment line and take the weighted average using the class probabilities. For the common 3-class case, the simplest anchors are `-1, 0, +1`. (Statsmodels)\n\nThat said, there is an important warning: ordered labels do **not** automatically come with a built-in numeric distance. The statsmodels ordinal-model docs state that ordinal labels are ordered, but the labels have **no numeric interpretation besides the ordering**. So the move from labels to `-1, 0, +1` is a modeling choice. It is usually a sensible one, but it is still a choice. (Statsmodels)\n\n## Is this “standard”?\n\nNot in the sense of one official universal rule. There is no general transformer standard that says “all sentiment probabilities must be collapsed this exact way.” Hugging Face documents logits, probabilities, and task setup, but not one canonical polarity-collapse formula. (Hugging Face)\n\nIn practice, though, the **probability-weighted ordered scale** is the cleanest and most defensible approach. For 3-way sentiment, that means `P(pos) - P(neg)`. (Statsmodels)\n\n## Binary case\n\nIf your model only has `negative` and `positive`, then:\n\n  * `P(pos) - P(neg)` works\n  * and because the probabilities sum to 1, it is the same as `2 * P(pos) - 1`\n\n\n\nSo the binary case is simpler. Most of the ambiguity comes from handling `neutral`, or more than three sentiment levels. (Hugging Face)\n\n## More than 3 classes\n\nIf your classes are ordered, like:\n\n  * very negative\n  * negative\n  * neutral\n  * positive\n  * very positive\n\n\n\nthen use the same idea with more anchors, for example:\n\n  * `-1.0`\n  * `-0.5`\n  * `0.0`\n  * `0.5`\n  * `1.0`\n\n\n\nand take the weighted average. (Statsmodels)\n\nThis is usually the right extension. Just document your anchors, because equal spacing is an assumption, not something the labels guarantee by themselves. (Statsmodels)\n\n## The biggest technical caveat: calibration\n\nThis part matters more than most people expect.\n\nEven if `P(pos) - P(neg)` is the right formula, the final scalar is only as trustworthy as the underlying probabilities. Scikit-learn’s calibration guide explains that models can produce poor probability estimates, and that calibration methods are used to improve them. The current sklearn calibration tools support **isotonic** , **sigmoid** , and **temperature scaling**. (scikit-learn)\n\nThis means:\n\n  * a model can choose the right top label\n  * but still be overconfident or underconfident\n  * and then your polarity magnitude can look more precise than it really is. (scikit-learn)\n\n\n\nFor multi-class sentiment, temperature scaling is especially relevant because sklearn describes it as a natural way to obtain better calibrated multi-class probabilities, and its `CalibratedClassifierCV` docs explain how it applies `softmax(logits / T)`. (scikit-learn)\n\n## What to do in practice\n\n### Good default\n\nUse:\n\n**`polarity = P(pos) - P(neg)`**\n\nThis is the best default for a normal 3-class sentiment model. It is easy to read, easy to explain, and usually matches intuition. (Statsmodels)\n\n### Better if the score matters a lot\n\nIf you will use the score for:\n\n  * thresholds\n  * rankings\n  * time-series monitoring\n  * downstream decision-making\n\n\n\nthen calibrate the probabilities on a held-out validation set first. Scikit-learn’s calibration docs explicitly recommend fitting calibration on data independent of the classifier training data. (scikit-learn)\n\n### Best if you truly want a continuous target\n\nIf the real task is inherently scalar, Hugging Face supports regression-style sequence-classification settings as well. In other words, if you truly want one continuous polarity output, it can be cleaner to train a regression model instead of forcing a classifier into post-hoc scalar conversion. The model-output docs note that sequence-classification outputs can also be used for regression when `num_labels == 1`. (Hugging Face)\n\n## One alternative people use\n\nSometimes people use:\n\n`(P(pos) - P(neg)) / (P(pos) + P(neg))`\n\nThis measures sentiment direction **conditional on being non-neutral**.\n\nIt can be useful, but it is not the best default. It tends to exaggerate tiny scraps of positive or negative evidence when most probability mass is neutral. That is why the simple `P(pos) - P(neg)` score is usually safer and easier to interpret. This is an inference from the formulas and the ordered-scale setup, rather than a separate official rule. (Statsmodels)\n\n## Simple code\n\n\n    def polarity_from_probs(p_neg: float, p_neu: float, p_pos: float) -> float:\n        return p_pos - p_neg\n\n\nGeneral ordered-class version:\n\n\n    import numpy as np\n\n    def ordered_polarity(probs, anchors):\n        probs = np.asarray(probs, dtype=float)\n        anchors = np.asarray(anchors, dtype=float)\n        return float(np.dot(probs, anchors))\n\n\nThese implementations are just the expected-value idea applied directly to ordered sentiment classes. (Statsmodels)\n\n## Bottom line\n\nUse this unless you have a strong reason not to:\n\n**`polarity = P(positive) - P(negative)`**\n\nThat is the clearest recommended mapping for `negative / neutral / positive` probabilities into one score in `[-1, 1]`. It is not a universal official standard, but it is the most natural and defensible default. The two main warnings are:\n\n  * your labels must really be ordered\n  * your probabilities may need calibration before the magnitude is trustworthy. (Statsmodels)\n\n\n\n## Best references\n\n  * Hugging Face model outputs: logits are scores **before SoftMax**. (Hugging Face)\n  * Hugging Face course and docs for sequence classification mechanics. (Hugging Face)\n  * Statsmodels OrderedModel: labels are ordered, but not inherently numeric. (Statsmodels)\n  * Scikit-learn probability calibration guide. (scikit-learn)\n  * Scikit-learn calibration API and temperature scaling support. (scikit-learn)\n  * Scikit-learn 3-class calibration example. (scikit-learn)\n\n",
  "title": "Extract polarity scores from probabilites"
}