{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreiamg2pu7gzxear67ixno32r56ynv37myj4nyp3vn7jbkqb3dmvbye",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mpzu7oejdy42"
  },
  "path": "/t/good-data-to-test-tensor-based-soft-sparsity-and-other-sparsity-models/177522#post_2",
  "publishedAt": "2026-07-07T04:10:51.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "UCI MicroMass dataset",
    "Functionplane paper",
    "(click for more details)"
  ],
  "textContent": "Hmm… For now, maybe something like this?:\n\n* * *\n\nI think **MicroMass is useful** , but I would not make it the first or only benchmark for this engine.\n\nThe UCI MicroMass dataset is reasonably high-dimensional at 1,300 variables, and it has more structure than an ordinary classification dataset: the associated study used 571 reference spectra from 20 bacterial species, plus 360 spectra from controlled two-species mixtures at nine nominal concentration ratios. That makes it a potentially good **real-data sanity check**.\n\nHowever, it does not provide a known true latent loading matrix, it is smaller than the thousands-by-thousands scale you are looking for, and the number of species is not automatically the true latent dimension. I would therefore use it after a small synthetic benchmark rather than as the primary test.\n\nMy default route would be:\n\nStage | Main purpose\n---|---\n1. Rotation-only synthetic benchmark | Test recovery of known sparse structure and avoidance of basis collapse\n2. Sampled synthetic benchmark | Add sampling noise and factor-extraction error\n3. MicroMass | Test behavior on structured real spectra and controlled mixtures\n4. Large real dataset | Test runtime, memory use, stability, and scaling\n\nThe useful distinction, at least as I understand the earlier description of the engine, is:\n\n> **Does it recover sparse/simple structure?**\n\nand separately:\n\n> **Does it avoid singular, duplicated, or badly conditioned bases while doing so?**\n\nThose are not the same success criterion. A method can produce very sparse-looking loadings by driving the transformation toward a nearly singular solution, while another method can remain numerically stable without recovering the planted sparse structure.\n\nFor literature searches, nearby established terms appear to be **oblique simple-structure rotation** , **sparse-loading rotation** , **hyperplane rotation** , and **factor-rotation local solutions**. I would keep your term “soft sparsity,” but adding those terms in the documentation would help future readers find relevant comparisons. Your original Functionplane paper is itself a useful anchor: it describes a smooth oblique simple-structure criterion that can be optimized by gradient methods.\n\nA practical first release would not need a huge dataset. A small benchmark with known answers would probably demonstrate the distinctive behavior more clearly.\n\n1. A small benchmark separating sparsity recovery from collapse avoidance (click for more details) 2. What I would record: four separate result families (click for more details) 3. Comparison methods, controls, and local solutions (click for more details) 4. Adding sampled data without losing the known answer (click for more details) 5. Where MicroMass fits (click for more details) 6. A larger Hugging Face stress test (click for more details) 7. A reproducible result package (click for more details)\n\n## A manageable first version\n\nIf I were reducing this to the smallest useful public example, I would start with:\n\n  1. one planted 15-by-3 or 60-by-6 loading pattern;\n  2. B0, B1, B2, B3, and a four-level B4 conditioning ladder;\n  3. an identity start plus a modest random-start saturation experiment;\n  4. unrotated, quartimin, oblimin, geomin, one sparse-loading baseline, and the proposed method;\n  5. separate tables for truth recovery, simple structure, and conditioning;\n  6. one MicroMass mixture-path demonstration;\n  7. fixed seeds and machine-readable result files.\n\n\n\nThat would already answer several important questions:\n\n  * Does the engine recover a known sparse pattern?\n  * Does it preserve genuine small cross-loadings?\n  * Does it falsely sparsify a dense pattern?\n  * Does it remain nonsingular under a controlled conditioning stress?\n  * Does the metric/inverse-metric mechanism visibly change that behavior?\n  * Are the results stable across starts?\n  * Does the method show coherent behavior on real controlled mixtures?\n\n\n\nAfter that, increasing the synthetic dimensions to thousands of variables and cases is mostly a scaling exercise rather than a search for a perfect real dataset.\n\nSo my answer to the original dataset question would be:\n\n  * **Yes, MicroMass is worth using** , especially because of its pure endpoints, mixture series, strain pairs, and replicates.\n  * **No, I would not use it as the sole or primary proof of sparse-factor recovery.**\n  * **For the exact requested dimensions and a known latent truth, a generated benchmark is probably the cleanest solution.**\n  * **The most informative first benchmark would measure sparse-structure recovery and collapse avoidance as two separate outcomes.**\n\n",
  "title": "Good data to test tensor based soft sparsity and other sparsity models"
}