{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreiey4p2agdlabkzflyvhx5slxb4qg2ezzwazo3eghcf7ggbd6mvofe",
    "uri": "at://did:plc:pgryn3ephfd2xgft23qokfzt/app.bsky.feed.post/3mq2idpkc47x2"
  },
  "path": "/t/good-data-to-test-tensor-based-soft-sparsity-and-other-sparsity-models/177522#post_5",
  "publishedAt": "2026-07-07T10:22:51.000Z",
  "site": "https://discuss.huggingface.co",
  "tags": [
    "Rotation to Simple Loadings Using Component Loss Functions: The Oblique Case",
    "Hyperspectral unmixing for Raman spectroscopy via physics-constrained autoencoders",
    "reproduction code on GitHub",
    "RamanBench",
    "(click for more details)",
    "SubstrateMixRaman",
    "Dataset repository and Dataset Card"
  ],
  "textContent": "Oh. Hmm, I see. In that case, how about something like this?:\n\n* * *\n\nThat Jennrich reference seems to narrow the empirical question in a useful way.\n\nI also found Jennrich’s later paper, Rotation to Simple Loadings Using Component Loss Functions: The Oblique Case. It explicitly cites Katz and Rohlf’s Functionplane paper and extends the component-loss-function treatment to oblique rotation.\n\nMy reading is that this shifts the most interesting benchmark away from merely asking whether an ideal concave CLF can recover an exactly perfect simple structure under ideal population conditions. The more informative empirical questions may be:\n\n  * what happens with approximate rather than perfect simple structure;\n  * whether small but genuine components or cross-loadings are preserved;\n  * sensitivity to finite-sample noise and spectral artifacts;\n  * local solutions and initialization;\n  * behavior near ill-conditioned transformations;\n  * whether the metric and inverse-metric machinery prevents the particular form of basis collapse you have in mind;\n  * computational behavior at thousands of spectra and thousands of variables.\n\n\n\nFor the Raman application, I found one dataset that seems unusually close to the scale and comparison setting you described: the experimental **Sugar Mixtures** data used in Hyperspectral unmixing for Raman spectroscopy via physics-constrained autoencoders, with its reproduction code on GitHub.\n\nIt is also now included in the larger RamanBench collection.\n\nSubset | Spectra | Variables | Acquisition\n---|---|---|---\nHigh SNR | 1,960 | 2,000 | 5 s integration\nLow SNR | 7,840 | 2,000 | 0.5 s integration\n\nThe experiment used mixtures of:\n\n  * glucose;\n  * sucrose;\n  * fructose;\n  * maltose;\n  * water.\n\n\n\nThe mixture concentrations are known, and additional reference spectra were measured from pure solutions. This seems particularly useful because it provides both a relatively favorable case with reference endmembers and a more difficult blind-unmixing case without adding those references to the observed mixture data.\n\nThe RamanBench regression version exposes four sugar-concentration targets; its benchmark excludes water as a supervised target because that target did not pass its learnability criterion. For an unmixing experiment, however, the original paper and repository are probably the more relevant references because they retain the five-component physical formulation and the pure-solution spectra.\n\nSo this may be a more direct practical candidate than MicroMass for the Raman side:\n\n  * it is real experimental Raman data;\n  * it is approximately the requested thousands-by-thousands size;\n  * mixture composition is known;\n  * high- and low-SNR conditions are available;\n  * pure reference spectra exist;\n  * there are published comparison methods and open reproduction code.\n\nA possible evaluation using the Sugar Mixtures data (click for more details) A matching synthetic Raman benchmark (click for more details)\n\nThere is also a second Hugging Face candidate, SubstrateMixRaman, with:\n\n  * 6,960 spectra;\n  * 1,878 columns;\n  * independently varied concentrations of eight fermentation-related metabolites;\n  * additional variation from mineral salt medium and antifoam;\n  * a CC BY 4.0 license.\n\n\n\nThat looks useful for a larger concentration-recovery or robustness test, especially because low concentrations were deliberately represented and the nuisance components affect overall signal intensity.\n\nI would place it after the Sugar Mixtures experiment, however. The Sugar data have a clearer published blind-unmixing setup, pure-solution reference spectra, established unmixing baselines, and matched high-/low-SNR conditions. SubstrateMixRaman appears more naturally suited to concentration prediction and practical stress testing unless an appropriate pure-reference library is also available.\n\nSo a relatively compact default route could now be:\n\n  1. **Fully synthetic simple-structure test**\nKnown loadings, known transformation, conditioning ladder.\n\n  2. **RamanSPy synthetic mixtures**\nKnown endmembers and abundances, with realistic Raman artifacts.\n\n  3. **Sugar Mixtures, high SNR**\nReal spectra under the easier acquisition condition.\n\n  4. **Sugar Mixtures, low SNR**\nReal noise-robustness comparison.\n\n  5. **Reference-assisted and blind variants**\nSeparate discovery of endmembers from fitting known endmembers.\n\n  6. **SubstrateMixRaman or another RamanBench dataset**\nLarger practical stress test.\n\n\n\n\nThis also gives a clean way to separate several claims:\n\nClaim | Best test\n---|---\nExact simple-structure recovery | Fully synthetic loading benchmark\nCollapse avoidance | Controlled transformation-conditioning ladder\nRaman artifact robustness | RamanSPy synthetic mixtures\nReal endmember/abundance recovery | Sugar Mixtures\nNoise robustness | High- versus low-SNR Sugar Mixtures\nLarge-data behavior | SubstrateMixRaman or wider RamanBench\n\nRegarding publication on Hugging Face: if the artifact is primarily source code, mathematics, and a reusable executable component, a Model Card can certainly serve as an overview and entry point. For the generated tests and result fixtures themselves, a separate Dataset repository and Dataset Card may be easier for others to load and compare.\n\nA practical division could be:\n\n  * Zenodo: archival releases, mathematics, DOI;\n  * source repository: Fortran, Octave wrapper, build and test scripts;\n  * HF Dataset repo: synthetic inputs, known truths, experimental configurations, outputs and metrics;\n  * Model Card or project page: explanation, intended uses, limitations, and links to the other artifacts.\n\n\n\nThe Jennrich result seems to provide a strong theoretical connection. The Sugar Mixtures data may then provide a rather good empirical connection: not merely “does an ideal CLF recover perfect structure?”, but “what happens with weak components, experimental noise, missing pure endmembers, local solutions, and conditioning in a reproducible Raman problem?”",
  "title": "Good data to test tensor based soft sparsity and other sparsity models"
}