{
  "$type": "site.standard.document",
  "bskyPostRef": {
    "cid": "bafyreiayxddfddtqjq2vpafeyf3vifbymtfoalj736lrahwbjkjdbl6m7u",
    "uri": "at://did:plc:25rdn5elo5izoxrmtis34zuk/app.bsky.feed.post/3mpv4edw65ul2"
  },
  "coverImage": {
    "$type": "blob",
    "ref": {
      "$link": "bafkreifmbvd7ygpd5yut4rdyvgm5vqh5yv3eejr75li7nkitez7doavq54"
    },
    "mimeType": "image/webp",
    "size": 75016
  },
  "path": "/teolex2020/can-you-build-an-alternative-to-llms-8-months-200-failed-experiments-one-wall-2-3776",
  "publishedAt": "2026-07-05T07:22:18.000Z",
  "site": "https://dev.to",
  "tags": [
    "ai",
    "machinelearning",
    "llm",
    "rust",
    "AuraSDK"
  ],
  "textContent": "_This is part 2 of a research series documenting an attempt to build something adjacent to — and in some ways alternative to — large language models. Part 1: \"My Synthetic Eval Said 30/30. LoCoMo Said 0.13.\" The code that survived lives in AuraSDK._\n\nNot a chatbot wrapper.\n\nNot another prompt stack.\n\nNot a vector database with a nicer UI.\n\nThe question was narrower and more dangerous:\n\n> Can a non-neural, CPU-only system accumulate experience, mutate its own internal state, and improve future behavior without retraining an LLM?\n\nAfter eight months, the honest answer is not \"yes\".\n\nThe honest answer is:\n\n\n\n    some mechanisms survived,\n    most carriers failed,\n    and the wall became precise.\n\n\nThe wall is not storage. Storage is easy.\n\nThe wall is transferable causal transition:\n\n\n\n    condition -> action -> consequence -> when NOT to apply it\n\n\nThat is the thing I kept failing to preserve.\n\n##  1. What I Mean by \"Alternative to LLMs\"\n\nThe phrase is too broad, so I need to narrow it.\n\nI was not trying to build a new frontier model. I was not training a transformer from scratch. I was not trying to beat GPT-class models at language.\n\nThe target was smaller:\n\n> Find a knowledge substrate that behaves like a tiny, mutable set of weights.\n\nThe required properties were:\n\n  * it changes after experience;\n  * it survives restart;\n  * it changes future behavior without code changes;\n  * it transfers to unseen but related cases;\n  * it fails shuffled/null controls if the signal is fake;\n  * it is compact in a behavioral sense, not merely compressed bytes;\n  * it abstains when unsupported instead of confidently guessing.\n\n\n\nIn other words, I was looking for something between memory and weights.\n\nMemory stores what happened.\n\nWeights change what happens next.\n\nThe experiment was to find a symbolic or hybrid substrate that could cross that line.\n\n##  2. Why Normal Memory Was Not Enough\n\nThe first tempting answer is always memory:\n\n  * append the conversation;\n  * summarize it;\n  * store facts;\n  * retrieve relevant chunks;\n  * build a graph;\n  * add embeddings;\n  * add scores;\n  * add typed edges.\n\n\n\nAll of those are useful engineering tools. None of them automatically become knowledge.\n\nThe distinction became brutal:\n\n\n\n    storage can preserve an event;\n    knowledge must change the next action.\n\n\nA log can say:\n\n\n\n    candidate A failed in situation X\n    candidate B worked in situation X\n\n\nBut a useful system must do more:\n\n\n\n    in a new situation X',\n    recognize what is shared with X,\n    avoid A-like actions,\n    try B-like actions,\n    and know when the analogy no longer applies.\n\n\nThat last line is where most designs died.\n\n##  3. The Carrier Bakeoff\n\nI tested many candidate \"carriers\" of knowledge. A carrier is the internal form that is supposed to hold experience and make it reusable.\n\nThe early list looked reasonable:\n\nCarrier | What it preserved | Where it failed\n---|---|---\nappend-only memory | event history | did not transfer to unseen cases\nsurface compression | shorter text | compressed language, not behavior\ngraph / n-gram links | co-occurrence | became an index, not a causal model\nroute state | role and expectation | changed behavior, but stayed narrow\ntyped edges | relation labels | helped only when the relation grammar was already given\ngraded vectors | magnitude and pressure | carried direction, not an executable mechanism\nliving cells | local conflict | transferred some signal, but could transfer the wrong action\nJacobian-like footprint | local numeric effect | worked on linear repairs, failed on branching and bit-shift semantics\ntopology fields | learned attraction basins | signal on synthetic worlds; on real cargo snippets — 0/5 correct, 5/5 abstain\n\nThe pattern was consistent.\n\nEach carrier preserved one aspect of consequence:\n\n  * that something was supported;\n  * that something was refuted;\n  * that a choice had pressure;\n  * that two roles were related;\n  * that a numeric direction changed;\n  * that a local conflict existed;\n  * that a state transition had a shape.\n\n\n\nBut none of the early carriers preserved the full causal transition.\n\nThat is the difference between:\n\n\n\n    \"A was good near B\"\n\n\nand:\n\n\n\n    \"When input has property P and state has condition C,\n    action A changes state S in direction D,\n    unless boundary condition B is active.\"\n\n\nThe second form is much closer to knowledge.\n\nIt is also much harder to acquire without already having a model that can do it.\n\n##  4. Signals That Were Real but Not Enough\n\nSome results were not failures in the simple sense. They were real signals.\n\nOne route-state experiment proved that internal state can change behavior without changing code. The system saw support and refutation, mutated its state, and later selected differently. The shuffled-state control failed. That was important.\n\nBut it did not prove a replacement for weights. It proved that a narrow symbolic state can influence selection.\n\nAnother experiment used graded consequence vectors. The vector was updated by experience, not by gradient training. It showed useful properties:\n\n\n\n    same direction, more support       -> larger magnitude\n    50% shared experience              -> similarity ~0.41\n    0% shared experience               -> similarity ~0.05  (margin 0.35)\n    support followed by scar evidence  -> direction flipped\n    random nudges                      -> near-zero similarity\n\n\nAn important detail: 0.41 (not 1.0) on partially shared experience is exactly what real generalization looks like, as opposed to memorization. The similarity emerges from structure; it is not hard-coded. That too was a real signal.\n\nBut it was not enough. It carried pressure and similarity. It did not carry the executable rule of _when_ a transition should fire.\n\nThe most honest label for these results:\n\n\n\n    SIGNAL, not solution.\n\n\nThat distinction saved months of self-deception.\n\n##  5. The \"Abstain\" Lesson\n\nOne of the most useful discoveries was negative:\n\n> A weak knowledge carrier should abstain instead of guessing.\n\nIn several gates, the system got better when unsupported cases were answered with \"I do not know\" instead of being forced into the nearest known pattern.\n\nThis matters because many memory systems look good only because they always answer. But in a causal system, a wrong transfer is worse than silence.\n\nIf a stored pattern says:\n\n\n\n    this action repaired the previous case\n\n\nand the new case looks similar but has a different boundary condition, reusing the action can be actively harmful.\n\nThe real requirement became:\n\n\n\n    transfer when supported;\n    abstain when the binding is not justified;\n    never convert weak similarity into certainty.\n\n\nThis is where many memory architectures quietly fail. They retrieve something. The model uses it. The answer looks grounded. But the binding between old evidence and the new situation may be invalid.\n\n##  6. Executable Worlds Made the Failure Sharper\n\nSynthetic gates are useful, but dangerous. If you design the world, you can accidentally design the success.\n\nSo part of the testing moved into small executable worlds: code snippets, controlled bugs, compiler feedback, repair attempts, and real pass/fail consequences.\n\nThe results became sharper.\n\nA local numeric footprint could transfer part of a lived consequence pattern. It worked when the change was close to a linear repair:\n\n\n\n    increase this value;\n    change this numeric boundary;\n    map this local residual to that local patch.\n\n\nBut it failed on cases that required real mechanism:\n\n\n\n    bit-shift semantics;\n    branch behavior;\n    boundary conditions;\n    state transitions.\n\n\nThat mattered. Passing a generated linear-repair gate does not mean the system learned code. It means the carrier can reuse a local numeric direction.\n\nThe criterion became stricter:\n\n\n\n    a carrier must survive mixed nonlinear executable snippets:\n\n    - numeric delta;\n    - bit shift;\n    - boundary condition;\n    - state transition;\n    - negative abstain.\n\n\nIf it only passes local-linear repair, it is a helper operator, not a candidate core.\n\n##  7. The Single-File Brain Attempt\n\nOne direction looked especially attractive:\n\n> What if the entire learned state lived in one mutable file?\n\nThe idea was simple:\n\n  1. The system acts.\n  2. The world returns consequences.\n  3. The state file mutates.\n  4. After restart, the file still changes future behavior.\n\n\n\nThis was not supposed to be a database. It was supposed to be a growing behavioral substrate.\n\nThe result was mixed:\n\n\n\n    transfer cycle: 3/5\n    blind baseline: 1/5\n    wrong: 1\n\n\nSo the file was not dead storage. It grew. It mutated. It transferred some lived transition experience. It beat the blind baseline.\n\nBut `wrong=1` matters more than the improvement.\n\nThe system did not merely abstain on unsupported transfer. It sometimes confidently selected a wrong action. For a knowledge carrier, that is a critical failure.\n\nThe diagnosis:\n\n\n\n    bound transition-cells increased capacity,\n    but did not preserve enough mechanism.\n\n\nThis is a recurring pattern:\n\n\n\n    more structure != more knowledge\n\n\nA richer container can still carry the wrong abstraction.\n\n##  8. The Wall Became a Binding Problem\n\nAfter many carrier failures, the question changed.\n\nAt first I asked:\n\n\n\n    Which carrier stores knowledge best?\n\n\nLater, the better question was:\n\n\n\n    How does a new situation bind to old experience?\n\n\nThe binding is the hard part.\n\nSuppose the system has learned a useful transition in one world:\n\n\n\n    node A influences node B\n    changing A repairs failure F\n\n\nNow it sees a new world with different node names and a different surface form.\n\nWhich node corresponds to A?\n\nWhich node corresponds to B?\n\nWhich local structure is the same mechanism, and which is only superficially similar?\n\nWithout an anchor, this turns into graph matching. In one of the gates I attempted an autonomous bijection between train and heldout worlds directly — and failed exactly here: computing the correspondence required already knowing the correspondence. That is graph isomorphism, NP-hard without an anchor. Not an engineering obstacle — a fundamental one.\n\nThis produced the anchor trilemma.\n\n##  9. The Anchor Trilemma\n\nEvery attempted solution fell into one of three buckets.\n\n###  Option 1: The anchor is given\n\nIf a human or a hand-written rule tells the system which parts correspond, transfer becomes much easier.\n\nBut then the hard part was not learned. It was supplied.\n\nThis can still be useful engineering. It is not an alternative knowledge substrate.\n\n###  Option 2: The anchor is searched\n\nIf the system searches over possible bindings, it can sometimes find the match.\n\nBut the search explodes quickly. The more nodes, relations, states, and conditions, the less attractive this becomes.\n\nYou have not built cheap knowledge. You have moved the cost into combinatorial search.\n\n###  Option 3: The anchor is learned by a model\n\nIf an LLM, an embedding model, an analyzer, or a world-probing system supplies the binding, the system works much better.\n\nBut then the symbolic substrate is no longer the source of the core intelligence. It becomes a memory, a cache, a verifier, or an optimizer around another intelligence source.\n\nThat may be a good product.\n\nIt is no longer the original hypothesis.\n\nThe trilemma:\n\n\n\n    given anchor    -> not learned\n    searched anchor -> too expensive or unstable\n    learned anchor  -> not independent from the model/teacher\n\n\nThis was the wall.\n\n##  10. What Actually Survived\n\nThe failed broad claim was:\n\n> A symbolic carrier can become alternative weights by storing enough structured consequences.\n\nThat did not survive.\n\nWhat survived was narrower.\n\n###  1. Consequence loops matter\n\nThe system improves only when it receives world feedback:\n\n\n\n    try -> observe -> mutate -> retry\n\n\nStatic documents, summaries, and graphs are weak unless tied to consequences.\n\n###  2. Deterministic guards matter\n\nSome fields must not be entrusted to a generative or fuzzy substrate:\n\n  * IDs;\n  * dates;\n  * amounts;\n  * names;\n  * paths;\n  * exact constraints;\n  * status changes.\n\n\n\nExtract them and preserve them explicitly.\n\n###  3. World judges matter\n\nThe compiler, tests, and executable checks were more honest than internal scores.\n\nThey do not care how elegant the architecture is. They return:\n\n\n\n    pass / fail\n\n\n###  4. Abstention is a feature\n\nA weak system that abstains is more useful than a weak system that always transfers.\n\n###  5. Some organs are still valuable\n\nRoute-state, scars, typed relations, date guards, append-only memory, cheap probes, and executable judges are not useless.\n\nThey are useful organs.\n\nThey are not a full brain.\n\n##  11. What I Am Not Claiming\n\nI am not claiming I built an alternative to LLMs.\n\nI am not claiming symbolic systems cannot work.\n\nI am not claiming all memory systems are useless.\n\nI am not claiming the wall is mathematically insurmountable in every form.\n\nThe honest claim is narrower:\n\n> In my experiments, every carrier that tried to preserve knowledge as stored structure eventually failed at transferable causal binding.\n\nAnd there is an even narrower boundary I have to state myself, before someone states it for me: this is one person working for 8 months on a CPU, not the output of a lab. There may be a carrier I never tried. But 150+ gates kept failing at the same point so consistently that the pattern outweighs any single failure — and that point coincides with why symbolic AI historically lost to learning: the correspondence metric has to be learned, not postulated.\n\nThat is still useful.\n\nIt prevents wasting another cycle on prettier containers.\n\n##  12. The Test for Any New Approach\n\nAfter these failures, a new approach is worth testing only if it clears a stricter bar.\n\nIt must:\n\n  * change future behavior without code changes;\n  * survive restart;\n  * beat shuffled and null controls;\n  * transfer to unseen cases;\n  * abstain on unsupported cases;\n  * preserve exact fields separately;\n  * operate in a world with consequences;\n  * handle nonlinear transitions, not only local numeric repair;\n  * show value over a simple baseline.\n\n\n\nAnd most importantly:\n\n\n\n    it must carry the transition,\n    not just an aspect of the transition.\n\n\nThis is the line that killed most of my designs.\n\n##  13. Why This Still Matters\n\nA negative result is not the same as no result.\n\nBefore these experiments, the problem looked like:\n\n\n\n    find a better memory structure\n\n\nAfter the failures, it looks like:\n\n\n\n    store and transfer executable causal transitions with valid binding\n\n\nThat is a much better problem statement.\n\nIt also changes product thinking. A useful system does not need to pretend it replaces an LLM. It can be valuable if it provides:\n\n  * cheaper long-session memory;\n  * evidence-preserving state;\n  * deterministic guards;\n  * world-verified actions;\n  * refusal to overwrite known scars;\n  * test generation from consequences;\n  * bounded automation around external judges.\n\n\n\nThose are real mechanisms.\n\nThey just should not be sold as \"alternative weights\" until they pass the binding wall.\n\n##  14. Conclusion\n\nThe original dream was:\n\n\n\n    build a CPU-only mutable substrate that behaves like alternative weights\n\n\nThe experiments produced something less glamorous and more useful:\n\n\n\n    storage is easy;\n    behavioral mutation is possible;\n    transfer is fragile;\n    causal binding is the wall.\n\n\nThe wall is not that the system cannot remember.\n\nThe wall is that remembering is not enough.\n\nTo act intelligently in a new situation, the system must know what old experience corresponds to, which transition applies, and where the analogy breaks.\n\nThat is the part I could not solve with another graph, vector, summary, route state, or mutable file.\n\nThe final number:\n\n\n\n    ~200 experiments, 30+ candidate carriers, 1 wall.\n",
  "title": "Can You Build an Alternative to LLMs? 8 Months, ~200 Failed Experiments, One Wall. 2"
}