Oris Notes

Notes

Vyuhu 1.0

The Vyuhu architecture has completed its initial architecture-validation stage. The current design is stable enough to move from architecture experiments to a full training run.

Development now focuses on Vyuhu 1.0, a larger generation of the architecture built from the lessons learned with OrisTeam/Vyuhu-280M-Base-1704m ↗ .

Architecture

Compute paths

The new generation keeps deterministic compute selection while increasing global attention depth and adding lightweight n-gram memory and multi-token prediction during training.

Frozen eval progression

A fixed 10M-token evaluation sample is reused across checkpoints. The important part is not only that all paths improve, but that after the early catch-up phase they continue to fall in a remarkably similar way.

Protocol: frozen 10M-token sample · identical tokenized data at every checkpoint · selected checkpoint evaluation.

Training exposure

The current qualitative comparison is intentionally early for Vyuhu 1.0. The 1.0 snapshot has seen about 1.041B tokens, while the old 280M checkpoint used for the generation examples had seen about 1.704B. The archived 280M run later continued to about 3.012B tokens.

That is why the generation comparison below should not be read as “1.0 already wins”. The useful observation is narrower: at an earlier training stage, the new run already shows a different failure profile — especially in how well even LOW preserves Polish syntax, document form and local continuity.

Dataset changes

The dataset change is real, but it is not a completely different pipeline. Vyuhu 1.0 still comes from the same Oris-style Polish filtering idea used in the older run: separate very clean text from usable text, keep the strongest Polish sources, and mix them deliberately.

Wikipedia Extended and Dense Knowledge Binary are not meant to dominate the mix. They are small Pure-Polish additions selected for text with a high density of coherent information: passages that stay mostly on one subject, contain multiple related facts, and express that knowledge in reasonably clean natural language.

Engram and MTP

Two additions in Vyuhu 1.0 sound more complicated than they are. The short version: Engram gives the model a cheap learned memory for recurring local token patterns; MTP adds one extra training target.

Engram: small hashed memory beside the main model

Engram does not replace attention and it is not an external retrieval system. The normal token stream still goes through the same anchors and elastic blocks. Engram simply builds an additional vector from the recent local token history and lets selected anchor positions decide how much of that vector is useful.

For each position, the trainer canonicalizes token IDs and forms the recent 2-token and 3-token histories. Each history is sent through several independent hashes. Those hashes index small learned tables; their embeddings are combined into one memory representation. A low-rank gated injector then compares that memory with the current hidden state and adds only the amount the model learns to use.

tokens
  ↓
canonical IDs
  ↓
recent 2-gram + 3-gram
  ↓
4 hashes for each order
  ↓
small learned embedding tables
  ↓
combined Engram vector
  ↓
gated low-rank injection
  ↓
selected Vyuhu anchors

The useful intuition is that common Polish local patterns do not always need to be reconstructed from scratch by the expensive path. Engram can learn a compact side representation for recurring morphology, short expressions and local lexical combinations. Because the injection starts effectively neutral, the main network can ignore it until training finds a useful signal.

MTP: one auxiliary future-token target

The ordinary next-token loss remains the main objective. During training, an auxiliary low-rank predictor also uses h_t to predict x_t+2; its loss is added with a smaller weight (0.25 in the current run). It is a training signal, not an extra generation step required at inference.

Generation comparison

The qualitative check used 8 prompts × 3 seeds × 6 compute paths = 144 sampled generations. All runs used the same sampling setup: temperature 0.8, top-k 40, top-p 0.95 and repetition penalty 1.15. These are examples from that batch, not benchmark scores.

Old Vyuhu paths: Vasudeva, Sankarshana and Aniruddha. Vyuhu 1.0 paths: HIGH, MEDIUM and LOW. The old text examples come from the 1.704B checkpoint; the new examples come from Vyuhu 1.0 at 1.041B tokens.

Historia Polski od początku XX wieku do końca XIX wieku, a Polska od początku XXI wieku.
Należałoby zatem wyróżnić się z tłumu. Każdy zainteresowany może zostać zaproszony na konferencję historyczną [...]

Historia Polski 1917–1918, cz. 2
1.2. Historia Polski 1918–1920, cz. 2
2.3. Historia Polski 1919–1920, cz. 3
3.4. Historia Polski 1918–1921, cz. 4 [...]

Polska jest, że jego "oznaczenie" i "animację moralna".
- A jednak to nie tylko jest to, jak by było. A może o tym powiedzieć nam wszyscy [...]

Rząd poinformował, że na terenie gminy Czerniaków w dniach od 6 do 14 czerwca nie będzie możliwości parkowania.
- W związku z tym, że w najbliższym czasie planujemy kolejne inwestycje. Będziemy starali się o to, by nie kolidowały z budową oświetlenia [...]

Historia Polski
Po zakończeniu I wojny światowej, Polska odzyskała niepodległość. W 1954 roku na mocy traktatu wersalskiego zawartego w Warszawie [...]

Historia Polski 1935–1945. Do 1945 roku w Polsce niepodległość, Polska utraciła na rzecz Polski przez okupację sowiecką, a następnie wcielona do III Rzeszy.
Na mocy traktatu ryskiego z 1792 r. [...]

Według najnowszych danych, w piątek minister zdrowia w rządzie Francji Jean-Claude Juncker poinformował, że od początku pandemii COVID-19 wzrosła liczba przypadków koronawirusa [...]

Według najnowszych danych, z raportu "Branża przemysłowa i elektroenergetyka" wynika, że w Polsce na początku maja tego roku z usług telekomunikacyjnych skorzystało 1,5 mln gospodarstw domowych [...]

What the current snapshot suggests

• The strongest visible difference is structural. Vyuhu 1.0 more often preserves Polish syntax, local coherence and the expected form of the document, including on LOW.
• This happens at an earlier training stage. The shown 1.0 snapshot is at ~1.041B tokens; the old comparison checkpoint is at 1.704B and the archived old run later reached ~3.012B.
• Knowledge use is beginning to appear, but it is not reliable yet. The new model reaches for dates, institutions, named entities, reports and historical framing more readily, while still mixing or inventing facts.
• That is not evidence for one specific component. Architecture, path layout, Engram, MTP and the data recipe changed together.
• LOW is already qualitatively different from the old minimum path. It often looks like a lower-compute view of the same shared model rather than a path that has lost basic language competence.
• HIGH and MEDIUM are still very close on frozen eval. Around step 10.5k the gap is only ~0.011 loss. Whether HIGH later benefits more from its extra capacity remains an open training question.

Status

Full base-model training in progress. Current qualitative generation comparison uses Vyuhu 1.0 around 1.041B training tokens against Vyuhu 280M at about 1.704B. The archived 280M architecture-validation run later continued to about 3.012B tokens.