# Motion System — Buildout & Findings

A working record, in my own words, of the Render UI Motion system as it stands after
this session: what it does, how the pieces fit, and — most importantly — what the
experiments actually found. Written to be read by the next person (or model) who picks
this up.

## What the system is

A single GH200 (94.5 GB) runs **one resident FLUX.1 pipeline** that animates a
**fixed-seed latent** along a structured path on the latent sphere — the *moment
operator*. No video model. The motion is authored, not generated frame-by-frame: the
operator moves one latent a little each frame, and each denoise gives a coherent next
frame, so the loop breathes.

Three processes cooperate:

- **model-manager** (`ui/inference/mm_server.py` → `model_manager.py`, systemd user
  service on :8188): the API + catalog + the spiral endpoints. Restarts are safe now —
  `KillMode=process` on the unit keeps detached children (Gemma) alive across a restart.
- **spiral worker** (`ui/inference/spiral/worker.py`): keeps FLUX resident (~33 GB) and
  drains `~/spiral/queue/*.json`. One worker is the default — two identical FLUX pipelines
  just duplicate the shared T5/VAE/CLIP, so a second worker buys nothing but VRAM.
- **CPU evaluator** (`spiral/evaluator.py`): scores every finished render across the
  named dimensions on the Grace cores, and calibrates the speed control.

The GPU never has to sit idle: `~/spiral/refill.sh` tops the queue up with a cherry-blossom
batch (a dimension sweep + a gain map) whenever it runs low, and the InMotion **feed
workers** button (`POST /api/mm/spiral/kick`) does the same on demand.

## The dimensions, and how they're controlled

The dimensions we care about: **blur, motion/direction, diff (frame-to-frame change),
speed, quality.** Two distinct control surfaces emerged, and the distinction is the
single most important thing to understand:

1. **The operator controls the pace.** `map_globals` maps the **speed** slider →
   *frames-per-second* (`fps = 6 + speed·26·speed_strength`) and the **diff** slider →
   *operator amplitude* (`amp = 0.2 + diff·1.2`). speed = how fast the loop plays;
   diff = how much it moves per frame. They're deliberately orthogonal.
2. **The CPU evaluator keeps speed honest.** It measures the perceived-motion ratio
   (Δ/frame × fps) between high- and low-speed renders and tunes `speed_strength` in
   `calibration.json`. Measured ratio **3.18** against a 3.0 target (within 6%), so
   `speed_strength` held at 1.0 — the slider already scales perceived motion ~3× from
   low to high, per model. Proof it's real: a speed-0.1 clip (9 fps) plays the same
   petal drift ~3.4× slower than a speed-0.95 clip (31 fps), same per-frame Δ.

## The map: which vectors drive which dimension

The **gain map** (`worker.py` `gain_map`) perturbs each FLUX direction (PCA components +
single neurons) at a block, renders a short α-ramp, and fits the *slope of each measured
dimension vs α* — a **per-dimension** gain `dDimension/dα`, not one combined Δ. Result
for cherry blossom (24 vectors across blocks 2/8/16): the dimensions land on **distinct**
vectors —

| dimension | top driver(s) |
|---|---|
| **blur**   | block 16 · neuron#2484, block 8 · neuron#2484 (negative = it blurs) |
| **motion** | block 2 · PC1, block 8 · PC1 |
| **diff**   | block 2 · PC3, block 16 · PC1 |
| **speed**  | block 2 · PC3 (≈ diff — confirming speed and diff are the coupled pair) |

So they *are* separable: blur is a specific neuron (#2484, consistent across two blocks),
motion is PC1 in the early blocks, diff/speed share PC3. The Map page renders this as a
ranked top-drivers list per selected dimension, each row led by its **strength**
(dominant → strong → moderate → minor → faint) so rows differentiate, with `block · PC`
as the muted address and a glossary so the tokens translate.

## The convergence study — the real finding

The Multivec optimizer searches *coefficient mixes* of a block's attention-direction
basis, trying to converge **real motion** (translation, label `motion`) at a small, even
per-frame Δ. We ran it to convergence. What it found, robustly:

- It **converges stably** — but to **`deform`** (small warps), never to clean `motion`,
  at every setting tried. Travel stays ~0.01 regardless of sigma, objective weights, or Δ.
- **Block 2** plateaus at best −6.45. **Block 8** is a genuinely better basis (−3.66) —
  which agrees with the gain map putting motion deeper — but still lands on `deform`.

The honest conclusion: **for this subject, clean controllable motion is not a single
steerable attention vector — it's carried by the moment operator's latent-sphere path.**
The attention directions characterize *deform / diff / blur*; the **operator** (mode,
amplitude, speed) is the motion lever, and we proved that lever works (3× perceived-motion
scaling). The vector search and the gain map agree on *where* change lives; they just show
that isolated translation isn't something you dial in with one vector here.

This is a result, not a failure of the tooling: the Map/Multivec are the right
instruments, and they told us the truth about where motion does and doesn't live.

## Other capabilities added

- **Video → animation**: `POST /api/mm/spiral/ingest` extracts an uploaded clip's frames
  (ffmpeg, CPU), builds a webm, scores it, and registers it as a *reloopable* render —
  treat the frames as frames; the existing reloop iterates on it. No image-conditioned
  model needed.
- **mp4 download**: `/api/mm/spiral/download?name=&fmt=mp4` transcodes webm→mp4 (h264,
  cached) on demand; download buttons on Output, InMotion, and the Map clips.
- **InMotion** shows the live GPU→CPU pipeline: DEV/SCHNELL generations with frame +
  denoise-step progress, the CPU evaluation with per-render dimension scores, GPU + CPU
  gauges.
- **Gemma** (`gemma-3-27b-it-Q4_K_M`) runs on :8080 alongside the worker with ~40 GB
  headroom, and survives manager restarts.
- **Model 3D viewer** (`tuning.tsx`): turbo |W| colormap, distinct Q/K/V/MLP segment hues,
  unlit material + tone-mapping off so the weights read at true brightness, hero canvas.
- **Research** tab: a publication-grade reader over these findings docs.

## Operational notes for next time

- The spiral worker, refill, and convergence loops live in `~/spiral/` (`worker.py`,
  `refill.sh`, `converge.sh`). The refill keeps the GPU warm; `converge.sh` drives the
  Multivec study and is parameterized by block / sigma / target_delta / weights.
- Convergence rounds must be pinned to the **running worker's model** (`fp16`) — a job
  pinned to `schnell` with no schnell worker sits orphaned forever. This was the one bug
  that silently stalled the whole study.
- A render record with `sheet: null` once 500'd the whole spiral listing (`Path(None)`);
  the listing now tolerates null fields. Watch for that pattern in any new record shape.