Why the transformers trained in hours, not days

Because each run was a small job. The model is small, each training example is small, the training data is made on the graphics card while the model trains, and after the first run no model started from zero.

The runs were also stopped at a fixed number of steps, not when the model was finished. So "hours" is partly a real saving and partly a shorter job. The models are not as good as yours yet.

Monday afternoon, to scale

Each bar is one training run on one H100. Times are Pacific. The numbers are minutes of training.

First model, wrong recordingTrained for the n9c54010 probe setup. Could not read capture 8, so it was done again.
57
First model, capture 8From zero. Simulated bubbles in plain noise. 2.56 million frames.
43
Real background, 7 framesStarts from the model above. Stopped early.
27
Echo shape from real echoesStarts from the model above.
8
Echo strength as in real framesStarts from the model above.
27
15 framesStarts from the model above.
21
Skull-like phase errorsStarts from the model above. 25,000 steps.
37

kept done again

The six kept runs add up to 2 hours 43 minutes of training. After 5:30 pm about sixteen more short runs went through the same card overnight, most of them 6,000 steps each, and each one started from the best model so far.

What made each run short

The model is small

It has 15 million weights: 8 standard transformer layers, 384 numbers wide. Nothing is pretrained. It fits on one H100 with room to spare, and one step takes 0.03 seconds for single frames and about 0.1 seconds for clips.

Each frame is cut into only 510 pieces

The input is one frame of filtered channel data: 5 transmits, 8 by 68 elements, 184 samples. We cut it into blocks of 4 elements by 4 elements by 12 samples and give the model one piece for each block. That gives 510 pieces for a frame. The cost of a transformer grows with the square of that count, so this choice sets most of the speed. The output box is also smaller than yours: 14 mm to each side, not 25.

The training data is made on the card, during training

There is no dataset on disk for the simulated part and nothing to load. For each step the code picks random bubble positions and writes each bubble's echo straight into the channel data: one short pulse for each element, at the round-trip time, with the right phase. No tissue, no wave simulation. This costs almost nothing, so the first run made and learned from about 1,000 new frames each second. No frame is used twice.

The real background sits in the card's memory

From the second run on, the simulated echoes are added on top of real recorded background from capture 8. That background is 12 blocks of 250 frames (the standard evaluation frames are left out). All of it is loaded into the card's memory once, so a step never waits for a disk.

Only the first model started from zero

Each later run starts from the weights of the run before it and changes one thing. A run that only has to learn one new thing needs 4,000 to 25,000 steps, which is 8 to 37 minutes. This was also a need, not only a saving: the 7-frame model started from zero on real background learned nothing in 6,000 steps.

Two smaller things

The model computes with 16-bit numbers where that is safe, which is faster on an H100. And each run was scored by a script as soon as it ended, on the same 250 frames with your tracker, so the next run could start in minutes.

What the speed cost

The runs are not trained to the end. Each one stops at a step count that we chose before the run. The score of the very first model was still going up when it stopped (0.64, then 0.73, then 0.75 over its last 16,000 steps). Some of "hours, not days" is that we did not wait.

The fast first model did not work on real frames. The model trained only on simulated bubbles in plain noise gave 327 bubbles per frame on capture 8, and still gave 286 with the elements in a random order. It had learned the simulator, not the recording. The real background fixed most of this, and that is why the chain of short runs exists.

It is still behind your model. On the standard frames with your tracker:

ModelTracks of 10 frames or moreOf those, moving
First model, simulation only282
End of the afternoon chain25671
Best overnight model292148
Yours (hp_struct), cut to our box356265

Your 356 includes some bubbles counted twice at half a wavelength; a fair count is about 300 to 320. These counts come from the tracker, with no ground truth, on the same 250 frames that we used to choose between runs. So read them as a rough guide.

In the viewer the difference is plain. Our bubbles are bigger than yours and many of them do not move. Your model reads 24 frames and ours reads 15, in a smaller box.

The short version

If the question is what to copy: make the simulated data on the card, keep the real background in memory, and start each new run from the last one. Those three are real savings. The rest of the speed is a small model on a small box that we stopped early, and it shows in the result.

The code is in research/iq2bubbles: train.py and echo.py for the first model, train_t.py and scenes_t.py for the clip runs. All numbers here are from the run logs of 16 and 17 November.