Why the transformers trained in hours, not days
Because each run was a small job. The model is small, each training example is small, the
training data is made on the graphics card while the model trains, and after the first run no model started
from zero.
The runs were also stopped at a fixed number of steps, not when the model was finished. So
"hours" is partly a real saving and partly a shorter job. The models are not as good as yours yet.
The six kept runs add up to 2 hours 43 minutes of training. After 5:30 pm about sixteen more
short runs went through the same card overnight, most of them 6,000 steps each, and each one started from
the best model so far.
What made each run short
The model is small
It has 15 million weights: 8 standard transformer layers, 384 numbers wide. Nothing is pretrained. It fits
on one H100 with room to spare, and one step takes 0.03 seconds for single frames and about 0.1 seconds
for clips.
Each frame is cut into only 510 pieces
The input is one frame of filtered channel data: 5 transmits, 8 by 68 elements, 184 samples. We cut it
into blocks of 4 elements by 4 elements by 12 samples and give the model one piece for each block. That
gives 510 pieces for a frame. The cost of a transformer grows with the square of that count, so this
choice sets most of the speed. The output box is also smaller than yours: 14 mm to each side, not 25.
The training data is made on the card, during training
There is no dataset on disk for the simulated part and nothing to load. For each step the code picks
random bubble positions and writes each bubble's echo straight into the channel data: one short pulse for
each element, at the round-trip time, with the right phase. No tissue, no wave simulation. This costs
almost nothing, so the first run made and learned from about 1,000 new frames each second. No frame is
used twice.
The real background sits in the card's memory
From the second run on, the simulated echoes are added on top of real recorded background from capture 8.
That background is 12 blocks of 250 frames (the standard evaluation frames are left out). All of it is
loaded into the card's memory once, so a step never waits for a disk.
Only the first model started from zero
Each later run starts from the weights of the run before it and changes one thing. A run that only has
to learn one new thing needs 4,000 to 25,000 steps, which is 8 to 37 minutes. This was also a need, not
only a saving: the 7-frame model started from zero on real background learned nothing in 6,000 steps.
Two smaller things
The model computes with 16-bit numbers where that is safe, which is faster on an H100. And
each run was scored by a script as soon as it ended, on the same 250 frames with your tracker, so the next
run could start in minutes.
What the speed cost
The runs are not trained to the end. Each one stops at a step count that we chose before the run.
The score of the very first model was still going up when it stopped (0.64, then 0.73, then 0.75 over its last
16,000 steps). Some of "hours, not days" is that we did not wait.
The fast first model did not work on real frames. The model trained only on simulated bubbles in
plain noise gave 327 bubbles per frame on capture 8, and still gave 286 with the elements in a random
order. It had learned the simulator, not the recording. The real background fixed most of this, and that
is why the chain of short runs exists.
It is still behind your model. On the standard frames with your tracker:
Your 356 includes some bubbles counted twice at half a wavelength; a fair count is about 300
to 320. These counts come from the tracker, with no ground truth, on the same 250 frames that we used to
choose between runs. So read them as a rough guide.
In the viewer the difference is plain. Our bubbles are bigger than yours and many of them do not move.
Your model reads 24 frames and ours reads 15, in a smaller box.
The short version
If the question is what to copy: make the simulated data on the card, keep the real background in
memory, and start each new run from the last one. Those three are real savings. The rest of the speed is a
small model on a small box that we stopped early, and it shows in the result.
The code is in research/iq2bubbles: train.py and echo.py
for the first model, train_t.py and scenes_t.py for the clip runs. All numbers
here are from the run logs of 16 and 17 November.