Stereo video: fish in 3D¶
Status: in the app since 2026-10-30, steps 1 to 4 below (session, synchronisation, calibration, stills to annotate, 3D measurements of the marks). Tracking in 3D, the anemone's reference frame and the space-use volumes come next. The first part of this page records what was measured on the test sessions; Using it says how to run it.
The goal: two cameras film the same scene (here, anemonefish around their anemone); each fish is found in both views, its position is computed in 3D in millimetres, and from there its speed, its length, its distance to the anemone and the volume of water it uses.
The first data set¶
LBMC_data/cmaslard/annotrain/datasets/nemo-3d-selfie/ (NAS, 367 GB; /import/nemo-3d-selfie for AnnoTrain):
- 14 colonies (sessions) of Amphiprion akindynos, one GoPro pair each:
left/<colony>_L<n>.MP4andright/<colony>_R<n>.MP4. 4K (3840 x 2160), 59.94 frames per second, HEVC, with sound and GoPro telemetry. A recording is cut into chapters of 534.5 s (4 GB); a session lasts 25 to 30 minutes. metadata_video_AEST_GMT+10.xlsx: start time of each colony (AEST) and, per fish, sex, rank and standard length measured by hand (sl_mm).- Oddities:
FP2AKP1_R4is missing; the right files of WAK2 are namedWAK2_L*; HSAK1 is one H.264.movper side, without telemetry; only EBAK2 has separate calibration videos (caliL_EBAK2,caliR_EBAK2). - The camera clocks are wrong (2021, 2025, 2026 dates): times in the files cannot synchronise anything. The two cameras swap sides between sessions (their firmware versions swap too).
- Every session starts with the same routine: the calibration frame is filmed on the sand, held in several positions for 1 to 3 minutes, then the rig is carried to the anemone (2 to 4 minutes in) and left there.
1. Synchronisation by sound¶
Both cameras record the same sounds (bubbles, knocks, voices). stereo_sync.py joins each side's
chapters, finds the offset over the whole session (GCC-PHAT cross-correlation), then measures it
again on a 20 s window every 30 s and fits a straight line: offset at the start, plus a drift.
| Session | Offset (right vs left) | Drift | Windows on the line | Scatter around the line |
|---|---|---|---|---|
| FP3AK | -1.814 s (-108.7 frames) | -49 ppm (-88 ms over the session) | 59 / 59 | 0.29 ms |
| LAK1 | +3.654 s (+219.0 frames) | +49 ppm (+86 ms) | 58 / 58 | 0.27 ms |
| CCAK1 | -1.377 s (-82.6 frames) | -49 ppm (-78 ms) | 52 / 52 | 0.43 ms |
A frame lasts 16.7 ms, so the synchronisation is exact to well under a frame. The drift is real (5 frames by the end of a session) and must be corrected; it is a straight line, the same two cameras running at slightly different rates (its sign follows the side swap).
2. Calibration with the frame¶
The calibration frame (VidSync style) has two faces 454 mm apart: a transparent front panel (4 mm polycarbonate) with targets every 87 mm, and a white back panel with targets every 174 mm across and 87 mm down. The printed labels are in units of 8.7 mm in one x, y system for both faces: front (10, 30) to (40, 30) is 3 steps of 87 mm, back (20, 30) to (60, 30) is 2 steps of 174 mm.
stereo_targets.pyfinds the targets: the centre of a black and white quartered disc is a saddle of the brightness; a candidate is kept when two rings around it read dark, light, dark, light with their edges at the same angles. About 60 of the 70 targets are found per view, plus a few false ones (rods, sand).stereo_calibrate.pyneeds 4 labelled targets per face and per view (as in VidSync, where the user clicks them); every other target is found from them. The frame is then a known 3D object: OpenCV fits each camera (focal length, centre, radial distortion), then the position of one camera relative to the other.
On FP3AK, 3 synchronised positions of the frame (at 60, 100 and 120 s), each in turn left out of the fit and measured:
| Position left out | Distance | Pairs of neighbouring targets | Error on their spacing |
|---|---|---|---|
| 60 s | 1.0 m | 36 | 0.35 % (0.3 +- 0.4 mm) |
| 100 s | 1.2 m | 47 | 0.25 % (0.0 +- 0.4 mm) |
| 120 s | 1.4 m | 80 | 0.60 % (-0.3 +- 1.0 mm) |
Baseline 338 mm, cameras almost parallel (1.5 degrees). For comparison, Lilkendey et al. (2024) report 2.5 to 3.4 % with a checkerboard.
Two traps found on the way:
- Handedness. With the labels' y going up, the frame's coordinates are left-handed; OpenCV then fits a mirror image of the scene (distances right, positions mirrored, depths negative). y must go down (x right, y down, z away from the cameras).
- Distortion away from the frame. The frame covers the middle of the image only; a free distortion polynomial (k3, tangential terms) went wild at the edges. Only k1, k2 are fitted for now. The separate distortion calibration the team made for VidSync, or more frame positions near the corners, would fix this properly.
3. Does the calibration still hold at the anemone?¶
The rig is carried after the calibration, and its supports can move during the recording.
stereo_check.py matches points of the scene (coral, sand) between the two synchronised stills:
if the cameras have not moved relative to each other, each point lies on its calibrated epipolar
line. No frame is needed; any moment can be checked.
| Time in FP3AK | 600 s | 900 s | 1200 s | 1500 s | 1750 s |
|---|---|---|---|---|---|
| Distance to the epipolar line (median) | 0.7 px | 0.5 px | 1.9 px | 2.7 px | 3.5 px |
The calibration survived the transport, then the cameras slowly moved relative to each other during the second half. Distances along the line (depth) can shift as well without showing here; at 1.75 m, 3.5 px is about 0.7 % of depth. Two answers: flag the periods where this exceeds a threshold, and re-fit the cameras' relative rotation from these scene points (the baseline length kept from the frame).
4. First measurements: standard length¶
stereo_measure.py triangulates points clicked in both views. Snout and caudal peduncle of the two
largest fish of FP3AK, on one pair of stills (900 s):
| Fish | Distance | Length in 3D | Length measured by hand | 1 px of clicking error |
|---|---|---|---|---|
| larger | 1.27 m | 62.7 mm | 62.11 mm (FP3AK_1, female) | +- 1.0 mm |
| the one below | 1.55 m | 59.2 mm | 58.65 mm (FP3AK_2) | +- 3.4 mm |
The second one looks smaller in the image only because it is 30 cm further: in 3D it is correctly the second largest.
5. Which way is up¶
The GoPro telemetry stream (gpmd) holds a gravity vector per frame (GRAV), plus the
accelerometer and gyroscope (about 200 Hz). With the cameras' rotation from the calibration, it
gives the vertical axis of the scene without any click, and the gyroscope shows every knock on
the rig (a moment where the calibration may have changed).
Using it¶
A stereo session is one generic thing, for any two-camera rig: each camera's chapters, read
where they are on the NAS (under the import folder). The heavy work runs in a thread of the ml
service beside its GPU queue (ml/scripts/stereo_session.py), one step at a time; the admin
panel Stereo videos (3D) of the project shows each step and is where the targets are clicked.
- Add the sessions (on the server): files named
<session>_L<n>in the left camera's folder and<session>_R<n>in the right one's (a file in the right folder counts for the right camera whatever its letter).
docker compose exec backend python -m app.cli import-stereo <project> /import/<...>/left /import/<...>/right --only FP3AK LAK1
- Synchronise and find the frame (prepare, about 5 minutes per session): the chapters are checked (one recording, no frame missing), both sounds are read and synchronised (offset, drift, scatter), and the left video's first 8 minutes are scanned on its key frames for the moments the frame stands still. Only well-contrasted targets count there: coral and sand give many weak saddles. Each position gets a still of both cameras at the same instant, with every target found.
- Click the labelled targets on at least 2 positions (3 to check the calibration): pick a printed label, click its target in each view; 4 per face are enough, the other targets are found from them. Then calibrate: each camera, their relative position, the positions left out measured with the others, and every 2 minutes over the session whether scene points still fall on their epipolar lines (the cameras moving relative to each other).
- Take pairs to annotate: synchronised stills of both cameras at chosen times (from, to, every), added to the project's images; with the project's group run above 0, the left and right images of an instant come one after the other.
- Measure: an object marked in both images of a pair (a box, with points inside it, e.g.
snout and caudal peduncle) is matched across views and measured: distance from the left
camera and the length between each two of its points (
GET .../stereo/<id>/measurements). -
Follow the fish in the videos (Follow the fish, both cameras): the project's active detector goes through each camera's whole video, its chapters one after the other, at the rate chosen (one frame in a whole number of them: 15 per second is one in 4 at 59.94). The frames followed are the 4K ones, decoded straight from the chapters, while a 1920 px copy is made to be shown; the tracks are in the 4K pixels, like the calibration. Each camera becomes a project video (
ProjectVideo.parts,stereo_session_id,stereo_side) and each run two tracked videos in a collection named after the session, shown with the project's predictions. About 0.3 s of the RTX 3090 Ti per frame: 2 h 15 per camera for 30 min at 15 per second. -
Name the tracks (Name the tracks, in the project's videos panel): the classifier of the sub-project chosen in the settings ("Above the suggested rectangles") scores up to 5 cut-outs of each track, framed like its training cut-outs, and the track takes their average (
ml/scripts/name_tracks.py). Each video keeps itstrack_classes.json, replaced by the next run; in the player, a box shows the classes and a filter shows one class only. Any project with tracked videos and such a sub-project can use it. About 20 minutes per 30 min view of 4K at 15 frames per second on the server's GPU, most of it decoding the video.
What comes next¶
- Reference: the anemone's centre clicked in both views (origin), the vertical from the GoPro's gravity vector.
- Fish over time: the detector trained on the annotated pairs as today; tracks in each view (done, see Follow the fish above), each named by the species classifier of the Clownfish sub-project, joined across views by the epipolar geometry, then in 3D; each track assigned to an individual of the colony (its 3D length helps); a timeline to correct swaps, merges and splits. Re-fitting the cameras' rotation from scene points where the check shows they moved.
- 3D tracks: per fish and frame, position (mm), quality (reprojection error, depth uncertainty), seen in both / one / no view; smoothed (Kalman) without bridging long gaps; every gap listed (out of view, hidden).
- Export: per frame and per fish (time in AEST, x, y, z, distance to the anemone, speed, quality); per fish (time observed, mean and max distance, 3D standard length against the hand measurement).
Space use (50 % and 95 % volumes)¶
Positions at 60 per second are far from independent. The volumes will come from a
movement-based kernel in 3D (Brownian bridge, as in the R package mkde): it follows the time
between positions, does not bridge gaps longer than a set time (exits, hidden periods), and takes
each position's 3D uncertainty as its spread. With them: the effective number of independent
positions (from the autocorrelation time), volume against observation time (does it level off?),
and the part of the 95 % volume outside what both cameras see (a truncated volume must be
flagged). Statistics (volumes against body length, reef structure and their interaction, fish
within colony) come afterwards, in R.