How a video-tracking AI follows the membranes inside a cell

In a video, neighbouring frames are separated by time. In ETSAM, they are slices at different depths inside a specimen. A model designed to keep track of objects as a film unfolds is being adapted to follow the membranes that partition a cell.[2][3]

The appeal is easy to picture. Imagine cutting through a folded sheet: its outline changes from one cut to the next, even though every fragment belongs to the same surface. Treating each slice as an unrelated picture throws away a useful clue. What happened to that boundary in the neighbouring slice?

Joel Selvaraj and Jianlin Cheng, at the University of Missouri, developed ETSAM for cryo-electron tomography, which images cellular structures in three dimensions. Their peer-reviewed paper appeared as an early release in Nature Communications on 11 October. The task is membrane segmentation: assigning parts of a measured volume to membrane or background so researchers can examine cellular organisation.[1]

A memory for the next slice

Cryo-electron tomograms are difficult material for image software. The signal is weak, and the limited angles from which a specimen can be imaged leave reconstruction artefacts. A faint boundary can be biologically meaningful without looking like a crisp edge in a photograph.[1]

The underlying model, Meta’s Segment Anything Model 2, or SAM2, carries information forward as it processes a video. Its prediction for the current frame can draw on stored representations of earlier frames and objects. A prompt such as a click or a mask tells the system what to follow; the memory helps it maintain that thread.[3]

ETSAM fine-tunes this architecture for tomogram slices. Its first pass starts with a grid of points on the first slice and propagates membrane predictions through the volume. The second pass combines the initial output with the normalised tomogram and refines the membrane mask. The original measurements therefore remain an input to the refinement.[2]

The interesting transfer is from continuity in time to continuity in space. The software is not watching a living membrane move. It is using the relationship between adjacent sections of a reconstructed specimen to make each section easier to interpret.[2]

Turning a prediction into something a biologist can inspect

Training combined 83 experimental tomograms from the CryoET Data Portal with 28 simulated tomograms made using PolNet. The current evaluation documentation specifies 17 experimental test tomograms with reference membrane annotations.[1][4]

Evaluation compares predicted masks with the reference outlines using overlap, precision and recall. Precision asks how much of the predicted membrane is correct; recall asks how much of the annotated membrane was recovered. For a biologist tracing a surface, both matter: extra connections and missing sections can tell very different stories about the same cell.[4]

The software’s ChimeraX integration makes that judgement visible. After segmentation, the predicted membranes appear as a surface over the tomogram. Researchers can move a threshold slider to include fainter predictions or demand stronger evidence. They can save either the resulting binary mask or the underlying scores, preserving information that a simple yes-or-no outline would discard.[5]

Closely apposed membranes get a specific option: processing the four image quadrants separately with a denser prompt grid. The documentation says this takes about four times longer. Optional clean-up removes small noise fragments and short bridges, but the authors advise comparing it with the raw output because small real membranes can also be removed.[5]

That is a useful design choice. A smooth-looking surface is not automatically the most faithful one. Letting researchers inspect the prediction against the measurements, adjust a threshold and retain the original scores keeps the biological question in view: which boundaries are actually supported by this specimen?

For this application, the surprising contribution of a video model is its ability to carry context across a sequence. The sequence happens to run deeper into a cell.

Report an error or suggest a correction