Obviously, we’re some time into the year already, but back in November, I got invited by PloS bio to review this paper on a new tracking method called BehaveAI.
My dear reader. I was not prepared for how much this paper would change my life. When I read the abstract I said to myself, “Well that’s impossible. What they’re promising is impossible.” but they fully delivered. This is maybe the most beautiful piece of tracking tech I have ever seen, it immediately solved a massive problem for us in the lab, and before the paper was even published I had completely rebuilt our tracking pipeline to implement it.
Stepping back, what does BehaveAI do? It’s designed to solve a simple problem in behavioral ecology: Nature is complex and animals often prefer to not be seen. This means that they many animals have a tendency to disappear into the background.
However, just like the fictional T-rex in Jurassic Park, our vision is (partially) based on movement, and spotting an animal moving over a background is trivially easy.
Now you could use Transformers (of ChatGPT fame), which are a very expensive and complex way to process temporal transitions and use context to parse things, including videos, but for many of us lowly fish biologists, we don’t have the luxury of unlimited processing power. What Troscianko et al (the authors of BehaveAI) did is figure out a simple way to encode behavioral context into a single, highly visual salient image that can be processed by extremely small neural networks (in this case, the Yolo nano models). They do this with frame subtraction, which is one of the first things you learn in an intro to computer vision class, but they color the successive frames differences in a way that encodes position, shape, velocity, and acceleration over multiple frames using a single image that is simpler than the original frame.
This makes things easy both for the humans doing the annotations and the neural network performing inference. All of this is wrapped into a very user friendly GUI. Now you can label, train, and run neural networks on your laptop (or on a raspberry pi!). I should mention here I have not been paid to hawk BehaveAI (which is free software anyway). I’m just a big fan.
So for our lab, we often want to track day old babies swimming over gravel. Here’s a picture of what they look like. Can you spot the fish?
How about now?
Up until now, we have been using SLEAP (which is itself a breathtakingly beautiful bit of software engineering), but SLEAP has a couple downsides for this job. First, SLEAP is designed to track pose, so it uses somewhat more complex models than we actually need when all we want is to detect an individual. Second, SLEAP is based on a single image, meaning that when the fish is swimming over gravel and functionally invisible, you’re going to have a hard time. In the past, I had combined SLEAP detections with a MOG Background subtraction using the Viterbi algorithm, which was very complicated and clever and I felt very good about myself, especially because it sort of worked. SLEAP could distinguish fish from algae, Background subtraction could distinguish moving things from background, and the Viterbi algorithm selected the most likely position that combined those two detections.
However, this was computationally pretty slow and barely managed 90% accuracy. In comparison, BehaveAI was practically perfect, and ran at 10x the speed, such that we could retrack all of our data in about a week, in time to submit our paper with the new method. And I haven’t even gotten to the whole point of behaveAI—classifying behavior—which we have used both to spot aggression and recognize tattoo color, but that’s a blog post for another day (or year, given my track record).
In any case, we now have a cool new tool in our tracking toolset, which I think of as follows:
Want to classify behavior in less than 6 hours of video? Just watch the video yourself, it will be faster and more accurate. Really.
Want to track position of a fish with a perfect background and lighting? Anything will do, I like using the MOG Background subtraction built into OpenCV. Tracktor is also nice.
Want to track the position of a fish over backgrounds ranging from “not quite perfect” to “why did I ever allow them to put gravel in that fish tank”? BehaveAI
Want to track the pose of a fish? (and they are at least visible in the frame) SLEAP
Want to distinguish the id of visually similar fish? idTracker
Need to do something else? I dunno, ask me about it, I probably have ideas.
~ap