Recently, I’ve been interested in the idea of being able to afford the cost of living in California, so I’ve been looking at industry jobs. There are plenty of generic data science jobs that I am qualified for and may very well apply for, but it always seems a shame to not take advantage of the past decade I’ve spent analyzing behavior from video data, so I’m trying to track down a job in that realm, and last week I saw that this climbing startup, Belay AI, is maybe hiring soon. Being able to merge climbing and behavioral quantification would be sort of a dream job for me, so I’ve been reflecting on what computer vision has to offer climbers. I’ve spent a lot of time sorting through joint angle data and other metrics of behavior, and it’s not always straightforward to get things out beyond what is already obvious to human observers. I think it’s cool to see yourself tracked, no doubt, but what’s the actual value added?
Well, I wrote down a bunch of ideas of what you could do with the pose data, but one of the best things I thought of is actually a much simpler tool, which is one that comes up in animal behavior quite often: live action recognition. So, let’s say you’ve decided to become a Serious Climber™, and you want to improve. One of the first suggestions is to record yourself climbing. But this means you have to set up your phone, walk over to it, turn it on, turn it off, or, alternatively, you have to sort through hours of footage. But if your phone just knew when you were climbing, it could do all that for you.
So I spent some time this morning building a little climbing detector. I’m using BehaveAI again, which seems well suited for this, because the motion-based detection should generalize across backgrounds and lighting conditions well, and also run on extremely small yolo models, which can run live even on very small processors, which is important if you wanted to run this locally on your phone. This means you could just put your phone down, and it would store each attempt session as a separate little video. It would also be able to serve as a workout tracker, grabbing total time on the wall, the ratio of falls to sends, etc.
Is this anything? Hard to say. Obviously it would need to be trained on more data and would take more than half an hour to get to work reliably. But as someone who has spent a fair bit working on both climbing and computer vision, it was fun to think a bit about what I would actually pitch as useful. If you want me to actually build this for you, hit me up. I will happily make it happen for the low, low cost of my living expenses.
~ap