Param Hegde on The Robot Brains Season 2 Episode 17
Pieter Abbeel
Not too many of you might know this, but I played a lot of basketball as a kid, including university level basketball, back in Belgium, before coming to the United States. Still, today, I interweave a good amount of sports into my workdays, especially running, biking and tennis. And I also follow a lot of professional sports. Sports and AI are both daily activities for me but they rarely intersect. So I'm very excited that they are about to intersect right now. On today's show, we welcome Param Hegde, the {former} Chief Technology Officer of Catapult Sports. Catapult Sports creates sport specific algorithms to help athletes improve performance. Soccer, basketball, football, cricket, hockey, rugby, baseball and NASCAR, Catapult Sports has it covered welcome to the show. I'm so excited to get to talk about AI and sports in the same conversation, here today, with you.
Param Hegde
Thanks, Pieter. Thanks for having me. I’m very excited. I'm also a sports fan like you and I play a lot of sports and I am also very interested in AI so it's a great topic. Excited to talk with you.
Pieter Abbeel
You found the perfect job.
Param Hegde
Yes.
Pieter Abbeel
To dive right in, Catapult says, we exist to unleash the potential of every athlete and team on Earth. And indeed, Catapult works with over three thousand two hundred elite teams globally, including the entire NFL, various NBA teams, Major League Baseball teams, NHL teams, NCAA teams, majority of the English Premier League, the Bundesliga, Super League Rugby and the list goes on. So what does it mean when you say unleashing the potential of every athlete and team on Earth?
Param Hegde
Great question. So as you narrated, our tagline is and our mission is to unleash the potential of athletes and teams. As you noted, you name any league in the world, there are teams who are our customers. So we have a lot of, several products. We are a software company, right? So we use our product offerings to help these athletes to be their best whenever they either they show up at the practice or at the game. So we have several ways we help them through our products. We have a lot of video analysis products, we have wearable technology products, we have analytics, data platforms. Through various ways we help them. And for me, particularly, it's super exciting because you wake up every morning and you show up at work thinking that you know the work you do and your teams do is making a huge difference in some athlete's life, right? I think it's all science driven and technology driven. So now when you aid that with the knowledge that the customers have, they have been doing this for years. But when you combine all these things there is a great outcomes for everyone involved.
Pieter Abbeel
Now you support so many athletes and teams. Can you give a few concrete examples of maybe some of your favorites of the kind of product, certain teams and athletes are using?
Param Hegde
Yeah, absolutely. I'm a cricket fan. I grew up in India, so like everybody there, I follow cricket. I wanted to be a cricketer growing up, right? But soon you realized that was not going to happen. So we have developed so many solutions at this point of time. One of my favorite things, we have developed a solution to help deliver a cricket ball with maximum accuracy, right? Because every fast baller, their dream is to get into the head of the hitter. The equivalent of a hitter in baseball. The batsman. So a lot of them are injury prone all the time. They play a couple of good games and then you get injured. So through our wearable technology, we have solutions which help them, how to maximize the run up because it's a long run up. Then how do you deliver the ball? So and more importantly, since I mentioned about injury, how to get them back into the real games? All right. After somebody is injured. So those are some of my favorite examples and you mentioned you are a basketball fan. So, yeah, we have a solution which accurately actually helps in predicting the jumps because getting a good jump in basketball is very important. So we recently developed a solution using our wearable plus video, right, together. Because when you know, we have a lot of high accurate cameras during the session, in our practice sessions, you can capture and also during the game. So we have a solution. We have developed the solution by looking at both together, accurately helping you how we are doing during the practice session, right? So a lot of examples like this and we recently, our athletes won something like 24 medals in the Tokyo Olympics. So yeah, so we take a lot of pride in supporting them. So whether it's rowing to any individual sport to a team sport. So we really help them get better every day. So far, a lot of times it depends upon coaches' intuition and their understanding of the game, like how to make you feel better. And we continuously see these athletes embracing the technology and coaches embracing the technology. So we believe when you put the existing knowledge of these coaches and physios and all the supporting staff, sports scientists combine that with data and AI. That's where we start to make a huge difference. That that's how we are helping.
Pieter Abbeel
Now when you said basketball jumping, how should I imagine this? Is this a technology where it can teach you to jump higher or what does it do?
Param Hegde
The timing of the jump is everything, right? I think I'm not a great basketball player, but I watch a lot of it. Yeah, it's anywhere between, anything between, you move with the ball and how did you time your jump before taking a shot, right? Whether it's a three pointer or what's your movement look like when you're taking free throws? And we've all seen, my favorite example is I've seen in the playoffs all these great players, how they struggle to take free throws, which can make a huge difference in the game's outcome, right? So when you're coaching that star player who is really great at the rim who jumps high there. But while you are taking a free throw your mechanics so you can easily capture all that to the wearable right now or wearable device. Then you have all these cameras. As I said, when you combine these two things, that's where our secret sauce starts to show up.
Pieter Abbeel
Now, I'm curious if I double click on this as a player, I'm being recorded. I have all these wearables. Do I then have to analyze this on my own as a player and watch myself in these feeds? Or is there actual suggestions that come directly back to me pointing out things to change or specific little snippets to watch? How does that work when I'm a player?
Param Hegde
So how it is done today, as you know, traditionally it is done, is you videotape the sessions, practice sessions or the games. You watch the game after the games, game is done and you go through it with the player, right? And know as a coaching staff, hey, this is what I want you to protect, they point out, right? So our solution, you know, started as a descriptive solution, right? So we have wearable device and we have video. So we have a lot of video analysis software and we help these coaches and coaching staff to work with these players in appointing them with the data. Along with what they saw, right?, Because they were there when that happened. But then they also had this software and the solution which helps them to double check or help them reinforce what they saw. That's how it all started. Then now it's moving into, more into predictive movements and prescriptive. So we are in that journey. So basically, we are taking away this burden from the player itself. But we are helping through software to help with the coaching staff and the players because we produce two sets of outcomes. How the coaching staff need to help the player and also the player wants to consume this after the session. What are they seeing? So we automate all that in our software. That's where a lot of AI and data come into picture. Our algorithms help prepare those recommendations. In some cases are even accurately pinpointing here, this is what happened. Therefore you couldn't do this if you had done this the outcome could have been better, right?. So you can have this conversation with more backed by the data and science, along with the expertise these folks have.
Pieter Abbeel
Now you're saying AI is backing a lot of this. It's more than just recording data. It's going to find the things that are interesting to look at and even make prescriptive recommendations. When you say AI, can you say a bit more about how does this work? What's behind the scenes? A lot of what we've discussed in this podcast is, you know, deep neural networks capturing patterns in data. Is something similar happening here? Can you say a bit more?
Param Hegde
Yeah, absolutely. I think there are multiple things at play. So we have a lot of supervised learning techniques which are in play. In fact, I wanted to point out before the whole data science and the data explosion which happened in the last decade, right? As you are very well aware. We had built a bunch of expert system based AI algorithms into these products. So it all starts for us with our wearable devices where these athletes wear and they generate very high frequency data for us, right? So we have a set of algorithms which runs on the edge. So that's how it started. So we have algorithms on the edge within the devices, right? Which helps you because these devices are very intelligent in nature. So there are accelerometers, gyros and whatnot. So three dimension data. So how do you take that and make sense of those high frequency data? That's where a lot of secret sauce exists, right? So we have some AI algorithms that's where the whole thing starts. Then when you, we also have a lot of video, computer vision based technologies. Then what we have done over the years is and as the data became much, much more and the computing and the storage in the cloud became more affordable, so we have taken advantage of that. So our whole AI solution has evolved in the last decade along with the landscape itself. So now it allows us to do interesting things. You're bringing all these things together into the cloud, taking advantage of the infrastructure there. And to answer your question specifically, that allows us to sit and observe, right? You can start thinking about how you can build bots, who are sitting and observing. These are all software bots. There are certain things we know, what's happening in the sports and there are certain things we don't know. We discover as we go. That's where a lot of neural networks and deep learning come into play. So we are extremely uniquely positioned as a company because if we have the wearable data and video together, that's where you can start building these very interesting algorithms using AI.
Pieter Abbeel
One of the things, for example, when watching soccer that's pretty common is they'll say something like this team has this percentage possession, right? 60 percent possession for this side. And they show that on TV. And so I guess either somebody is tracking that or they have some software that's tracking that, I mean that they use for tracking this. And that's one thing but I imagine that, for example, for soccer, you're able to use computer vision to track a lot more detail than just the possession. Can you say a bit more about what are some examples of things that you're able to extract?
Param Hegde
Yeah, absolutely specifically using computer vision. We have a product called MatchTracker, by the way, which does exactly what you just described. So when the game is going on or a practice session is going on, it's very common to tag the events, right? A goal is scored or somebody missed it. All those things happen. That's part of the tagging and the event generation during the broadcast. And you see all that. And you also keep track of the positions, and these are all basic broadcasting use cases. But on the other hand, as I mentioned, we are in the performance business, right? Our company, we are observing how to help these, this team and these athletes to get better next time and next time after that, right? So it's a long game for us. So we are taking that video as it happens, they are like and also possession. We are looking for things from the performance analysis perspective. Classic example is, hey, there is a kick which happened at the 65th minute. But however it didn't happen the way this person, I expected this player to do. Then we go back and look, we have the wearable data. We have supplementary data sets to add context to why, right? So that helps us to come back in, okay, in the video, plus the wearable data together, and then we can add more context. So then you go back. And so there are several examples like this. So you can start creating during the practice session as well as during the game.
Pieter Abbeel
Could I imagine something where some substitutions are recommended to the coaches to consider because it's tracking real time somebody starts to move slower, react a bit slower than they normally do, things like that?
Param Hegde
Yeah. So every league, as you probably know, has different rules. So I think eventually some of the rules will be relaxed. I'm not an expert in some of these rules across the leagues, but you are absolutely right. So that's the kind of use cases which will start to form. And we're already talking about whether it's in football or in American football, right? It's a lot of contact sports, in general, where you can start preventing doing these intelligent substitutions. Or even prevent an injury, for example, because you do see some things in the sports, either through wearable data which is coming, these players are wearing or through the video, which you observe, right? So when you correlate these two things together and with few other data sets, we will have access to, that is when you can start building this extremely intelligent use case. Yeah.
Pieter Abbeel
Now thinking back to the computer vision side of things, which I mean computer vision has been completely revolutionized in the last five to 10 years compared to what was possible before. I'm kind of curious what kind of progression you have seen in terms of what you're able to get from these video streams, let's say, five, 10 years ago and what we're able to get now. And where do you see this headed? What is the information that could be extracted maybe a few years from now as AI keeps getting better and you click more and more video data from all the players and teams?
Param Hegde
That’s a great question, Pieter. I think in threes, right? One is the cameras themselves. The data generators are going to be fast and more accurate. And that data, than the pipe infrastructure carrying that to the cloud, right? Which as you are tremendously in the last two decades, will continue to evolve whether it is on 5G or 6G, because now you really don't need to be constrained by the local area networks in the stadiums and other places. Then the computing in the cloud is evolving extremely fast pace, right? Whether it is at the chip sets or like the storage in general and the computing in general. So when you combine these three things together and in my head, all the research elements which have gone, I was thinking before coming here before 2008, 2009, if you are doing computer vision with so many limitations, right? And there are not many, too many practical applications. So now you wonder if you are accurately like marrying two sites, the research angle of it, which is happening and also the advancements which are happening in computing and in the industry in general. To me, that's where now you have more tools available, software tools, open source and the whole group of people who are doing it has enormously increased. Therefore, you are the beneficiary of that and sports will be no different in my head. So now you can take a live video stream and start applying machine learning on that because it was unthinkable a few years ago. How do you take all these video streams and apply CV on it because you have multiple camera angles, right? In the stadium, so you have to triangulate and you have to build a system where each camera can be providing you with interesting data points. So how do you? So the compute, the horsepower, which is available in the compute will change all these things, so now, you can start to process it very quickly. Then once you start processing it, you can apply the models run through that real time data through the prebuilt models, then you can start creating pretty much real-time insights. That's where I'm excited about, this whole thing is going to be headed.
Pieter Abbeel
Hmm. So you get these real time insights and it can really change the game in many cases. I can see why there might be rules about it and to make sure there's a level playing field. As you start building this and this becomes more and more automated, right? It seems like one of the big things that could happen is right now you're focused on professional sports. They used to already have scouts taking notes on so many things. And so now they get more consistent coverage. But they were already doing this. But there are so many other sports happening, so many places that are not professional where there is no scouting. And I wonder, can the same thing be applied there? Imagine you just set up your iPhone on a tripod on the side of the field and let it record and see what can you get out of this?
Param Hegde
Absolutely. I think we actually have a business unit which focuses exactly on what you are describing. So in my head, as you correctly noted, there are a lot of technologies and tools available in the pro sports because that's about gaining that extra advantage, right? An extra one second. Advantages in motorsporting, for example, I don't know how much you follow motorsporting. So yesterday there was an Abu Dhabi, Formula 1 race. Yesterday, I was watching last night.
Pieter Abbeel
Yes. Biggest race of many years.
Param Hegde
So you get passed in the last second right to go into the second place so that those optimizations will continue in pro sport and AI will continue to help those use cases. But real interesting things and real innovation will happen, already happening in the consumer space. As well as even before that, this tier three, tier four sports teams, we call it. That's because in our lingo, tier one and tier two are, you can be thinking of them as pro sports teams and college teams, right? Then you get into high school teams. The example you are providing, I want to record a high school team where my kid is playing and can the same technology, which is available in the pro can be made available through, you know, as I said, like iPhones? Versus your general smartphones, that camera technology is getting better in every release, right, and getting cheaper. And so and the processing you can do, you want on these devices with the computing power available on the edges, unbelievable compared to like eight, 10 years ago. So, now, you start to kind of bring these technologies, which are at the pro level and then the tier two at the college level into high school or even, you said, look, you are an aspiring or continue to aspire to be a better basketball player. I am the same way, you know, playing in the leagues, right? Of my age group. So I'm still competitive there, right? So I want the same technologies there. I want to win. So I keep thinking about how does this technology help not only for my use case and the kids in high school. Or even some sports leagues, which don't allow these kind of resources, right, which these leagues have. And that's where the real innovation in the next few years. This advancement in computer vision or wearables and you name it, that's going to make a huge difference. As I said, going to a descriptor to predictor, prescription because I dream of a day where somebody coaches me on my marathon training like so. So there are so many of these aspiring athletes who can now take a device and wear it before starting it or have a camera or something captures their training and comes back and tells them how to get better. I think that that's a great next set of things how to bring this technology into consumers.
Pieter Abbeel
Well I look forward to that continuing to happen. Now, one thing with today's machine learning, of course, is that assuming it's built on supervised learning, then it's going to do as well as essentially the data that was annotated in the past. And so it seems like it would assume that you would have professional coaches on your staff who can help annotate data that the neural nets can learn from. So I'm curious about that. Do you have professional coaches that do this? Because usually data annotation is seen as something that's an easy thing to do, but it seems in this case, it would be very different. Yeah.
Param Hegde
Yeah. So it's a great point you're making actually. Some of our video products started as, you capture the video, I don't know whether you're aware, there are humans sitting behind and annotating that video actually. Right now, that's what happens. So our first generation of products help to exactly do that. So there is a goal scored, in the case of American football, it's a first or second down, third down, right? So you are explaining, you're watching the video in the stadium and you are annotating using our software, right? So our software actually has that video and it has a capability to annotate that event. So that's a leg up we have, right? Because we have all these annotated videos already. So we always think about how do you automate that? So using computer vision so that there are no human beings that need to take it right? But secondly, like once, since we have so many annotated video sets already, that, to your point, the biggest challenge in CV was to like, having all these people classifying your cats and dogs, right? In any match. So you have to go and find these, I heard one of your guests talking about how cheap undergrad students are, like somebody who can do this right for you. But here we have that advantage of annotated videos. So that's a big advantage for us because now when you are watching the video, it's annotated and then you combine our wearables. So we already have some interesting things going on. But you have to continue to label it better. It's not always going to be accurate, but it's because of the need in the sports leagues after the session is over or games are over, not only these people who are watching it are labeling it. Then comes the sport, you know, then the coaching staff comes and reviews it and relabels. So by the end of 48 hours after the game, you have very good annotated data, datasets available already. But in order to bring that technology to the consumer level. So now you need to automate it because you are the person who is taking the video and you are the person behind watching the game so you don't want to sit and label that. So there are some good opportunities there to automate it using CV.
Pieter Abbeel
Yeah. One thing that I imagine will be interesting in the future, I'm curious about your take, especially for team sports. If you think about kids playing team sports, I mean, there's so much going on at the same time. And if I think about, for example, kids playing soccer, it's very easy to track who scored the goals and you say, okay, this kid scored 10 goals the last few weeks, so clearly they're playing well. But if I think about the other side, people playing on the defensive side, like who's making all the stops on the defensive side for the goalie, it'll be clear, but for the defenders in front of the goalie, like who is even tracking down, who knows who's playing well there? It seems like that information is much harder to get who has the right positioning and so forth to actually help the team win.
Param Hegde
Yeah, absolutely. Because you are making a great point there. We have seen historically, we ourselves, creating a lot of possession based analytics products. We just recently announced a partnership with the Boston Bruins and LA Kings creating hockey analytics. So similar things. So we are similar in that we help soccer goalie analytics. So you focus on that. But the next level of things is how these formations are looking, right and from the defensive angle. So for me, it's an unsolved territory. The problem that you start getting into now, as I said, start recognizing those patterns, right? Because you did this X, Y and Z in the last game, therefore you are allowed this many goals. It's not just the goalie who made, who is responsible for that like, but wow, what happened? Four minutes before that happened, right? So you should be able to go back and put in the context. So then while you're doing that, so you have to sit there and start analyzing and recognizing those patterns. That's why, like, I think your deep neural nets will start to kind of form because, not only you are inputting what happened four minutes ago, then what are the subsequent things which happen to create these formations? Therefore, like, something, you are allowed the goal, right? So those are the moments you should be able to come back and take into account and imagine I use that in game preparations right. And also like, you add another dimension to this. Now, who is your opponent for the next game and a lot of data sets and how they are done, so there is a layer of coaching decisions you can start making. And that's where, as I said, like in the pro, you will start to optimize and then you start bringing that into the consumer level, right? Some version of it. So therefore, if you are a high school coach, coaching, or little league, right, in your hometown. How can you have these conversations with these players, these young kids who can understand it and start putting the same techniques? That, to me, is really exciting at some point now.
Pieter Abbeel
So you've been at Catapult for a little while now. You've already done a lot. But as you look ahead and think about kind of the big picture vision of what might be possible in the next five to 10 years at the intersection of AI and sports. What do you see? What do you see at Catapult? Maybe what are some things you see that Catapult might not be undertaking, but it will also be big in AI and sports?
Param Hegde
Yeah, as I said, sports, you can think of that there are a lot of things happening in sports that we are focused on. Coaching and performance analysis aspects of it because we are in the business of helping teams and coaches and athletes. That's our mission. But you can also think that a lot of AI can be used in sports for betting and whatnot. So we are not super interested in that as a company right now because so I do see the biggest thing which is happening is, as I explained before bringing this, tools in pro into the consumer. How do you bridge that gap as you make the advancement in AI, right? And do it in a way that's affordable, right? Because now we are democratizing the data, pretty much. So now you have to democratize the AI. That to me is the big thing. And as an industry for us specifically, we are excited about contextualizing everything, right? Because as I said, there are wearable devices which tells you certain things from the data. Then you have vision, computer vision. So when you mix these things and eventually with my health data, right? You ought to be very careful about it with privacy. And all those things which we understand as a company, extremely, that's our DNA, actually. All right, we've been doing this, keeping our customers data, in such a secure way. So but when you start bringing those third element of it, then you start to kind of create very interesting solutions at the pro level. Then you've got to bring that to the next level. So now you imagine all of a sudden you are putting tens of millions of athletes because I'm a firm believer that everybody's an athlete, right? Every human being is an athlete. It is a question of some people talk about, some people don't. But how do you help each one of them all around the world to be a better athlete? That's the mission which excites me.
Pieter Abbeel
Well, that's absolutely beautiful, Param. I hope you can make it happen. Maybe it can improve my tennis game in the next few years.
Param Hegde
Yeah, absolutely. I'm up for a game at some point, so I hope somebody builds that robot I'm talking about with the vision and something which basically tells me, like, you know, during the practice session, hey, there is a lot of wind going on like, you may want to consider how hard you are hitting your forehand or something like that.
Pieter Abbeel
Oh, beautiful. Param, thanks so much for coming on. It's a really wonderful conversation. Absolutely great to have you here.
Param Hegde
Thank you very much. Thanks for having me.