The importance of experience in hackathons
photo: Brad Larson, private archive
Recently we’ve had the chance to ask Brad Larson some questions, he is the man behind On-UAV realtime detection of storm damage. This was definitely one of the most exciting projects submitted to the NVIDIA Jetson™ Developer Challenge, after all it won 3rd place. Let’s get into questions.
Brad, what made you decide participate in the NVIDIA Challenge?
We had been developing a product using the Jetson TX1 and TX2, and came across the challenge in a mailing by NVIDIA towards the end of 2017. We thought it would be a great target to work towards, and used it to motivate us to get our project to a proof-of-principle stage. It caused us to focus on what we needed to have operational, and to solve a series of problems in a short amount of time.
Do you have any prior experience in building AI solutions?
Personally, I’ve been fascinated with machine learning since the late 1990’s, and spent a while tinkering with neural networks and evolutionary algorithms before the last AI winter set in. For the last several years, we’ve been working on accelerating machine vision operations using GPUs on mobile and embedded devices. When convolutional networks came roaring back on the scene, I was fascinated by the exceptional results being generated and started moving our work in that direction.
For the last two years, we have been developing software and custom convolutional network designs to perform accelerated object detection and image classification on power-constrained hardware like mobile devices or embedded systems. We’ve relied heavily on NVIDIA hardware and software in this development, so it was a natural step to extend some of our work to the Jetson TX1 and TX2.
With this much past experience we’re certain coming up with an idea for this hackathon was not an issue, correct?
Two years ago, Perceptual Labs was spun out of SonoPlot in order to commercialize machine vision technology that had been created in pursuit of improving the latter’s robotic systems. Perceptual Labs also drew in work that had been done on optimizing GPU-based image and video processing on mobile devices. The company was founded to apply GPU-accelerated machine vision techniques and machine learning to solving visual problems on mobile and embedded hardware.
At roughly the same time, Perceptual Labs and Rapid Imaging realized we had complementary technology and partnered to solve problems in a few areas. Rapid Imaging grew out of work by augmented reality pioneer Mike Abernathy who developed one of the first successful applications of augmented reality in 1993 to track space debris. Since then, Rapid Imaging has developed and provided software solutions that analyze UAV and other camera data and telemetry to georeference exact points within the video.
One such application of Rapid Imaging’s software was in disaster recovery, where they were able to provide live augmented reality overlays to UAV pilots that showed locations of streets and other landmarks overlaid on live video from flooded or storm-damaged areas. However, that could only describe what should be there under the camera, and wasn’t able to analyze what the camera was seeing. There was significant demand from federal and state agencies, as well as from insurance companies, to be able to do onsite damage analysis from the air in an automated fashion, but that would need for us to be able to process camera video and identify what was being seen from the air.
That’s where our work on applying convolutional neural networks to object detection could close the loop. We decided to start by seeing if we could identify and localize storm-damaged buildings in UAV video streams, classify their level of damage, and then georeference their location based on where they were detected in UAV camera video. There are many other places this could be applied, but this was a good starting point for a proof-of-principle.
Rapid Imaging partnered with Embry Riddle Aeronautical University in mid-2017 to gather UAV footage and telemetry from flights over tornado-damaged areas in the central U.S. This recorded video and telemetry was used as training and validation data for a custom object detection convolutional neural network. Once we had demonstrated that the network was able to reliably detect storm-damaged buildings, we combined it with Rapid Imaging’s georeferencing software and worked to get the compound software working efficiently on the Jetson TX1/TX2.
We chose the Jetson devices as targets because they are rapidly becoming part of standard computing modules and control hardware for UAVs in a variety of applications. Being able to do all this processing on a TX1 or TX2 would allow for live, in-the-field analysis of storm damaged areas without the need for network connectivity or recording large amounts of video for later processing. The Jetson Developer Challenge gave us a target to finish the first stage of our work and demonstrate a proof-of-principle.
Despite all the thorough preparations, were there any unexpected, unpleasant surprises that you’ve encountered?
Even with the start we had when we decided to take part in the competition, it was still quite a crunch to get everything ready in time. Getting a product to a demonstrable state on new hardware is always a challenge, and issues arise that you didn’t expect. For example, we were having performance problems with our object detection implementation that were in fact due to a bug in the way we were drawing boxes on the screen. Once we fixed that bug, we realized that the object detection itself was faster than we had expected, but that had been masked by the impact of this bug.
Satisfying the needs of the contest also proved to be more difficult than we had expected, because gathering and editing all the video footage we needed took quite a bit of time. Luckily, we had equipment that allowed us to capture the video output of the TX1 and TX2 boards, which saved us some effort and made for a cleaner presentation.
We were tuning and refining our software and video presentation up until the last minute, but we feel it all came together well.
Using as much time as you could, that’s noteworthy. Now, what would you say was something completely new you’ve learned while working on this project?
We learned quite a bit about the Jetson TX1/TX2 hardware, the TensorRT accelerated inference library, and overall techniques for deploying convolutional neural networks to hardware like this. The process of porting Rapid Imaging’s georeferencing software to the Jetson TX1/TX2 also required learning quite a bit about the characteristics of this platform.
And besides that, did you encounter any other possibilities with to develop your skills?
In line with the answer above, we expanded our knowledge and skills for handling deployment of convolutional neural networks to devices such as this and became familiar with the development process on these computers. Performance tuning on this class of hardware was something that we spent quite a bit of time on, and our ability to attend NVIDIA’s GTC as a result of this competition allowed us to get hands-on training with NVIDIA engineers in this area. We were able to get all of our questions answered about applying these techniques to the development of our software.
We’re much more confident now in our ability to bring this out as a commercial product as a result.
With working on this project being sort of an elaborate learning process, could you name the most difficult, and the easiest moments?
Getting initially set up on the Jetson TX1 and TX2 computers was probably the easiest part of the competition, because the setup process is straightforward and there is plenty of sample code and direction out there for that.
As I said above, the video production for our demonstration might have been the most difficult part of the process. It surely was one of the most time-consuming aspects because of the footage that needed to be gathered, edited together, and narrated. The product was worth the effort, though.
Now for something different, do you enjoy the idea of hackathons?
That depends on the duration and nature of it. Very short hackathons aren’t my favorite, but a longer-term directed competition like this one are very attractive. I like having a concrete goal to work towards and enough time to make it practical.
You seem very professional in you approach to those events. Does that mean you participate in them often?
It has been a while since I last participated in a short-duration hackathon. I usually don’t feel that I have the ability to work quickly enough to assemble a complete demonstration during the time constraints of your typical weekend-length hackathon.
Would you say it’s possible for you to participate in more hackathons in the future?
If they are structured like this competition, probably. As I said, the shorter hackathons usually don’t give me enough time to assemble something I’m proud to show off. A longer-duration competition with a specific goal is much more interesting to me.
Do you believe hackathons can help you in some way in the future?
If they take place in areas that I’d like to work in or that use new technology, they can be great bootcamps for finally learning that new technology or concept.
As for the NVIDIA Challenge, is there something particularly valuable that you’ve learned, or gained otherwise?
This competition has been hugely useful for everyone on our side. It gave us the structure and motivation to complete a proof-of-principle and taught us a tremendous amount about the Jetson TX1/TX2 hardware. The ability to attend GTC 2018 exposed us to leading-edge research in our fields, and connected us with many potential partners. The hands-on labs alone that we were able to participate in put us months ahead of where we would be otherwise.
Beyond all that, the exposure from being a finalist in the competition has been transformative for our companies and is already helping us out in many ways.
And lastly, if you could, what would be your piece of advice to give to all the people willing to participate in similar hackathons in the future?
I’d make sure that you had a concrete plan for your project, know exactly the problem it is solving and how it will do so, and be able to explain that clearly. Put together a plan for your video presentation and start working on your writeup for the project early, so that you’ll know what you need to record or describe as you’re working on the technology. Get as much as you can done ahead of time, so that when the inevitable crunch comes at the end you only have a few last things to focus on.
If you would like to participate in a similar hackathon visit our website with list of hackathons and challenges. Pick an event that suits your interests. See you soon?
