Using Argo to Train Predictive Models at FlightAware
As a Software Engineer 2 on FlightAware's Predictive Technology crew, Andrew Brooks continually works to maintain, analyze, and improve FlightAware's predictive models as well as the software and infrastructure that supports them. In order to support thousands of destinations around the world, we need to train about 3500 different models. Before we can discuss training models at scale, it's necessary to describe the process we use to train a single model. As previously mentioned, we need to train approximately 3500 different models. Surprisingly, the duration of the "Train" phase for an airport does not have a clear relationship with the dataset size: low-traffic airports sometimes take much longer to train than models for high-traffic airports. Thankfully, AWS EC2 has several instance types that have enough memory to train even the largest of our models. Deciding how many instances to provision to train our models is difficult: provisioning too few will require us to wait for a long time for all models to train, yet provisioning too many runs the risk of leaving some instances idle.