Ray Summit 2020

After data scientists train a machine learning (ML) model, the model needs to be served for interactive scoring or batch predictions. The go-to solution is often to wrap the model inside a Flask microservice. But when is that not enough? In this talk, I will discuss the short-comings of the Flask-only solution and then discuss the more common alternative, the “tensor prediction service” approach used by TFServing, SageMaker, and others. I will then introduce an easy-to-use, scalable ML serving system “Ray Serve” that overcomes the deficiencies of the two approaches. I will highlight the architectural innovations in Ray Serve.