Introducing Databricks Runtime 5.0 for Machine Learning

Today we are excited to release the second iteration including Conda support, the latest version of TensorFlow, HorovodRunner API for Distributed Deep Learning training, and performance optimizations for Graphframes and MLlib..Our customers’ excitement and reception of the first experimental release of Databricks Runtime for ML, version 4.1, was beyond our expectations..This encouraged us to move the Runtime to a regular production cadence..Starting with Runtime 5.0, we will release a new Runtime for ML with every new DBR release with the most recent stable versions of the main frameworks, such as TensorFlow..This 5.0 release is available on all Databricks tiers, including the Community Edition..You can find the list of included libraries in our release notes, most notably our new API for distributed deep learning training with HorovodRunner..In addition, we are introducing several key improvements that data scientists and machine learning engineers rely on..Conda Managed Runtime Databricks Runtime 5.0 for ML is the first one on which we use Conda for Python package management..All Python packages are installed in a single environment..This is the same environment our library management will install Egg and PyPi packages into..This is our first step toward a much more data scientist-friendly environment..We will be adding many more features using Conda and make it more prominent as a package manager on our Runtime for ML.. More details

Leave a Reply