CONTENTS Overview Why do we need yet another deployment framework? Simplifying CI/CD on Databricks via reusable templates Development lifecycle using Databricks Deployments How to create and deploy a new data project with Databricks Labs CI/CD Templates in 10 minutes? Create a new project using the Databricks Labs CI/CD Templates project template Let’s deploy our project to target Databricks workspace Test Automation using Databricks Labs CI/CD Templates Deploying production pipelines using Databricks Deployments Dependency and configuration management How to learn more Outlook and next steps How to contribute? Overview Databricks Labs continuous integration and continuous deployment (CI/CD) Templates are an open source tool that makes it easy for software development teams to use existing CI tooling with Databricks Jobs.
Furthermore, it includes pipeline templates with Databricks’ best practices baked in that run on both Azure and AWS so developers can focus on writing code that matters instead of having to set up full testing, integration and deployment systems from scratch.
CI/CD Templates in 3 steps: Pip install cookiecutter Cookiecutter https://github.
com/databrickslabs/cicd-templates.
git Answer the interactive questions in the terminal such as which cloud you would like to use and you have a full working pipeline.
Pip install databricks_cli && databricks configure –token Start pipeline on Databricks by running .
/run_pipeline.
py pipelines in your project main directory Add your databricks token and workspace URL to github secrets and commit your pipeline to a github repo.
Your Databricks Labs CI/CD pipeline will now automatically run tests against databricks whenever you make a new commit into the repo.
When you are ready to deploy your code, make a github release and templates will automatically package and deploy your pipeline to databricks as a job.
That’s it! You now have a scalable working pipeline which your development team can use and develop off of.
Additionally, you can always modify the template to be more specific to your team or use-case to ensure future projects can be set up with ease.
.