In today’s post, I’m diving into Lakeflow Declarative Pipelines in Databricks—what they are, why they’re useful, and how you can get started. These pipelines offer a streamlined approach to batch and streaming data processing, helping you build efficient ETL workflows with minimal code.
What Are Lakeflow Declarative Pipelines?
Lakeflow Declarative Pipelines (LDPs) are Databricks’ way of simplifying the data pipeline process using a declarative, low-code approach. Rather than scripting every detail of your pipeline, you define what you want your data to look like—and Lakeflow handles the rest.
One of the biggest advantages is the ability to incorporate automated data quality checks. While you still define the rules, the framework enforces them consistently, saving time and ensuring cleaner data.
Incremental Processing
A major feature of LDPs is incremental processing.
Instead of reprocessing entire datasets, Lakeflow pipelines detect and ingest only new or changed records. This reduces resource usage and cost, which becomes increasingly important as your data grows.
This feature was previously part of Delta Live Tables, but Databricks has enhanced it further with Lakeflow. The functionality is improving rapidly, making this a great time to start using it.
Real-Time Monitoring in Notebooks
Lakeflow now allows you to monitor pipeline execution directly within a Databricks notebook. This means you no longer need to flip between environments—you can build, run, and inspect everything in one place.
The new UI is clean and modern, and although it may take a little getting used to, it makes the development experience much smoother.
Built-in Data Quality and Validation
Lakeflow pipelines allow for powerful data validation features. For example, you can define rules such as:
- Drop rows with null values in critical fields (e.g., artist name or song title)
- Filter records with invalid durations (e.g., less than or equal to zero)
- Limit processing to specific columns
Once the pipeline runs, the system clearly shows you which records passed or failed validation. You can choose to drop or handle exceptions as needed. There are also options for grouping, aggregating, and transforming your data—all within the same flow.
Advanced use cases like Slowly Changing Dimensions (SCD) Type 1 and 2 are also supported, which I’ll cover in a future post.
Why Use Lakeflow Declarative Pipelines?
If you’re working within the medallion architecture (Bronze, Silver, Gold), Lakeflow provides a modern way to manage and scale your data pipelines. Key benefits include:
- Efficient incremental ingestion
- Seamless notebook integration
- Low-code support for data quality enforcement
- Scalable design for processing small or large volumes of data
Whether you’re just exploring automation or looking to standardize complex pipelines across your team, this is a tool worth considering.
Building a Pipeline: Step-by-Step
Here’s how I set up a pipeline using Databricks and the Microsoft Lakeflow tutorial:
- Start with Example Code
The code I used reads in raw song data and applies filtering rules—dropping records with nulls, validating durations, and selecting only relevant columns. - Create the Pipeline in Databricks
- Navigate to Jobs & Pipelines > Create ETL Pipeline
- Add your existing notebook assets
- Set the root folder to your saved notebook location
- Give your pipeline a clear name (e.g., “LDP Demo”)
- Run the Pipeline
- The updated UI provides helpful prompts and feedback
- You can run the pipeline directly from the notebook or UI
- Watch the data flow in real time and track record counts, validation results, and errors
- Inspect the Output
- Review the final dataset for accepted and dropped records
- Check that all transformations (like grouping or aggregations) were applied correctly
- Monitor performance and processing logic within the same environment
Final Thoughts
Lakeflow Declarative Pipelines are a powerful step forward for data engineering in Databricks. For me, the standout features are:
- Incremental processing that keeps data fresh without waste
- Automated data quality that enforces rules consistently
- Low-code development that enables faster builds and cleaner workflows
If you’re looking to modernize your data pipelines or simply reduce manual effort, I highly recommend giving Lakeflow a try.