What Are Databricks Secrets? Databricks Secrets provide a secure way to manage sensitive information—like passwords, API keys, and tokens—directly inside your Databricks workspace. They prevent you from hardcoding credentials in notebooks or jobs, helping you avoid accidental leaks (like uploading…
The Databricks CLI (Command Line Interface) is a lightweight, scriptable tool that allows you to interact with your Databricks workspace directly from your terminal. Whether you’re a data engineer, DevOps professional, or developer, the CLI makes it easier to automate…
Working with Excel files in Databricks can feel tricky, especially when you’re handling large datasets or dealing with formatting quirks. In this guide, we’ll explore how to read and write Excel files using Pandas and PySpark, highlight some common challenges,…
In today’s post, we’re going to explore Lakeflow Declarative Pipelines (LDPs) using SQL. I’ve done a couple of videos on LDPs before, but this time I want to explicitly show how it works with SQL. If you haven’t seen the…
Hello and welcome back! 👋 In this post, I’m going to walk through Markdown in Databricks — what it is, why it’s useful, and the advantages of using it (along with a helpful cheat sheet). I’ll also share my own…
Hello and welcome back!In this post, I’ll walk you through how to use parameters in Databricks notebooks and share a few tips and tricks. Parameters are a simple but powerful way to make your notebooks dynamic, reusable, and easier to…
Hello and welcome back to the blog! In today’s post, we’re diving into one of the newest features in Databricks: SQL Stored Procedures. This feature was released just a few weeks ago, and it’s a game-changer for anyone who’s been…
Hello and welcome back! In today’s post, we’ll explore file ingestion using Autoloader in Databricks, specifically how to efficiently process incoming files with incremental ingestion and pair it with Lakeflow Declarative Pipelines for transformations. What is Autoloader? Autoloader is a…
Hi, welcome back to the blog where we go through the different compute types, serverless vs non serverless and how to avoid overpaying for the clusters you use. Serverless Non-Serverless (Provisioned) The Main Compute Types in Databricks Here’s a breakdown…
Being proactive about monitoring your Databricks environment saves time, prevents surprises, and helps you act before small issues turn into big problems. In this guide, I’ll walk you through how to set up proactive alerts across Jobs, Pipelines, Dashboards, Operations,…
