Hi, welcome back to the blog where we go through the different compute types, serverless vs non serverless and how to avoid overpaying for the clusters you use.
Serverless
- Provisioned and managed entirely by Databricks.
- Databricks automatically creates the cluster for you and will autoscale resources.
- If you run a heavy query, more resources are added automatically. If it’s idle, resources are reduced.
- Starts up almost instantly.
Non-Serverless (Provisioned)
- You create and configure the cluster yourself (for example, All-purpose compute).
- You choose the size and type of cluster, and you pay for what you’ve provisioned — whether you’re using all of it or not.
- Can take longer to start up (sometimes 10 minutes or more).
- You can configure auto-termination settings to shut down after inactivity and reduce costs.
The Main Compute Types in Databricks
Here’s a breakdown of the compute options you’ll see in Databricks and what they’re for.
1. Serverless SQL Warehouse
- Runs SQL queries in the SQL editor or interactive notebooks.
- Pay-as-you-go model — billed in Databricks Units (DBUs) per hour, based on cluster size.
- Ideal for ad-hoc analytics, quick queries, and scaling with demand.
2. Classic SQL Warehouse
- Same as Serverless SQL Warehouse, but you manage and provision it yourself.
- You decide the size and configuration.
- Starts up slower but gives you full control.
3. Serverless Compute for Notebooks
- Runs SQL or Python directly in notebooks.
- Automatically scales based on workload.
- Great for interactive exploration without worrying about cluster management.
4. Serverless Compute for Jobs
- Automatically provisions and scales clusters for scheduled Lakeflow jobs.
- Databricks handles scaling to speed up job completion and reduce idle time.
5. All-Purpose Compute
- Provisioned manually for interactive analysis.
- You can start, stop, and restart at will.
- Flexible but can lead to higher costs if left running.
6. Jobs Compute
- A one-time, provisioned cluster created for a job run.
- Shuts down immediately after completion.
- You might see many job clusters created over time, but they’re only billed for the runtime.
7. Instance Pools
- Ready for immediate use.
- Reduces cluster startup time.
- Useful for frequent workloads, but not always necessary for light use.
Cost Management Tips
Running Databricks efficiently means keeping costs under control.
Here are my main recommendations:
- Enable Auto-Termination
Set idle time to 10 minutes or less so you’re not paying for unused compute. - Use Serverless Where Possible
Even though hourly rates may be higher, you only pay for what you use. - Right-Size Your Clusters
Avoid spinning up large clusters for small datasets. - Monitor Usage in System Tables
Join data frombillingandusagetables (from the system catalog) to track costs per cluster or compute type over time. - Avoid Leaving Provisioned Clusters Running
Unlike traditional SQL Server where you pay once, Databricks charges for compute while it’s running.
Final Thoughts
Most users will find their biggest costs come from:
- SQL Warehouses
- Compute for notebooks
If you manage these carefully, you’ll avoid unnecessary spend while keeping performance high.
Use the right compute for the right job, keep auto-termination switched on, and monitor your usage regularly in the system tables — and Databricks can be both powerful and cost-effective.
