đź‘‹ Hey there! This video is for members only.
Please log in or create a free account below.
Login / Sign Up
One thing that often gets missed when working in Databricks is the cost of queries and what users are running over time. It’s something that can really add up, especially when multiple people are involved across different projects.
Why You Should Be Tracking This
- It gives you visibility into what people are doing and how much it’s costing.
- You can start to spot patterns — like someone constantly running really heavy queries.
- If needed, you can step in and offer training to help people optimise what they’re doing.
- In some cases, it’s a chance to rethink access altogether if things are being overused.
- It just helps you be a bit smarter with how compute is being used.
Auditing with System Tables
The system tables in Databricks are really useful for this. They let you track user behaviour — what’s been run, when, and how much it’s cost. You don’t always get that level of insight unless you dig into this.
- It’s great for auditing, especially if you’re trying to reduce costs over time.
- You get a clear view of which queries are expensive, and who’s running them.
- It’s a way to make sure people are more conscious of the impact of their work.
It’s Not About Policing — It’s About Working Smarter
This isn’t about watching over people’s shoulders. It’s about being aware of what’s going on in the platform and making sure resources are being used in the best way possible.
- You can reduce unnecessary cost without much effort.
- You can support your team better by showing them where they might be going wrong.
- It’s a great way to build a healthier culture around data usage.
Setup
Follow instructions here: https://medium.com/dbsql-sme-engineering/introducing-granular-cost-monitoring-for-databricks-sql-e7ea4e77daf5
If you’re not already tracking this kind of stuff, it’s definitely worth looking into. Even just seeing what’s happening under the hood can help you make better decisions and avoid letting small inefficiencies turn into big costs later down the line.