Managing Databricks Secrets in the UI vs CLI

What Are Databricks Secrets?

Databricks Secrets provide a secure way to manage sensitive information—like passwords, API keys, and tokens—directly inside your Databricks workspace. They prevent you from hardcoding credentials in notebooks or jobs, helping you avoid accidental leaks (like uploading secrets to GitHub).

Why Use Them?

Secrets in Databricks let you:

• Securely connect to databases, APIs, and cloud storage.
• Control who can access sensitive data through scope-level permissions.
• Keep credentials hidden and redacted when used in notebooks.

Managing Secrets in the UI

In the Databricks UI, you can create secret scopes by navigating to:

https://<your-workspace>#secrets/createScope

From there you can assign a key and password to your scope, then test that it will work:

from databricks.sdk import WorkspaceClient
w = WorkspaceClient()
w.secrets.put_secret("im-the-scope","im-the-key",string_value ="superSecretPassword")

You can see if its working when it doesnt error and shows that its “redacted” or obfuscated so it doesnt show you the password:

password = dbutils.secrets.get(scope = "im-the-scope", key = "im-the-key")
print(password)

Once succesfully tested you can then use the password variable in your code like so:

jdbc_url = "jdbc:sqlserver://myserver.database.windows.net:1433;database=mydb"
connection_properties = {
    "user": "myuser",
    "password": password,   # using the secret here
    "driver": "com.microsoft.sqlserver.jdbc.SQLServerDriver"
}
df = spark.read.jdbc(url=jdbc_url, table="mytable", properties=connection_properties)

Managing Secrets in the CLI

For automation and repeatable workflows, the Databricks CLI is far more efficient.

Here are some commands you can use:

databricks secrets create-scope xxx

➡️ Creates a new secret scope named xxx where you can securely store secrets.

databricks secrets list-scopes

➡️ Lists all existing secret scopes available in your Databricks workspace.

databricks secrets delete-scope xxx

➡️ Deletes the secret scope named xxx and all secrets within it (use with caution!).

databricks secrets create-scope im-a-scope

➡️ Another example creating a new scope called im-a-scope.

databricks secrets put-secret im-a-scope im-a-key --string-value "superSecretPassword"

➡️ Adds or updates a secret called im-a-key inside the im-a-scope scope with the value "superSecretPassword".

databricks secrets get-secret im-a-scope im-a-key

➡️ Retrieves metadata for the secret (not the actual plaintext value, which remains hidden).

databricks secrets delete-secret im-a-scope my-key

➡️ Deletes a secret (e.g., my-key) from the im-a-scope scope.

databricks secrets put-acl im-a-scope contact@journiql.com MANAGE

➡️ Grants the user contact@journiql.com MANAGE permissions on the im-a-scope scope (can control access and settings).

databricks secrets list-acls im-a-scope

➡️ Lists all access control entries (ACLs) for the im-a-scope scope, showing who has what permissions.

databricks secrets get-acl im-a-scope contact@journiql.com

➡️ Displays the specific access level that contact@journiql.com has for the im-a-scope scope.

Functionality & Best Practices

• Secrets are always obfuscated, never printed in plaintext.
• Ideal for JDBC connections, API tokens, and cloud storage keys.
• You can manage access control and rotate passwords easily.
• Integrates with Azure Key Vault for enterprise-grade secret management.

Conclusion

Databricks Secrets are excellent for internal credential management within notebooks and jobs. For large-scale or enterprise use, pairing them with Azure Key Vault offers additional features like versioning, auditing, and cross-service integration.
Use Databricks Secrets for convenience—and Key Vault for compliance.