Deleting a Dependency Instead of Duplicating It

A duplication-cleanup ticket sat between two internal modules. Both created the same Databricks objects — catalog, schema, volume — one through the SDK’s WorkspaceClient, the other through Spark SQL IF NOT EXISTS. Two implementations of one operation; the brief was to consolidate them.

The obvious fix was to add an SDK variant to the shared utilities library so both callers went through it. A reviewer flagged that this would break the library’s consistent SparkSession-based API — so that path was out. The fallback, extracting a private SDK helper, only hides the duplication rather than removing it. Both answers assumed the SDK had to stay.

So the question changed from how do I consolidate this to why is the SDK here at all? The only thing it did that Spark SQL couldn’t was pass properties={} — a metadata dict — when creating the catalog. Everything hinged on whether that one feature was load-bearing.

The 30-second check sql

-- The only SDK-specific feature in play was properties={} on the catalog.
-- Before designing around it: is anything actually reading it?
DESCRIBE CATALOG EXTENDED main;
-- properties don't surface here, aren't in the workspace UI, and a grep
-- across the repo found zero consumers. The only way to read them back
-- was a separate SDK REST call — which nothing in the system made.

The one feature justifying the dependency was observable nowhere.

That single observation collapsed the whole decision. With properties confirmed unread, the SDK had no remaining justification in this layer — so the fix got bigger, not smaller. Replace all three DDL operations with Spark SQL, drop WorkspaceClient from the bootstrap module entirely, and fold the informational intent of properties into COMMENT, which is visible:

The swap python

# Before — the SDK, for a metadata dict nothing reads
WorkspaceClient().catalogs.create(name, properties={"owner": team})

# After — Spark SQL, and the intent that mattered lands somewhere visible
spark.sql(f"CREATE CATALOG IF NOT EXISTS {name} COMMENT '{comment}'")

The dependency wasn’t centralized — it was deleted. The bootstrap module became thin orchestration over a single, consistent Spark SQL surface, and the utilities library kept its one API shape.

The lesson I keep: before you split a thing to make it easier to live with, test whether it should live at all. Invisible metadata isn’t worth a dependency — and a 30-second observability check beats a careful refactor of code that earns nothing.