What mergeSchema Actually Merges

.option("mergeSchema", "true") gets treated like a universal "make it fit" switch. It is much narrower than that — and the day you learn the difference is usually the day an append fails in production. mergeSchema does one thing: additive schema evolution.

Both behaviours python

# source has a NEW column → appended schema, old rows null
(df_new_cols.write.mode("append")
    .option("mergeSchema", "true")
    .saveAsTable("silver.events"))          # works

# source: quantity BIGINT · target: quantity INT
(df_wider.write.mode("append")
    .option("mergeSchema", "true")
    .saveAsTable("silver.events"))          # AnalysisException

The missing piece is type widening. With delta.enableTypeWidening = true on the target, that second write stops failing: the column widens to BIGINT automatically instead — for supported widening paths only. mergeSchema handles new columns, widening handles growing types; together they cover the schema drift that ingestion actually produces.

And for an arbitrary type change — string → int, a semantic rewrite? That is overwriteSchema with mode("overwrite"): a full-table rewrite. Different tool, different blast radius, and it should feel different when you reach for it.

The mental model that sticks: mergeSchema grows the schema sideways (columns), type widening grows it upward (types), overwriteSchema replaces it. Before flipping any of the three, say out loud which one you actually mean.