The Writer That Refuses to Guess

With the classic V1 writer, appending to a table that does not exist is not an error. Spark helpfully creates it — schema inferred from whatever this batch happened to contain. Which means a typo in a table name does not fail: it quietly becomes infrastructure.

# V1 — a typo becomes a table
(df.write
   .mode("append")
   .saveAsTable("gold.ordres"))     # gold.ordres didn't exist. Now it does.

# V2 — a typo becomes an error
df.writeTo("gold.ordres").append()  # NoSuchTableException: fail fast, fix the name

The V1 failure mode is nasty precisely because nothing fails. The real table goes stale, the typo table grows, and the first symptom is a dashboard three teams away. The wrong environment variant is worse: a dev job pointed at prod happily manufactures the missing table there.

DataFrameWriterV2 splits intent that V1 blurred: .append() requires the table to exist; creating one is its own explicit act — .create(), or better, DDL owned by deployment while jobs only ever append. As a bonus, V2 resolves columns by name instead of by position, which retires the silent column-order bug that insertInto made possible.

When a table genuinely should be created by code, createOrReplace() is there — the point is not "never create", it’s that creation is a decision someone wrote down, not a side effect of a misspelled append. Fail fast is a feature. V2 simply refuses to guess.