Counting Rows Without Paying for Them

The habit is everywhere: write a table, then call df.count() so the log line can report how many rows landed. That count is not free. It is a second job — Spark re-computes the plan to produce one number for a log message, and on a big transform you just paid for the step twice.

There are two cheaper answers, and they cover different needs. The first is Spark’s Observation API: attach aggregate metrics to the DataFrame, and they get collected as a side effect of the write job that runs anyway.

from pyspark.sql import Observation
import pyspark.sql.functions as F

obs = Observation("write_stats")
observed = df.observe(
    obs,
    F.count(F.lit(1)).alias("rows"),
    F.sum("amount").alias("total_amount"),
)

observed.write.mode("append").saveAsTable("gold.orders")

obs.get   # {'rows': 1284302, ...} — filled by the write job

The second answer is even cheaper: for Delta tables, the write already recorded its own statistics in the transaction log. If all you want is "how many rows did that commit add", the table can tell you for free.

DESCRIBE HISTORY gold.orders LIMIT 1;
-- operationMetrics: numOutputRows, numFiles, numOutputBytes
-- recorded by the write itself, zero extra compute

One limit to respect: an Observation fills only after its action completes, and it evaluates aggregate expressions — it is not a place for sort-dependent logic. But for the everyday case, the rule is simple: the write already knows what it wrote. Ask it, instead of recomputing it.