數(shù)據(jù)湖自身會提供一些元數(shù)據(jù)的信息，我們可以利用這些信息進(jìn)行快速粗略的數(shù)據(jù)校驗
分區(qū)數(shù)據(jù)數(shù)校驗
以Iceberg為例斜姥，catalog.db.tb2.partitions可以得到分區(qū)的元數(shù)據(jù)，里面包括了分區(qū)下數(shù)據(jù)條數(shù)和文件大小等信息
select a.*, b.*
from (
select partition.date, sum(record_count) record_count, 'old' flag
from catalog.db.tb1.partitions
group by partition.date
) as a
full join (
select partition.date, sum(record_count) record_count, 'new' flag
from catalog.db.tb2.partitions
group by partition.date
) as b
on a.date = b.date
where a.record_count is null or b.record_count is null or a.record_count != b.record_count
