Can you de-duplicate as a separate step during the load process for duplicate records within the same batch. For removing records from different loads, and thus different epochs, you can use the undocumented 'epoch' column.

select *,epoch from duplicate_table;