Just a few sample records. For example, some sample data we might take in is: z_timestamp - datetime - moid - counter1 - counter2 - counter3 2012-01-01 12:00:00|2012-01-01 11:00:00|RNC-A|300|500|123 2012-01-01 12:00:00|2012-01-01 11:00:00|RNC-B|200|400|456 2012-01-01 12:00:00|2012-01-01 11:00:00|RNC...

It would be easier to explain if you could provide some sample data

Also using the delete function is fine, Vertica recommends you purge the data when the deleted data exceeds 10% of the database storage size.

@ Jim, hmm interesting, could work, i'll try it out and let you know, cheers. The only issue I have is that adding a column to the table, even for a short ammount of time may effect the loading of data to the table (as I said before the tables are being constantly loaded). Also, with the tables some...

That would work, however I didn't explain my situation fully (My apoligies). Copying just duplicate records to a file is far quicker than writing 20+ million rows to a table, then dropping a table with 20 million+ rows (and often 500+ columns) Also, as the tables are constantly being loaded througho...

Ah, I posted old code. The first part where it uses a rank() function is incorrect, that should also use row_number() (as with the second function), as with exact duplicates it is likely that the counter1 field will be exactly the same, so ordering by counter1 in a rank function will not be possible...

Nice idea, could get messy though with large databases!

Hey, so i currently work with a very large database that takes in around 20 million records a day at 15 minute intervals (which vertica is ideal for). One issue that i've come across, is that sometimes (for reasons out of vertica's control), duplicate data comes in. The solution for this that i've c...