Hi,
we are having a look into the vertica hadoop connector using pig.
Use case is, that we load data from hdfs and then store it into vertica. Simple tests worked fine so far.
But now... we would like to insert only changed data and new data.
To be honest - I have currently no idea how to do that. Is there a way to store the hdfs data in in tmp table and then merge this somehow with the 'Master'-table, meaning insert the new data and update the changed data?
I would be happy about any suggestions.
Thanks
Herberth
Store into vertica using pig
Moderator: NorbertKrupa
- JimKnicely
- Site Admin
- Posts: 1825
- Joined: Sat Jan 21, 2012 4:58 am
- Contact:
Re: Store into vertica using pig
Hi herberth,
Did you find a solution to your issue?
If not, check out the MERGE command here:
viewtopic.php?f=7&t=153
You should be able to execute this command via Pig...
Thanks!
Did you find a solution to your issue?
If not, check out the MERGE command here:
viewtopic.php?f=7&t=153
You should be able to execute this command via Pig...
Thanks!
Jim Knicely
Note: I work for Vertica. My views, opinions, and thoughts expressed here do not represent those of my employer.
Note: I work for Vertica. My views, opinions, and thoughts expressed here do not represent those of my employer.