Loading compressed files

[nnani]

Hello All,

I was trying to load compressed files using Vertica copy statement

It says cannot read file.
The file has been compressed like this

Code: Select all

compress file_name.txt

After compression file is like this file_name.Z

Vertica is not loading .Z files. Nor we have this option in the COPY statement for loading these files.

Is there any way we can achieve this?

[harryrundles]

Hi,

I believe that compress is an older compression utility. Linux typically uses gzip or bzip.

I tried installing the compress and uncompress utilities on my CentOS box using the command "yum install ncompress". It worked, but when I try to compress a file using the compress command, it doesn't do anything.

I think gunzip can decompress a .Z file. Maybe you can decompress using gunzip and then gzip the file? Have you tried simply renaming the .Z file to use a .gz extension and then try to load it?

Also, from the docs, it says that the COPY LOCAL command can load "compressed" files. I don;t know if that is referring to files compressed with the compressed command or not...

Here is a quote from the SQL Manual:

COPY LOCAL loads data in a platform-neutral way. The COPY LOCAL statement loads all files from a local client system to the HP Vertica host, where the server processes the files. You can copy files in various formats: uncompressed, compressed, fixed-width format, in bzip or gzip format, or specified as a bash glob.

Maybe try a COPY LOCAL?

Good luck!

[nnani]

Hi harry,

I was trying with COPY from local option previously,
This doesn't seem to work.
The three option provided by Vertica are GZIP, BZIP and uncompressed.
I tried it with uncompressed option, but this doesn't work.

[id10t]

Hi!

1. You can use in zcat + copy from stdin
2. znew to recompress .Z files to .gz files


Example of zcat:

Code: Select all

daniel@synapse:~$ file data.Z 
data.Z: compress'd data 16 bits

Code: Select all

daniel@synapse:~$ zcat data.Z 
1|foo|bar
2|egg|etc

Code: Select all

daniel@synapse:~$ vsql -c "create table foo (id int, c1 char(3), c2 char(3))"
CREATE TABLE
daniel@synapse:~$ zcat data.Z | vsql -c "copy foo from stdin direct abort on error"
daniel@synapse:~$ vsql -c "select * from foo"
 id | c1  | c2  
----+-----+-----
  1 | foo | bar
  2 | egg | etc
(2 rows)

[nnani]

Thanks Skwa,

This looks a good solution.