README
¶
lz4 : LZ4 compression in pure Go
Overview
This package provides a streaming interface to LZ4 data streams as well as low level compress and uncompress functions for LZ4 data blocks. The implementation is based on the reference C one.
Install
Assuming you have the go toolchain installed:
go get github.com/pierrec/lz4/v4
There is a command line interface tool to compress and decompress LZ4 files.
go install github.com/pierrec/lz4/v4/cmd/lz4c@latest
Usage
Usage of lz4c:
-version
print the program version
Subcommands:
Compress the given files or from stdin to stdout.
compress [arguments] [<file name> ...]
-bc
enable block checksum
-l int
compression level (0=fastest)
-sc
disable stream checksum
-size string
block max size [64K,256K,1M,4M] (default "4M")
Uncompress the given files or from stdin to stdout.
uncompress [arguments] [<file name> ...]
Example
// Compress and uncompress an input string.
s := "hello world"
r := strings.NewReader(s)
// The pipe will uncompress the data from the writer.
pr, pw := io.Pipe()
zw := lz4.NewWriter(pw)
zr := lz4.NewReader(pr)
go func() {
// Compress the input string.
_, _ = io.Copy(zw, r)
_ = zw.Close() // Make sure the writer is closed
_ = pw.Close() // Terminate the pipe
}()
_, _ = io.Copy(os.Stdout, zr)
// Output:
// hello world
Contributing
Contributions are very welcome for bug fixing, performance improvements...!
- Open an issue with a proper description
- Send a pull request with appropriate test case(s)
Contributors
Thanks to all contributors so far!
Special thanks to @Zariel for his asm implementation of the decoder.
Special thanks to @greatroar for his work on the asm implementations of the decoder for amd64 and arm64.
Special thanks to @klauspost for his work on optimizing the code.
Documentation
¶
Overview ¶
Package lz4 implements reading and writing lz4 compressed data.
The package supports both the LZ4 stream format, as specified in http://fastcompression.blogspot.fr/2013/04/lz4-streaming-format-final.html, and the LZ4 block format, defined at http://fastcompression.blogspot.fr/2011/05/lz4-explained.html.
See https://github.com/lz4/lz4 for the reference C implementation.
Example ¶
// Compress and uncompress an input string.
s := "hello world"
r := strings.NewReader(s)
// The pipe will uncompress the data from the writer.
pr, pw := io.Pipe()
zw := lz4.NewWriter(pw)
zr := lz4.NewReader(pr)
go func() {
// Compress the input string.
_, _ = io.Copy(zw, r)
_ = zw.Close() // Make sure the writer is closed
_ = pw.Close() // Terminate the pipe
}()
_, _ = io.Copy(os.Stdout, zr)
Output: hello world
Index ¶
- Constants
- Variables
- func CompressBlock(src, dst []byte, _ []int) (int, error)
- func CompressBlockBound(n int) int
- func CompressBlockHC(src, dst []byte, depth CompressionLevel, _, _ []int) (int, error)
- func UncompressBlock(src, dst []byte) (int, error)
- func UncompressBlockWithDict(src, dst, dict []byte) (int, error)
- func ValidFrameHeader(in []byte) (bool, error)
- type BlockSize
- type CompressingReader
- type CompressionLevel
- type Compressor
- type CompressorHC
- type Option
- func BlockChecksumOption(flag bool) Option
- func BlockSizeOption(size BlockSize) Option
- func ChecksumOption(flag bool) Option
- func CompressionLevelOption(level CompressionLevel) Option
- func ConcurrencyOption(n int) Option
- func LegacyOption(legacy bool) Option
- func OnBlockDoneOption(handler func(size int)) Option
- func SizeOption(size uint64) Option
- type Reader
- type Writer
Examples ¶
Constants ¶
const ( // ErrInvalidSourceShortBuffer is returned by UncompressBlock or CompressBLock when a compressed // block is corrupted or the destination buffer is not large enough for the uncompressed data. ErrInvalidSourceShortBuffer = lz4errors.ErrInvalidSourceShortBuffer // ErrInvalidFrame is returned when reading an invalid LZ4 archive. ErrInvalidFrame = lz4errors.ErrInvalidFrame // ErrInternalUnhandledState is an internal error. ErrInternalUnhandledState = lz4errors.ErrInternalUnhandledState // ErrInvalidHeaderChecksum is returned when reading a frame. ErrInvalidHeaderChecksum = lz4errors.ErrInvalidHeaderChecksum // ErrInvalidBlockChecksum is returned when reading a frame. ErrInvalidBlockChecksum = lz4errors.ErrInvalidBlockChecksum // ErrInvalidFrameChecksum is returned when reading a frame. ErrInvalidFrameChecksum = lz4errors.ErrInvalidFrameChecksum // ErrOptionInvalidCompressionLevel is returned when the supplied compression level is invalid. ErrOptionInvalidCompressionLevel = lz4errors.ErrOptionInvalidCompressionLevel // ErrOptionClosedOrError is returned when an option is applied to a closed or in error object. ErrOptionClosedOrError = lz4errors.ErrOptionClosedOrError // ErrOptionInvalidBlockSize is returned when ErrOptionInvalidBlockSize = lz4errors.ErrOptionInvalidBlockSize // ErrOptionNotApplicable is returned when trying to apply an option to an object not supporting it. ErrOptionNotApplicable = lz4errors.ErrOptionNotApplicable // ErrWriterNotClosed is returned when attempting to reset an unclosed writer. ErrWriterNotClosed = lz4errors.ErrWriterNotClosed )
Variables ¶
var ( DefaultBlockSizeOption = BlockSizeOption(Block4Mb) DefaultChecksumOption = ChecksumOption(true) DefaultConcurrency = ConcurrencyOption(1) )
Default options.
Functions ¶
func CompressBlock ¶
CompressBlock is equivalent to Compressor.CompressBlock. The final argument is ignored and should be set to nil.
This function is deprecated. Use a Compressor instead.
Example ¶
s := "hello world"
data := []byte(strings.Repeat(s, 100))
buf := make([]byte, lz4.CompressBlockBound(len(data)))
var c lz4.Compressor
n, err := c.CompressBlock(data, buf)
if err != nil {
fmt.Println(err)
}
if n >= len(data) {
fmt.Printf("`%s` is not compressible", s)
}
buf = buf[:n] // compressed data
// Allocate a very large buffer for decompression.
out := make([]byte, 10*len(data))
n, err = lz4.UncompressBlock(buf, out)
if err != nil {
fmt.Println(err)
}
out = out[:n] // uncompressed data
fmt.Println(string(out[:len(s)]))
Output: hello world
func CompressBlockBound ¶
CompressBlockBound returns the maximum size of a given buffer of size n, when not compressible.
func CompressBlockHC ¶
func CompressBlockHC(src, dst []byte, depth CompressionLevel, _, _ []int) (int, error)
CompressBlockHC is equivalent to CompressorHC.CompressBlock. The final two arguments are ignored and should be set to nil.
This function is deprecated. Use a CompressorHC instead.
func UncompressBlock ¶
UncompressBlock uncompresses the source buffer into the destination one, and returns the uncompressed size.
The destination buffer must be sized appropriately.
An error is returned if the source data is invalid or the destination buffer is too small.
func UncompressBlockWithDict ¶ added in v4.1.5
UncompressBlockWithDict uncompresses the source buffer into the destination one using a dictionary, and returns the uncompressed size.
The destination buffer must be sized appropriately.
An error is returned if the source data is invalid or the destination buffer is too small.
func ValidFrameHeader ¶ added in v4.1.10
ValidFrameHeader returns a bool indicating if the given bytes slice matches a LZ4 header.
Types ¶
type BlockSize ¶
type BlockSize uint32
BlockSizeIndex defines the size of the blocks to be compressed.
type CompressingReader ¶ added in v4.1.20
type CompressingReader struct {
// contains filtered or unexported fields
}
func NewCompressingReader ¶ added in v4.1.20
func NewCompressingReader(src io.ReadCloser) *CompressingReader
NewCompressingReader creates a reader which reads compressed data from raw stream. This makes it a logical opposite of a normal lz4.Reader. We require an io.ReadCloser as an underlying source for compatibility with Go's http.Request.
func (*CompressingReader) Apply ¶ added in v4.1.20
func (zrd *CompressingReader) Apply(options ...Option) (err error)
Apply applies useful options to the lz4 encoder.
func (*CompressingReader) Close ¶ added in v4.1.20
func (zrd *CompressingReader) Close() error
Close simply invokes the underlying stream Close method. This method is provided for the benefit of Go http client/server, which relies on Close for goroutine termination.
func (*CompressingReader) Read ¶ added in v4.1.20
func (zrd *CompressingReader) Read(p []byte) (n int, err error)
Read allows reading of lz4 compressed data
func (*CompressingReader) Reset ¶ added in v4.1.20
func (zrd *CompressingReader) Reset(src io.ReadCloser)
Reset makes the stream usable again; mostly handy to reuse lz4 encoder instances.
func (*CompressingReader) Source ¶ added in v4.1.20
func (zrd *CompressingReader) Source() io.ReadCloser
Source exposes the underlying source stream for introspection and control.
type CompressionLevel ¶
type CompressionLevel uint32
CompressionLevel defines the level of compression to use. The higher the better, but slower, compression.
const ( Fast CompressionLevel = 0 Level1 CompressionLevel = 1 << (8 + iota) Level2 Level3 Level4 Level5 Level6 Level7 Level8 Level9 )
func (CompressionLevel) String ¶
func (i CompressionLevel) String() string
type Compressor ¶ added in v4.0.3
type Compressor struct {
// contains filtered or unexported fields
}
A Compressor compresses data into the LZ4 block format. It uses a fast compression algorithm.
A Compressor is not safe for concurrent use by multiple goroutines.
Use a Writer to compress into the LZ4 stream format.
func (*Compressor) CompressBlock ¶ added in v4.0.3
func (c *Compressor) CompressBlock(src, dst []byte) (int, error)
CompressBlock compresses the source buffer src into the destination dst.
If compression is successful, the first return value is the size of the compressed data, which is always >0.
If dst has length at least CompressBlockBound(len(src)), compression always succeeds. Otherwise, the first return value is zero. The error return is non-nil if the compressed data does not fit in dst, but it might fit in a larger buffer that is still smaller than CompressBlockBound(len(src)). The return value (0, nil) means the data is likely incompressible and a buffer of length CompressBlockBound(len(src)) should be passed in.
type CompressorHC ¶ added in v4.0.3
type CompressorHC struct {
// Level is the maximum search depth for compression.
// Values <= 0 mean no maximum.
Level CompressionLevel
// contains filtered or unexported fields
}
A CompressorHC compresses data into the LZ4 block format. Its compression ratio is potentially better than that of a Compressor, but it is also slower and requires more memory.
A Compressor is not safe for concurrent use by multiple goroutines.
Use a Writer to compress into the LZ4 stream format.
func (*CompressorHC) CompressBlock ¶ added in v4.0.3
func (c *CompressorHC) CompressBlock(src, dst []byte) (int, error)
CompressBlock compresses the source buffer src into the destination dst.
If compression is successful, the first return value is the size of the compressed data, which is always >0.
If dst has length at least CompressBlockBound(len(src)), compression always succeeds. Otherwise, the first return value is zero. The error return is non-nil if the compressed data does not fit in dst, but it might fit in a larger buffer that is still smaller than CompressBlockBound(len(src)). The return value (0, nil) means the data is likely incompressible and a buffer of length CompressBlockBound(len(src)) should be passed in.
type Option ¶
type Option func(applier) error
Option defines the parameters to setup an LZ4 Writer or Reader.
func BlockChecksumOption ¶
BlockChecksumOption enables or disables block checksum (default=false).
func BlockSizeOption ¶
BlockSizeOption defines the maximum size of compressed blocks (default=Block4Mb).
func ChecksumOption ¶
ChecksumOption enables/disables all blocks or content checksum (default=true).
func CompressionLevelOption ¶
func CompressionLevelOption(level CompressionLevel) Option
CompressionLevelOption defines the compression level (default=Fast).
func ConcurrencyOption ¶
ConcurrencyOption sets the number of go routines used for compression. If n <= 0, then the output of runtime.GOMAXPROCS(0) is used.
func LegacyOption ¶ added in v4.1.0
LegacyOption provides support for writing LZ4 frames in the legacy format.
See https://github.com/lz4/lz4/blob/dev/doc/lz4_Frame_format.md#legacy-frame.
NB. compressed Linux kernel images use a tweaked LZ4 legacy format where the compressed stream is followed by the original (uncompressed) size of the kernel (https://events.static.linuxfound.org/sites/events/files/lcjpcojp13_klee.pdf). This is also supported as a special case.
func OnBlockDoneOption ¶
OnBlockDoneOption is triggered when a block has been processed. For a Writer, it is when is has been compressed, for a Reader, it is when it has been uncompressed.
func SizeOption ¶
SizeOption sets the size of the original uncompressed data (default=0). It is useful to know the size of the whole uncompressed data stream.
type Reader ¶
type Reader struct {
// contains filtered or unexported fields
}
Reader allows reading an LZ4 stream.
func (*Reader) Reset ¶
Reset clears the state of the Reader r such that it is equivalent to its initial state from NewReader, but instead reading from reader. No access to reader is performed.
type Writer ¶
type Writer struct {
// contains filtered or unexported fields
}
Writer allows writing an LZ4 stream.
func (*Writer) Close ¶
Close closes the Writer, flushing any unwritten data to the underlying writer without closing it.
func (*Writer) Flush ¶ added in v4.1.15
Flush any buffered data to the underlying writer immediately.
func (*Writer) ReadFrom ¶
ReadFrom efficiently reads from r and compressed into the Writer destination.
func (*Writer) Reset ¶
Reset clears the state of the Writer w such that it is equivalent to its initial state from NewWriter, but instead writing to writer. Reset keeps the previous options unless overwritten by the supplied ones. No access to writer is performed.
w.Close must be called before Reset or pending data may be dropped.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
internal
|
|
|
lz4block
Package lz4block provides LZ4 BlockSize types and pools of buffers.
|
Package lz4block provides LZ4 BlockSize types and pools of buffers. |
|
lz4stream
Package lz4stream provides the types that support reading and writing LZ4 data streams.
|
Package lz4stream provides the types that support reading and writing LZ4 data streams. |
|
xxh32
Package xxh32 implements the very fast XXH hashing algorithm (32 bits version).
|
Package xxh32 implements the very fast XXH hashing algorithm (32 bits version). |