New to Vault 2013 is the ability to compress data when uploading or downloading files. By default, file data is uncompressed, just like it has been in previous releases. This article will show you how to enable the compression feature so that you can see performance gains when transferring large amounts of file data.
LZ4 is only compression algorithm that Vault supports. We looked into ZIP, but it was too slow. The time it took to zip and unzip offset the time we saved with smaller network traffic. So we decided on LZ4 because it was much quicker. If you have a slow network or do a lot of file transfer, you will definitely want to enable this feature. Even if you do small amounts of file transfer, it doesn’t hurt to turn the feature on.
The steps to enable compression in Vault are pretty easy. First, you create a CompressionHeader for your DocumentService. Next you transfer the data and do the compression or decompression. It’s best if I just show the source code. The ByteArray class will make much more sense.
NOTE: You will need to get an LZ4 library for the compression and decompression. Here is one such library.
|
// C#
// Compresses a file and uploads it to Vault private File UploadCompressed(string filePath, { mgr.DocumentService.CompressionHeaderValue = { Supported = Compression.LZ4 };
byte[] uncompressed = System.IO.File.ReadAllBytes(filePath); byte[] compressed = LZ4Sharp.LZ4.Compress(uncompressed);
ByteArray bytes = new ByteArray() { Bytes = compressed, Compression = Compression.LZ4, UncompressedSize = uncompressed.Length };
return m_svcMgr.DocumentService.AddFile( vaultFolder.Id, System.IO.Path.GetFileName(filePath), "upload with compression", System.IO.File.GetLastWriteTime(filePath), null, null, FileClassification.None, false, bytes); }
// Downloads a file and uncompresses it. private byte[] DownloadCompressed(long fileId, { mgr.DocumentService.CompressionHeaderValue = { Supported = Compression.LZ4 };
ByteArray byteArray; mgr.DocumentService.DownloadFile(fileId, true,
byte[] uncompressed = null; if (byteArray.Compression == Compression.LZ4) { uncompressed = new byte[byteArray.UncompressedSize]; LZ4Sharp.LZ4.Decompress(byteArray.Bytes, uncompressed); } else if (byteArray.Compression == Compression.None) uncompressed = byteArray.Bytes;
return uncompressed; } |
|
' VB
' Compresses a file and uploads it to Vault Private Function UploadCompressed(filePath As String, _ vaultFolder As Folder, mgr As WebServiceManager) As File
mgr.DocumentService.CompressionHeaderValue = _ New CompressionHeader() mgr.DocumentService.CompressionHeaderValue.Supported = _ Compression.LZ4
Dim uncompressed As Byte() = System.IO.File.ReadAllBytes(filePath) Dim compressed As Byte() = LZ4Sharp.LZ4.Compress(uncompressed)
Dim bytes As New ByteArray() bytes.Bytes = compressed bytes.Compression = Compression.LZ4 bytes.UncompressedSize = uncompressed.Length
Return mgr.DocumentService.AddFile(vaultFolder.Id, _ System.IO.Path.GetFileName(filePath), _ "upload with compression", _ System.IO.File.GetLastWriteTime(filePath), _ Nothing, Nothing, _ FileClassification.None, False, bytes) End Function
' Downloads a file and uncompresses it. ' Returns the uncompressed file bytes. Private Function DownloadCompressed(fileId As Long, _ mgr As WebServiceManager) As Byte()
mgr.DocumentService.CompressionHeaderValue = _ New CompressionHeader() mgr.DocumentService.CompressionHeaderValue.Supported = _ Compression.LZ4
Dim byteArray As ByteArray mgr.DocumentService.DownloadFile(fileId, True, byteArray)
Dim uncompressed As Byte() = Nothing If byteArray.Compression = Compression.LZ4 Then uncompressed = New Byte(byteArray.UncompressedSize – 1) {} LZ4Sharp.LZ4.Decompress(byteArray.Bytes, uncompressed) ElseIf byteArray.Compression = Compression.None Then uncompressed = byteArray.Bytes End If
Return uncompressed End Function |
If you have a large file which needs to be transferred in multiple parts, each part gets compressed/decompressed independently. For example, if a file requires you to call DownloadFilePart twice, you would run LZ4 decompression twice. You would download the first part, decompress it, download the second part, decompress the second part, then combine the two decompressed byte arrays.
Leave a Reply to Doug RedmondCancel reply