0
442

Written by

Dmitry Maslennikov
IRIS Developer Advocate, Software developer at CaretDev, Tabcorp
Follow
Question Dmitry Maslennikov · Mar 30, 2024

Deflate compression

#InterSystems IRIS #ObjectScript

During the realization of some required functionality, I discovered that I need to use Deflate compression, and found, that IRIS does not offer any options, $system.Util.Compress, works one way open (GZIP/COMPRESS) totally different, even for the same algorithms, I see that what I would expect as a result for Deflate somewhere there, but Compress returns something more. Is there any way to get a correct result using just ObjectScript without a need to use external tools? Was it really so difficult to have Deflate implemented in the first place?

set original = "my very long string which needs to be deflated"zwrite original

  write !,"using $system.Util.Compress",!
  set compressed = $system.Util.Compress(original, "zlib")
  zwrite compressed
  zzdump compressed write !
  set deflated = $system.Encryption.Base64Encode(compressed, 1)
  zwrite deflated
  
  set tmpFile = ##class(%File).TempFilename("bin")
  
  set io = $ioopen tmpFile:("WNS":::/COMPRESS="zlib":/GZIP=0):0use tmpFile
  write original
  use io
  close tmpFile

  set stream = ##class(%Stream.FileBinary).%New()
  set stream.Filename = tmpFile
  set stream.RemoveOnClose = 1set compressed = stream.Read(stream.Size)  
  
  write !,"using open file",!
  zwrite compressed
  zzdump compressed write !
  set deflated = $system.Encryption.Base64Encode(compressed, 1)
  zwrite deflated

  write !,"Expected",!
  set base64 = "y61UKEstqlTIyc9LVyguKcoEUuUZmckZCnmpqSnFCiX5CkmpCimpaTmJJakpAA=="set compressed = $system.Encryption.Base64Decode(base64)
  zwrite compressed
  zzdump compressed write !

Why the results are different, and how can I make it understandable by not only IRIS?

What is the point of having a compress function, which says that it uses some standards but only IRIS can decompress?

But results I got from IRIS

ZLIB using OPEN command recognized as GZIP, not a Deflate anyway

Product version: IRIS 2023.3
$ZV: IRIS for UNIX (Ubuntu Server LTS for ARM64 Containers) 2023.3 (Build 254U) Wed Nov 8 2023 13:04:07 EST
Discussion (0)0
Log in or sign up to continue

Comments

Enrico Parisi · Mar 31, 2024

I've been there before on IRIS zlib compression library....

The problem is explained in Compress(), from Class Reference %SYSTEM.Util, Compress() method (emphasis mine):

Returns the compressed form of a given string.
When Type is specified, it is the compress algorithm used to compress the data. Its value can be "zlib", "zstd" or "lz4". If Type is not specified then the "zlib" compression algorithm is used. A compressed string is a compressed data block with a customized wrapper. Use Decompress() to decompress it.

So, after Compress() you need to remove the "customized wrapper":

set compressed=$e(compressed,4,*-5)

Unfortunately Decompress() needs the custom wrapper, and I don't know how to make it up, however, as in your sample, I/O can be used, using file in your sample.

Regarding the file I/O, note that form Open command documentation:

"zlib" — Use the zlib compression library. /COMPRESS="zlib" is equivalent to /GZIP=1

So, if you change:

open tmpFile:("WNS":::/COMPRESS="zlib":/GZIP=0):0

With:

open tmpFile:("WNS":::/GZIP=0):0

It will then work just fine.

I fully agree that it's a pity to have zilib compress/uncompress utility "custimized" with proprietary wrapper that makes it unusable for any "standard use". However in that case you can use I/O, file or other devices. Personally I use XDEV for compress/decompress and leave the silly %SYSTEM.Util.Compress()/Decompress() alone.

0
Dmitry Maslennikov  Apr 1, 2024 to Enrico Parisi

The way to remove the wrapper from the Compress method, did not help. It's still not recognizable. 

Only file way, and removing the first 11 bytes of GZIP header worked for me.

But I suppose, it's just luck. With different data and size of that data, it may not work.

0
Vitaliy Serdtsev · Apr 1, 2024

The resulting archive is easily recognized/unpacked in WinRAR, 7z:

<FONT COLOR="#0000ff">s </FONT><FONT COLOR="#800000">f</FONT><FONT COLOR="#000000">=</FONT><FONT COLOR="#000080">##class</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#008080">%Stream.FileCharacterGzip</FONT><FONT COLOR="#000000">).</FONT><FONT COLOR="#0000ff">%New</FONT><FONT COLOR="#000000">()
</FONT><FONT COLOR="#0000ff">s </FONT><FONT COLOR="#800000">f</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">TranslateTable</FONT><FONT COLOR="#000000">=</FONT><FONT COLOR="#008000">"UTF8"
</FONT><FONT COLOR="#0000ff">s </FONT><FONT COLOR="#800000">f</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">Filename</FONT><FONT COLOR="#000000">=</FONT><FONT COLOR="#008000">"C:\test.gz"
</FONT><FONT COLOR="#0000ff">d </FONT><FONT COLOR="#800000">f</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">WriteLine</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#008000">"test"</FONT><FONT COLOR="#000000">)
</FONT><FONT COLOR="#0000ff">d </FONT><FONT COLOR="#800000">f</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">Write</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#008000">"Привет Caché"</FONT><FONT COLOR="#000000">)
</FONT><FONT COLOR="#0000ff">d </FONT><FONT COLOR="#800000">f</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#0000ff">%Save</FONT><FONT COLOR="#000000">()</FONT>
0
Enrico Parisi · Apr 1, 2024

For DEFLATE use:

open tmpFile:("WNS":::/COMPRESS="DEFLATE"):0

set original = "my very long string which needs to be deflated"zwrite original

	write !,"using $system.Util.Compress",!
	set compressed = $system.Util.Compress(original, "zlib")
	set compressed=$e(compressed,4,*-5)
	zwrite compressed
	zzdump compressed write !
	set deflated = $system.Encryption.Base64Encode(compressed, 1)
	zwrite deflated

	set tmpFile = ##class(%File).TempFilename("bin")

	set io = $ioopen tmpFile:("WNS":::/COMPRESS="DEFLATE"):0use tmpFile
	write original
	use io
	close tmpFile

	set stream = ##class(%Stream.FileBinary).%New()
	set stream.Filename = tmpFile
	set stream.RemoveOnClose = 1set compressed = stream.Read(stream.Size)  

	write !,"using open file",!
	zwrite compressed
	zzdump compressed write !
	set deflated = $system.Encryption.Base64Encode(compressed, 1)
	zwrite deflated

	write !,"Expected",!
	set base64 = "y61UKEstqlTIyc9LVyguKcoEUuUZmckZCnmpqSnFCiX5CkmpCimpaTmJJakpAA=="set compressed = $system.Encryption.Base64Decode(base64)
	zwrite compressed
	zzdump compressed write !

Result:

original="my very long string which needs to be deflated"
 
using $system.Util.Compress
compressed="Ë­T(K-ªTÈÉÏKW(.)Ê"_$c(4)_"Rå"_$c(25,153)_"É"_$c(25,10)_"y©©)Å"_$c(10)_"%ù"_$c(10)_"I©"_$c(10)_")©i9"_$c(137)_"%©)"_$c(0)
 
0000: CB AD 54 28 4B 2D AA 54 C8 C9 CF 4B 57 28 2E 29         Ë­T(K-ªTÈÉÏKW(.)
0010: CA 04 52 E5 19 99 C9 19 0A 79 A9 A9 29 C5 0A 25         Ê.Rå..É..y©©)Å.%
0020: F9 0A 49 A9 0A 29 A9 69 39 89 25 A9 29 00               ù.I©.)©i9.%©).
deflated="y61UKEstqlTIyc9LVyguKcoEUuUZmckZCnmpqSnFCiX5CkmpCimpaTmJJakpAA=="
 
using open file
compressed="Ë­T(K-ªTÈÉÏKW(.)Ê"_$c(4)_"Rå"_$c(25,153)_"É"_$c(25,10)_"y©©)Å"_$c(10)_"%ù"_$c(10)_"I©"_$c(10)_")©i9"_$c(137)_"%©)"_$c(0)
 
0000: CB AD 54 28 4B 2D AA 54 C8 C9 CF 4B 57 28 2E 29         Ë­T(K-ªTÈÉÏKW(.)
0010: CA 04 52 E5 19 99 C9 19 0A 79 A9 A9 29 C5 0A 25         Ê.Rå..É..y©©)Å.%
0020: F9 0A 49 A9 0A 29 A9 69 39 89 25 A9 29 00               ù.I©.)©i9.%©).
deflated="y61UKEstqlTIyc9LVyguKcoEUuUZmckZCnmpqSnFCiX5CkmpCimpaTmJJakpAA=="
 
Expected
compressed="Ë­T(K-ªTÈÉÏKW(.)Ê"_$c(4)_"Rå"_$c(25,153)_"É"_$c(25,10)_"y©©)Å"_$c(10)_"%ù"_$c(10)_"I©"_$c(10)_")©i9"_$c(137)_"%©)"_$c(0)
 
0000: CB AD 54 28 4B 2D AA 54 C8 C9 CF 4B 57 28 2E 29         Ë­T(K-ªTÈÉÏKW(.)
0010: CA 04 52 E5 19 99 C9 19 0A 79 A9 A9 29 C5 0A 25         Ê.Rå..É..y©©)Å.%
0020: F9 0A 49 A9 0A 29 A9 69 39 89 25 A9 29 00               ù.I©.)©i9.%©).
0
Dmitry Maslennikov  Apr 2, 2024 to Enrico Parisi

Yep, that works, thanks, pity that it was not implemented in Compress method, but at least something

0
Lynn Lantz · Jun 11

Just dropping this here because I spent DAYS try to figure out how to get PDF to read a zlib from Intersystems.

You'll see code above to use $e(string,4,*-5) to make it "deflate" but PDF wants ZLIB.

The issue turned out to be that Intersystems adds a first # character and a last character around the zlib compression.

If you do this:

S pdfcompress=$E($System.Util.Compress(instring,"zlib"),2,*-1) ; remove first and last character

THEN the pdf reader properly reads and processes that in the stream object (when using /FlateDecode)

Wasted days just because of 2 added characters.  Hope this helps someone else.

0