Written by

Question Jonathan Perry · Dec 17, 2024

Converting iris stream to python bytes is very slow

I'm trying to use embedded python code that receives an Iris %Stream.GlobalBinary and uses image manipulation library PIL.

Because PIL can't work with IRIS %Stream, I need to convert the image to python bytes.

This process seems to have very bad performance compared to writing to a file and then loading it from a file in python.

Is there a better way to send a stream into python? The conversion code I'm using is below.

Thanks.

defiris_stream_to_bytes(stream):
	stream.Rewind()
	s = ""whilenot stream.AtEnd:
		r = stream.Read(1024)
		s += r
	b = bytearray()
	b.extend(map(ord, s))
	return b

Product version: IRIS 2024.2

$ZV: IRIS for Windows (x86-64) 2024.2 (Build 247U) Tue Jul 16 2024 09:57:03 EDT

Discussion (0)0

Add reply

Comments

Guillaume Rongier · Dec 18, 2024

I'm using it all the time for string and i haven't seen any performance issue so far.

May be try to increase the buffer :

defstream_to_string(stream,buffer=1000000)-> str:
    string = ""
    stream.Rewind()
    whilenot stream.AtEnd:
        string += stream.Read(buffer)
    return string

0 0

Jonathan Perry Dec 20, 2024 to Guillaume Rongier

Thanks, increasing the buffer to 1000000 reduced the processing of a 18MB image from 20 seconds to 2-3 seconds.

0 0

Vitaliy Serdtsev · Dec 18, 2024

ClassMethod Stream2BytesArray(stream As %Stream.Object) As %SYS.Python { set builtin = $SYSTEM.Python.Builtins()

set b = builtin.bytearray()

do stream.Rewind() while 'stream.AtEnd { set r = $system.Python.Bytes(stream.Read($$$MaxLocalLength)) do b.extend(r) } quit b }

0 0

Jonathan Perry Dec 20, 2024 to Vitaliy Serdtsev

Haven't had the chance to compare this method to converting it through python.

Do you think that doing the conversion on the cache side will be faster?

0 0

Vitaliy Serdtsev Dec 23, 2024 to Jonathan Perry

Simple test

Class dc.test [ Abstract ]
{

ClassMethod Stream2BytesArray1(stream As %Stream.Object) As %SYS.Python [ Language = python ]
{
  stream.Rewind()
  s = ""
  while not stream.AtEnd:
    r = stream.Read(1000000)
    s += r
  b = bytearray()
  b.extend(map(ord, s))
  return b
}

ClassMethod Stream2BytesArray2(stream As %Stream.Object) As %SYS.Python [ Language = python ]
{
  import iris

  maxlen = iris.system.SYS.MaxLocalLength()
  b = bytearray()

  stream.Rewind()
  while not stream.AtEnd:
    b.extend(iris._SYS.Python.Bytes(stream.Read(maxlen)))
  return b
}

ClassMethod Stream2BytesArray3(stream As %Stream.Object) As %SYS.Python
{
  set maxlen = $$$MaxLocalLength,
      b = $SYSTEM.Python.Builtins().bytearray()

  do stream.Rewind()
  while 'stream.AtEnd {do b.extend($system.Python.Bytes(stream.Read(maxlen)))}
  return b
}

/// d ##class(dc.test).Test()
ClassMethod Test()
{
  s builtin=$system.Python.Builtins(),
    maxstr=$tr($j("",$$$MaxStringLength)," ","0"),
    stream=##class(%GlobalBinaryStream).%New()
  f i=1:1:7 d stream.Write(maxstr)
  w "Stream.Size=",stream.Size,!
  
  f n=1:1:3 {
    s fn="Stream2BytesArray"_n,t=$zh,r=$CLASSMETHOD($this,fn,stream),t=($zh-t)_" s."
    w $$$FormatText("%1 len(r)=%2 time=%3",fn,builtin.len(r),t),!
  }
}

}

USER>d ##class(dc.test).Test()
Stream.Size=25488008
Stream2BytesArray1 len(r)=25488008 time=.752708 s.
Stream2BytesArray2 len(r)=25488008 time=.14895 s.
Stream2BytesArray3 len(r)=25488008 time=.080003 s.

The numbers speak for themselves.

0 0