0
1575

Written by

Claus Odgaard
Developer at Vocast ApS
Follow
Question Claus Odgaard · Apr 11, 2022

MAXSTRING limit issue

#Caché #JSON

MAXSTRING (longstring) is enabled.

We have a Class containing a property definition

Property SettingsJSON As %Text(MAXLEN = 3600000)

The property is used for storing a string of JSON data however in some cases we get a Cache error: %SaveData error when trying to save a string of JSON a lot less than maximum "allowed" length, any ideas anyone? 

Product version: Caché 2017.1
Discussion (14)0
Log in or sign up to continue

Comments

Claus Odgaard  Apr 11, 2022 to Dmitry Maslennikov

Unfortunately not possible to make such changes currently.

But thanks anyway.

0
Dmitry Maslennikov  Apr 11, 2022 to Claus Odgaard

It's the only reasonable way to go. It's a bad idea to store such data as %String

I would suggest wrapping this field with Get and Set to make it compatible with the previous storage

0
Dmitry Maslennikov  Apr 11, 2022 to Vitaliy Serdtsev

Not sure that indexing whole JSON is a case for them

0
Vitaliy Serdtsev  Apr 11, 2022 to Dmitry Maslennikov

I'm not sure if it's %Text at all.

Made a small example:

<FONT COLOR="#000080">Class dc.test Extends %Persistent
</FONT><FONT COLOR="#000000">{

</FONT><FONT COLOR="#000080">Index </FONT><FONT COLOR="#000000">mySimilarityIndex On SettingsJSON(KEYS) [ </FONT><FONT COLOR="#000080">Data </FONT><FONT COLOR="#000000">= SettingsJSON(ELEMENTS) ];
</FONT><FONT COLOR="#000080">Property </FONT><FONT COLOR="#000000">SettingsJSON </FONT><FONT COLOR="#000080">As %Text</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#000080">LANGUAGECLASS </FONT><FONT COLOR="#000000">= </FONT><FONT COLOR="#800080">"%Text.English"</FONT><FONT COLOR="#000000">, </FONT><FONT COLOR="#000080">MAXLEN </FONT><FONT COLOR="#000000">= </FONT><FONT COLOR="#000080">3600000</FONT><FONT COLOR="#000000">, </FONT><FONT COLOR="#000080">SIMILARITYINDEX </FONT><FONT COLOR="#000000">= </FONT><FONT COLOR="#800080">"mySimilarityIndex"</FONT><FONT COLOR="#000000">);
</FONT><FONT COLOR="#000080">ClassMethod </FONT><FONT COLOR="#000000">Test()
{
  </FONT><FONT COLOR="#0000ff">d </FONT><FONT COLOR="#000000">..</FONT><FONT COLOR="#0000ff">%KillExtent</FONT><FONT COLOR="#000000">()
  
  </FONT><FONT COLOR="#0000ff">s </FONT><FONT COLOR="#800000">json</FONT><FONT COLOR="#000000">=</FONT><FONT COLOR="#0000ff">$tr</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#0000ff">$j</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#008000">""</FONT><FONT COLOR="#000000">,3600000),</FONT><FONT COLOR="#008000">" "</FONT><FONT COLOR="#000000">,</FONT><FONT COLOR="#008000">"0"</FONT><FONT COLOR="#000000">)
  </FONT><FONT COLOR="#800080">&sql(</FONT><FONT COLOR="#0000ff">insert </FONT><FONT COLOR="#000080">into </FONT><FONT COLOR="#008000">del</FONT><FONT COLOR="#000000">.</FONT><FONT COLOR="#008000">t</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#008000">SettingsJSON</FONT><FONT COLOR="#000000">) </FONT><FONT COLOR="#000080">values</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">:json</FONT><FONT COLOR="#000000">)</FONT><FONT COLOR="#800080">)
  
  </FONT><FONT COLOR="#0000ff">w $l</FONT><FONT COLOR="#000000">(</FONT><FONT COLOR="#800000">json</FONT><FONT COLOR="#000000">),</FONT><FONT COLOR="#008000">":"</FONT><FONT COLOR="#000000">,</FONT><FONT COLOR="#800000">SQLCODE
</FONT><FONT COLOR="#000000">}
}</FONT>

Output: 3600000:0

0
Claus Odgaard  Apr 12, 2022 to Dmitry Maslennikov

Please see my addon...

0
Claus Odgaard  Apr 12, 2022 to Dmitry Maslennikov

Please see my addon...

0
Steven Hobbs · Apr 11, 2022

If your long strings are coming from JSON representation then

   Set DynObj=##class(%DynamicObject).%FromJSON(...)

will create a  %Library.DynamicObject or %Library.DynamicArray object in memory containing the JSON array/object elements where the sizes are limited only by the amount of virtual memory the platform will allocate to your process.  A string element of an object/array can have many gigabytes of characters (virtual memory permitting) and you can get the value of such a huge string element in the form of an in-memory, read-only %Stream doing:

   Set StreamVal=DynObj.%Get(key,,"stream")

in cases where DynObj.%Get(key) would get a <MAXSTRING>.

The StreamVal (class %Stream.DynamicBinary or %Stream.DynamicCharacter) is a read-only, random-access %Stream and it shares the same buffer space as the 'key' element of the DynObj (class %Library.DynamicObject) so the in-memory %Stream does not need additional virtual memory.

You can then create a persistent object from a class in the %Steam package (%Stream.GlobalBinary, %Stream.GlobalCharacter, %Stream.FileBinary, %Stream.FileCharacter, or some other appropriate class.)  You can then use the CopyFrom method to populate the persistent %Stream from the  read-only, in-memory %Stream.DynamicBinary/Character.

0
Claus Odgaard · Apr 12, 2022

Thanks for the overall input but the problem is that in some cases it seems that a string of json with the length of just a few characters beyond the half of the MAXSTRING limit is unable to saved.

In the class in question there actually is a second property (Property SettingsJSONbackup As %Text(MAXLEN = 3600000)) and it all most seems as if the content of thoose two are calculated (summed) together, but only in same cases?

0
Vitaliy Serdtsev  Apr 12, 2022 to Claus Odgaard

By default, data in the global is stored as

glbD(ID)=$LB(%%CLASSNAME,prop1,prop2,..,propN)

The total size of the string cannot exceed 3641144. Therefore, if you have a field length >3.6E6, and there are several such fields, the limit is exceeded. To work around this, you need to change storage for your class.

For example so:

glbD(ID)=$LB(%%CLASSNAME)glbD(ID,"prop1")=prop1glbD(ID,"prop2")=prop2
...
glbD(ID,"propN")=propN
 

Simple example

Class dc.test Extends %Persistent
{

Index mySimilarityIndex On SettingsJSON(KEYS) [ Data = SettingsJSON(ELEMENTS) ];

Index mySimilaritybackupIndex On SettingsJSONbackup(KEYS) [ Data = SettingsJSONbackup(ELEMENTS) ];

Property SettingsJSON As %Text(LANGUAGECLASS = "%Text.English", MAXLEN = 3600000, SIMILARITYINDEX = "mySimilarityIndex");

Property SettingsJSONbackup As %Text(LANGUAGECLASS = "%Text.English", MAXLEN = 3600000, SIMILARITYINDEX = "mySimilaritybackupIndex");

ClassMethod Test()
{
  
  d ..%KillExtent()
  
  s json=$tr($j("",3600000)," ","0")
  &sql(insert into dc.test(SettingsJSON,SettingsJSONbackup) values(:json,:json))
  
  w $l(json),":",SQLCODE
}

Storage Default
{
<Data name="backup">
  <Attribute>SettingsJSONbackup</Attribute>
  <Structure>node</Structure>
  <Subscript>"SettingsJSONbackup"</Subscript>
</Data>
<Data name="json">
  <Attribute>SettingsJSON</Attribute>
  <Structure>node</Structure>
  <Subscript>"SettingsJSON"</Subscript>
</Data>
<Data name="testDefaultData">
  <Value name="1">
    <Value>%%CLASSNAME</Value>
  </Value>
</Data>
<DataLocation>^dc.testD</DataLocation>
<DefaultData>testDefaultData</DefaultData>
<IdLocation>^dc.testD</IdLocation>
<IndexLocation>^dc.testI</IndexLocation>
<StreamLocation>^dc.testS</StreamLocation>
<Type>%Library.CacheStorage</Type>
}

}
0
Claus Odgaard  Apr 12, 2022 to Vitaliy Serdtsev

Thank you. I think you just solved my problem.

0