High-Performance Message Searching in Health Connect

The Problem

Have you ever tried to do a search in Message Viewer on a busy interface and had the query time out? This can become quite a problem as the amount of data increases. For context, the instance of Health Connect I am working with does roughly 155 million Message Headers per day with 21 day message retention. To try and help with search performance, we extended the built-in SearchTable with commonly used fields in hopes that indexing these fields would result in faster query times. Despite this, we still couldn't get some of these queries to finish at all.

Hello. I need to transform a message

FROM:

MSH|^~\&|
SCH||61490||
PID|1||
RGS|1||1
AIS|1||
AIS|2||
AIS|3||
AIL|1||
AIP|1||

TO:

MSH|^~\&|
SCH||61490||
PID|1||
RGS|1||1
AIS|1||
AIL|1||
AIP|1||
RGS|1||1
AIS|2||
AIL|1||
AIP|1||
RGS|1||1
AIS|3||
AIL|1||
AIP|1||

The RGS, AIS, AIL and AIP are all under the RGS group. The one RGS segment that comes in will be copied across the group. If 3 AIS segments come in then I need 3 RGS groups, if 2 I need 2 RGS groups etc.

Hello community,

I wanted to share my experience about working on Large Data projects. Over the years, I have had the opportunity to handle massive patient data, payor data and transactional logs while working in an hospital industry. I have had the chance to build huge reports which had to be written using advanced logics fetching data across multiple tables whose indexing was not helping me write efficient code.

Here is what I have learned about managing large data efficiently.

Choosing the right data access method.

Hello!!!

Data migration often sounds like a simple "move data from A to B task" until you actually do it. In reality, it is a complex process that blends planning, validation, testing, and technical precision.

Over several projects where I handled data migration into a HIS which runs on IRIS (TrakCare), I realized that success comes from a mix of discipline and automation.

Here are a few points which I want to highlight.

1. Start with a Defined Data Format.

This article describes a significant enhancement of how InterSystems IRIS deals with table statistics, a crucial element for IRIS SQL processing, in the 2025.2 release. We'll start with a brief refresher on what table statistics are, how they are used, and why we needed this enhancement. Then, we'll dive into the details of the new infrastructure for collecting and saving table statistics, after which we'll zoom in onto what the change means in practice for your applications. We'll end with a few additional notes on patterns enabled by the new model, and look forward to the follow-on phases of

This will be a short article about Python dunder methods, also known as magic methods.

What are Dunder Methods?

Dunder methods are special methods in Python that start and end with double underscores (__). They allow you to define the behavior of your objects for built-in operations, such as addition, subtraction, string representation, and more.

Some common dunder methods include:

Hi Community,

Enjoy the new video on InterSystems Developers YouTube from our Tech Video Challenge:

⏯ Using Character Slice Index in InterSystems IRIS

Hello,

I have a class with a "Unique" index (pxfactidIndex) on a %Numeric property (pxfactid) (partially-edit code snippet below):

Hey Community,

Enjoy the new video on InterSystems Developers YouTube:

⏯ A better way of buidling index for IRIS

Hi all,

I'm working on a requirement to loop through all encounter streamlets(SDA) to identify specific encounters based on an encounter extension property for a patient fetch request. However, this current process is time-consuming, and we need to create indexes for that property to quickly retrieve the expected results without going through all the encounter streamlets of a patient.

I would appreciate help on how to achieve this, as I couldn't find any documentation explaining how to create indexes on a SDA element.

Thanks in advance.

We recently changed the 'UserID" property in a "User" class from type of %String to be %Library.Username. This is for better consistency across our codebase regarding MAXLEN limit.

%Library.Username is a system wrapper datatype which extends %String and has a MAXLEN of 160. This change should have minimal/no impact on code behavior. However, we found that some SQL query cannot return expected rows after the change. Query will return empty values even if the entry is in the table.

Hello everyone, 

My team is currently developing guidance and best practices for the generation, storage, and deployment of TUNE TABLE statistics across development and production environments. With that in mind, we want to get an idea of what methods teams in the field have developed for handling this data. In particular, we’d like to know the following: 

I have a unique index on two properties in a class, and I want to open the instance by index key with one of the two properties is set to the empty string. Is this possible to perform?

Take the example class here

Class TestIndex
{

Property Mandatory as TestClass [Required];Property Alternate as TestClass;Property AltOrEmpty As%String [ Calculated, Required, SqlComputeCode = {Set {*} = $Case({Alternate},"":$c(0),:{Alternate})}, SqlComputed ];
Index AlternateMandatory on (Mandatory, AltOrEmpty) [ Unique ];
}

And example data as

A recent question from @Vivian Lee reminded me of a rather ancient example.
It was the time when DeepSee's first version was released.
We got Bitmap Index.
And we got BitSlice Index: mapping a numeric value by its binary parts.
So my idea: Why not indexing strings by their characters?
The result of this idea was presented first in June 2008. 
IKnow wasn't publicly available at that time.

We've recently made available a new version of InterSystems IRIS in the Vector Search Early Access Program, featuring a new Approximate Nearest Neighbor index based upon the Hierarchical Navigable Small World (HNSW) indexing algorithm. This addition allows for highly efficient, approximate nearest-neighbor searches over large vector datasets, dramatically improving query performance and scalability.

Suppose you have an application that allows users to write posts and comment on them. (Wait... that sounds familiar...)

For a given user, you want to be able to list all of the published posts with which that user has interacted - that is, either authored or commented on. How do you make this as fast as possible?

Here's what our %Persistent class definitions might look like as a starting point (storage definitions are important, but omitted for brevity):

Hello Community,

As per the Build index documentation "If you use BUILD INDEX on a live system, the index is temporarily labeled as notselectable, meaning that queries cannot use the index while it is being built. Note that this will impact the performance of queries that use the index." Is this  hiding/not selectable is only applicable for BUILD INDEX or it supports class level %BuildIndices as well. as far as my analysis both syntax setting this setting SetMapSelectability. 

Thanks!

Hello IRIS Community,

InterSystems FAQ rubric

By specifying the start and end values ​​of the IDs for which you want to rebuild indexes in the arguments of the %BuildIndices() method provided in the persistent class (=table) definition, you can rebuild only the indexes within that range.

For example, to rebuild the NameIDX index and ZipCode index in the Sample.Person class only for ID=10 to 20, execute the following code (the ID range is specified in the 5th and 6th arguments). 

 set status = ##class(Sample.Person).%BuildIndices($LB("NameIDX","ZipCode"),1,,1,10,20) 

InterSystems FAQ rubric

For volatile tables (tables with many INSERTs and DELETEs), storage for bitmap indexes can become inefficient over time.

For example, suppose that there are thousands of data with the following definition, and the operation of bulk deletion with TRUNCATE TABLE after being retained for a certain period of time is repeatedly performed.

Hi,
just want to know if indexes in IRIS / SQL are autobuilding.

I define an index in a table and then i costantly do a lot of insert/delete in that table. Will i face index problem? Do i need to rebuild it often?

Thanks

An interesting pattern around unique indices came up recently (in internal discussion re: isc.rest) and I'd like to highlight it for the community.

As a motivating use case: suppose you have a class representing a tree, where each node also has a name, and we want nodes to be unique by name and parent node. We want each root node to have a unique name too. A natural implementation would be:

I want to store data in an index global without defining an index in an inherited class.
Example:

Class Aclass [ Abstract ]
{
Index TXSBI On TextSearch(KEYS);
Index TXSSI On TextSimilarity(KEYS) [ Data = TextSimilarity(ELEMENTS) ];
Property TextSearch As %Text(LANGUAGECLASS = "%ZText.Czech", MAXLEN = 1000, XMLPROJECTION = "NONE");
Property TextSimilarity As %Text(LANGUAGECLASS = "%ZText.CzechSim", MAXLEN = 1000, SIMILARITYINDEX = "TXSSI", XMLPROJECTION = "NONE");
...
//other code

Class AAclass Extends (%Persistent, ClassType.SuperClass, Aclass, Bclass...)
{

}

I have a %Persistent class with properties that are of %SerialObject. I want to add an index to a property of the %SerialObject class.

Is this possible?

Hi!

I'd like to know if there are any issues if an index is inserted into a table without running the %BuildIndices() method.

It's important to note that data inserted before the index is not important for retrieval, so it's not a problem data inserted before the index don't show up in queries.

The reason why I'm asking this is that I'd like to avoid index reconstruction on big tables which I need to inser such index.

I'm using Cache 2018.1.

Thanks,

José

Image search like Google's is a nice feature that wonder me - as almost anything related to image processing.

A few months ago, InterSystems released a preview for Python Embedded. As Python has a lot of libs for deal with image processing, I decided to start my own attemptive to play with a sort of image search - a much more modest version in deed :-)

A tast of theory 🤓

See Part 1 here.

Part 2: Index Handling

Now you have a good idea of what kind of indices you need for your class and how to define them. Next, how do you handle them?

Query Plan

This is the first in a pair of articles on SQL indices.

Part 1 - Know your indices

What is an index, anyway?

Picture the last time you went to a library. Typically they have books sorted by subject matter (and then author and title), and each shelf has an end-plate with a code describing the subject of its books. If you wanted to collect books of a certain subject, instead of walking across every aisle and reading the inside cover of every book, you could head straight for the bookshelf labelled with your desired subject matter and choose your books.

I'm using Cache SQL and want the ability to choose a specific index.

I've boiled the problem down to one table and simplified the query down to

SELECT *
FROM Registration.PatResp
WHERE SchedApptNum=8450022

SchedApptNum is indexed, but instead of using that column, "Show Plan" indicates that it's looping through the entire Registration.PatResp table on Id (the primary key for the table).

I've done a tune-table with no change.

SQL provides %IGNOREINDEX, which allows me to ignore a specific index, but I don't need that.  I need the reverse - %CHOOSEINDEX so that I can force SQL to use a specific index.

Is there a way to get a good performing index on a date field? I have tried various date property indexes and the query plan is always in a pretty high range. Below are query plan result values I have observed:

   StartDate > '2019-12-01'  --cost = 699168
   StartDate = '2019-12-21'  --cost 70666
   StartDate between '2019-12-21' and '2019-21-28' --cost = 492058

The query plans above were for type %TimeStamp.