High-Performance Message Searching in Health Connect

The Problem

Have you ever tried to do a search in Message Viewer on a busy interface and had the query time out? This can become quite a problem as the amount of data increases. For context, the instance of Health Connect I am working with does roughly 155 million Message Headers per day with 21 day message retention. To try and help with search performance, we extended the built-in SearchTable with commonly used fields in hopes that indexing these fields would result in faster query times. Despite this, we still couldn't get some of these queries to finish at all.

Motivation

I didn't know about ObjectScript until I started my new job. Objectscript isn't actually a young programming language. Compared to C++, Java and Python, the community isn't as active, but we're keen to make this place more vibrant, aren't we?

I've noticed that some of my colleagues are finding it tricky to get their heads around the class relationships in these huge projects. There aren't any easy-to-use modern class diagram tool for ObjectScript.

Related Work

I have tried relavant works:

Hello community,

I wanted to share my experience about working on Large Data projects. Over the years, I have had the opportunity to handle massive patient data, payor data and transactional logs while working in an hospital industry. I have had the chance to build huge reports which had to be written using advanced logics fetching data across multiple tables whose indexing was not helping me write efficient code.

Here is what I have learned about managing large data efficiently.

Choosing the right data access method.

Hello!!!

Data migration often sounds like a simple "move data from A to B task" until you actually do it. In reality, it is a complex process that blends planning, validation, testing, and technical precision.

Over several projects where I handled data migration into a HIS which runs on IRIS (TrakCare), I realized that success comes from a mix of discipline and automation.

Here are a few points which I want to highlight.

1. Start with a Defined Data Format.

Technical Documentation — Quarkus IRIS Monitor System

1. Purpose and Scope

This module enables integration between Quarkus-based Java applications and InterSystems IRIS’s native performance monitoring capabilities.
It allows a developer to annotate methods with @PerfmonReport, which triggers IRIS’s ^PERFMON routines automatically around method execution, generating performance reports without manual intervention.

2. System Components

2.1 Annotation: @PerfmonReport

I need to build an integration solution that reads messages from a Kafka topic. The topic has 3 partitions and contains several million messages.

For certain reasons, I can only use the standard EnsLib.Kafka.Service class and cannot use either KafkaClient or Python.

To measure performance and collect statistics I created a simple key + timestamp table with no indexes (so it is unlikely to be a bottleneck). Next, I started an instance of EnsLib.Kafka.Service. In the OnProcessInput method, I receive a message, extract the key from it, get the current time, and write the row to the table.

This article describes a significant enhancement of how InterSystems IRIS deals with table statistics, a crucial element for IRIS SQL processing, in the 2025.2 release. We'll start with a brief refresher on what table statistics are, how they are used, and why we needed this enhancement. Then, we'll dive into the details of the new infrastructure for collecting and saving table statistics, after which we'll zoom in onto what the change means in practice for your applications. We'll end with a few additional notes on patterns enabled by the new model, and look forward to the follow-on phases of

This article outlines the process of utilizing the renowned Jaeger solution for tracing InterSystems IRIS applications. Jaeger is an open-source product for tracking and identifying issues, especially in distributed and microservices environments. This tracing backend that emerged at Uber in 2015 was inspired by Google's Dapper and Twitter's OpenZipkin. It later joined the Cloud Native Computing Foundation (CNCF) as an incubating project in 2017, achieving graduated status in 2019. This guide will demonstrate how to operate the containerized Jaeger solution integrated with IRIS.

Optimizing Performance of Apache Web Server and Web Gateway

By Patrick Jamieson, M.D., Product Technical Manager, InterSystems IRIS for Health

InterSystems IRIS 2024 recently introduced the vector types. This addition empowers developers to work with vector search, enabling efficient similarity searches, clustering, and a range of other applications. In this article, we will delve into the intricacies of vector types, explore their applications, and provide practical examples to guide your implementation.

At its essence, a vector type is a structured collection of numerical values arranged in a predefined order. These values serve to represent different attributes, features, or characteristics of an object.

Hi,

If I test the Native api for Node.js from the documentation, I noticed (if I'm correct) all methods and calls are synchronous. By default due to the nature of Node.js, there is only one thread of execution and normally  all JavaScript methods and all calls should be asynchronous and use either a callback function (the "old way") or promises or the async/await contruct to return their result, e.g.:

• myFunction(params, callbackFunction(response))
• myFunction(params).then(resolveFunction(response))
• async someFunction() {
  ​​​​​  response = await myFunction(params)
  }

In IRIS, every time a request need to be processed, a specific IRIS process (IRISDB.EXE) need to be assigned to handle that request.

If there is no spare IRIS process at that time, process will need to be created (and later destroyed). On some Windows systems (especially with security/antimalware solutions being active) creating new processes can be slow, which can results in delays during peak times (due to inrush of requests).

I have a database that is 2TB in size (from a LIVE system).

I would like to defragment it but I have some questions. I took a look at official documentation but it does not help.

Having been inspired with Shared code execution speed question/discussion, I dare to ask another one which is annoying me and my colleagues for several weeks.

We have a routine called Lib that comprises 200 $$-functions of 1500 code lines total. It was noticed that after calling _any_ function of another rather big routine (1900 functions, 32000 lines) the next call of $$someFunction^Lib(x) is getting 10-20% slower than previous call of the same function. This effect doesn't depend on: 

Hey everyone,

I'm diving deeper into Caché ObjectScript and would love to open a discussion around the most useful tips, tricks, and best practices you’ve learned or discovered while working with it.

Whether you're an experienced developer or just getting started, ObjectScript has its own set of quirks and powerful features—some well-documented, others hidden gems. I’m looking to compile a helpful set of ideas from the community.

Some areas I’m especially interested in:

Hello,
I have created this script that does lot of writes to a single global. DB write performance is much slower than expected (compared to another similar systems).

set rec = "..."//fill it with somethingset time = $piece($horolog,",",2)
while(($piece($horolog,",",2)-time) < 30) //30 secondsset^A($System.Util.CreateGUID()) = rec
}

I have notified the following : 

IRIS and Ensemble are designed to act as an ESB/EAI. This mean they are build to process lots of small messages.

But some times, in real life we have to use them as ETL. The down side is not that they can't do so, but it can take a long time to process millions of row at once.

To improve performance, I have created a new SQLOutboundAdaptor who only works with JDBC.

BatchSqlOutboundAdapter

Extend EnsLib.SQL.OutboundAdapter to add batch batch and fetch support on JDBC connection.

Benchmark

Benchmarks released on Postgres 11.2 with 1 000 000 rows fetched and 100 000 rows inserted on 2 columns.

Introduction

MonLBL is a tool for analyzing the performance of ObjectScript code execution line by line. codemonitor.MonLBL is a wrapper based on the %Monitor.System.LineByLine package from InterSystems IRIS, designed to collect precise metrics on the execution of routines, classes, or CSP pages.

The wrapper and all examples presented in this article are available in the following GitHub repository: iris-monlbl-example

Let's suppose two different routines use one and the same chunk of code. From the object-oriented POV, a good decision is to have this chunk of code in a separate class and have both routines call it. However, whenever you call code outside of the routine as opposed to calling code in the same routine, some execution speed is lost. For reports churning through millions of transactions this lost speed might be noticeable. Any advice how to optimize specifically speed?P.S. Whenever someone is talking about the best choice for whatever, I am always tempted to ask: "What are we optimizing?".

Hello Community,

I have a queue that I need to traverse and perform operations on. To distribute the workload across multiple processes, I used a Work Queue Manager do process and I'm not expecting any status from it So, I skipped Sync / WaitForComplete in my implementation

Hello Community,

The subroutine ^routine is not executed while the queue is being processed in WorkMgr. However, it works when defined as a function. Is it mandatory to define subroutine^routine as a function for it to execute properly?

Introduction

Database performance has become a critical success factor in a modern application environment. Therefore identifying and optimizing the most resource-intensive SQL queries is essential for guaranteeing a smooth user experience and maintaining application stability. 

This article will explore a quick approach to analyzing SQL query execution statistics on an InterSystems IRIS instance to identify areas for optimization within a macro-application.

Since the introduction of Embedded Python there has always been doubt about its performance compared to ObjectScript and on more than one occasion I have discussed this with @Guillaume Rongier , well, taking advantage of the fact that I was making a small application to capture data from public competitions in Spain and to be able to perform searches using the capabilities of VectorSearch I saw the opportunity to carry out a small test.

Data for the test

Public tender information is provided monthly in XML files from this URL  and the typical format of a tender information is as follows:

InterSystems has been at the forefront of database technology since its inception, pioneering innovations that consistently outperform competitors like Oracle, IBM, and Microsoft. By focusing on an efficient kernel design and embracing a no-compromise approach to data performance, InterSystems has carved out a niche in mission-critical applications, ensuring reliability, speed, and scalability.

A History of Technical Excellence

Hello everyone,

I’m facing issues with replicating data from my Caché 2016 database to a PostgreSQL database. I need to handle around 300 data updates per minute, and whenever certain tables are modified, those changes must be reflected in other databases.

So far, I’ve tried various approaches, including:

• Setting up an intermediary API,
• Using Azure Service Bus,
• Leveraging Caché Jobs,
• All of which rely on table triggers as the entry point.

Here at InterSystems, we often deal with massive datasets of structured data. It’s not uncommon to see customers with tables spanning >100 fields and >1 billion rows, each table totaling hundred of GB of data. Now imagine joining two or three of these tables together, with a schema that wasn’t optimized for this specific use case. Just for fun, let’s say you have 10 years worth of EMR data from 20 different hospitals across your state, and you’ve been tasked with finding….   every clinician within your network      who has administered a specific drug         between the years of 2017-2019

There are three things most important to any SQL performance conversation:  Indices, TuneTable, and Show Plan.  The attached PDFs includes historical presentations on these topics that cover the basics of these 3 things in one place.  Our documentation provides more detail on these and other SQL Performance topics in the links below.  The eLearning options reinforces several of these topics.  In addition, there are several Developer Community articles which touch on SQL performance, and those relevant links are also listed.

The rise of Big Data projects, real-time self-service analytics, online query services, and social networks, among others, have enabled scenarios for massive and high-performance data queries. In response to this challenge, MPP (massively parallel processing database) technology was created, and it quickly established itself. Among the open-source MPP options, Presto (https://prestodb.io/) is the best-known option. It originated in Facebook and was utilized for data analytics, but later became open-sourced. However, since Teradata has joined the Presto community, it offers support now.

I use the following code to calculate the SHA1 of a file :

set stream = ##class(%Stream.FileBinary).%New()
do stream.LinkToFile(filename)
write$SYSTEM.Encryption.Base64Encode($SYSTEM.Encryption.SHA1HashStream(stream))

This code is called thousands of time and performance is critical. I have tried to code same logic in another language (which is lower level) and it's almost twice as fast. It's unclear why so I started investigating.

We've recently made available a new version of InterSystems IRIS in the Vector Search Early Access Program, featuring a new Approximate Nearest Neighbor index based upon the Hierarchical Navigable Small World (HNSW) indexing algorithm. This addition allows for highly efficient, approximate nearest-neighbor searches over large vector datasets, dramatically improving query performance and scalability.