We’re building out a data lake in IRIS 2025.1 that aggregates data across multiple business systems and departments. I’m trying to establish best practices for schema design and separation.

Right now, I’m thinking of using a separate schema for each distinct system of record feeding into the data lake - for example, one schema per upstream source system, rather than splitting based on function (e.g. staging, raw, curated). The idea is that this would make it easier to manage source ownership, auditing, and pipeline logic, especially when multiple domains are contributing data.

The invention and popularization of Large Language Models (such as OpenAI's GPT-4) has launched a wave of innovative solutions that can leverage large volumes of unstructured data that was impractical or even impossible to process manually until recently. Such applications may include data retrieval (see Don Woodlock's ML301 course for a great intro to Retrieval Augmented Generation), sentiment analysis, and even fully-autonomous AI agents, just to name a few!

TL;DR

This article introduces using the langchain framework supported by IRIS for implementing a Q&A chatbot, focusing on Retrieval Augmented Generation (RAG). It explores how IRIS Vector Search within langchain-iris facilitates storage, retrieval, and semantic search of data, enabling precise and up-to-date responses to user queries. Through seamless integration and processes like indexing and retrieval/generation, RAG applications powered by IRIS enable the capabilities of GenAI systems for InterSystems developers.

I'm facing an issue while converting an ORU_R01 HL7 message to XML, specifically with the <pidgrpgrp> kind elements. When I use the getvalueat() method before conversion, the XML includes the <pidgrpgrp> and other <grp> elements, but when I don't use the getvalueat() method, the XML is generated without these <grp>elements.

I've attempted to debug the issue using zwrite on the HL7 message before and after calling getvalueat(). Before calling it, the content appears different, and after calling it, the content shows buildmap=1, etc.Please see the xml generated in 2 cases.

What is Unstructured Data?
Unstructured data refers to information lacking a predefined data model or organization. In contrast to structured data found in databases with clear structures (e.g., tables and fields), unstructured data lacks a fixed schema. This type of data includes text, images, videos, audio files, social media posts, emails, and more.

Why Are Insights from Unstructured Data Important?
According to an IDC (International Data Corporation) report, 80% of worldwide data is projected to be unstructured by 2025, posing a significant concern for 95% of businesses. Forbes Article
 

Before we start talking about databases and different data models that exist, first we'd better talk about what a database is and how to use it.

A database is an organized collection of data stored and accessed electronically. It is used to store and retrieve structured, semi-structured, or raw data which is often related to a theme or activity.

At the heart of every database lies at least one model used to describe its data. And depending on the model it is based on, a database may have slightly different characteristics and store different types of data.

Hey community! How are you doing?

I hope to find everyone well, and a happy 2022 to all of you!

Over the years, I've been working on a lot of different projects, and I've been able to find a lot of interesting data.

But, most of the time, the dataset that I used to work with was the customer data. When I started to join the contest in the past couple of years, I began to look for specific web datasets.

I've curated a few data by myself, but I was thinking, "This dataset is enough to help others?"

I have multiple files with different columns, first 9 values are fixed, so i want to ignore the first value, and next 8 values i want to combine into one value using ^ sign

Current Format

|||||||||||^^||||||^^|||||||||||||||||
|||||||||||^^||||^^|||||||||||||||||||||||
|||||||||||^^|||^^||||||||

Desired Format

^^^^^^|||^^||||||^^|||||||||||||||||
^^^^^^|||^^||||^^|||||||||||||||||||||||
^^^^^^|||^^|||^^||||||||

Reading each line from the file use below code.

#dim line as %String = tInput.ReadLine(, .status)

"here i was to put some string function to change format of the data in line variable"

This is the second post of a series explaining how to create an end-to-end Machine Learning system.

Exploring Data

The InterSystems IRIS already has what we need to explore the data: an SQL Engine! For people who used to explore data in csv or text files this could help to accelerate this step. Basically we explore all the data to understand the intersection (joins) which should help to create a dataset prepared to be used by a machine learning algorithm.

Posts Table ( Provided by Intersystems Team )

Tags Table ( Provided by Intersystems Team )

A More Industrial-Looking Global Storage Scheme

In the first article in this series, we looked at the entity–attribute–value (EAV) model in relational databases, and took a look at the pros and cons of storing those entities, attributes and values in tables. We learned that, despite the benefits of this approach in terms of flexibility, there are some real disadvantages, in particular a basic mismatch between the logical structure of the data and its physical storage, which causes various difficulties.

Introduction

In the first article in this series, we’ll take a look at the entity–attribute–value (EAV) model in relational databases to see how it’s used and what it’s good for. Then we'll compare the EAV model concepts to globals.

Sometimes you have objects with an unknown number of fields, or perhaps hierarchically nested fields, for which, as a rule, you need to search.

Introduction

We used the InterSystems iKnow technology to create a review assessment system called iKnow Reviews Analyzer (iKRA). Some information about the prototype of the system can be found here. iKRA analyzes users’ text reviews and automatically rates the object being reviewed. This functionality may come in very handy on e-commerce sites, forums or collections of media content – in other words, everywhere where people discuss products, places or services, for example.

What does the solution do?

Hi Developers,

Please welcome a new video on InterSystems Developers YouTube Channel:

Multi-Model Development

Hi, Community!

This week we have two videos.

Please find the second Developer Community Video of the week on InterSystems Developers YouTube Channel: 

Turning Accountants into Explorers

Hi Community!

Check the new video of the week on the InterSystems Developers YouTube Channel:

Adding Alchemy to Unstructured Data

The Data Platforms department here at InterSystems is gearing up for this year's crop of interns, and I for one am very excited to meet them all next week!

We've got folks from top technical colleges with diverse specialties from hard core engineers to pure computer scientists to mathematicians to business professionals. They come from countries around the world like Vietnam, China, and Finland and they all come with impressive backgrounds. We're sure they will do very well this summer.

This article contains the tutorial document for a Global Summit academy session on Text Categorization and provides a helpful starting point to learn about Text Categorization and how iKnow can help you to implement Text Categorization models. This document was originally prepared by Kerry Kirkham and Max Vershinin and should work based on the sample data provided in the SAMPLES namespace.

A group of students at the Chalmers University of Technology (Gothenburg, Sweden) tried different approaches to automatically rating the quality of emergency calls, including iKnow.

Excerpt: "The most impressive results produced by iKnow is its ability to correctly classify 100% of the calls using the Average algorithm. This is quite surprising since iKnow only compares low-level concepts, how words relates to each other."

Full story: http://publications.lib.chalmers.se/records/fulltext/244534/244534.pdf

Presenter: Danny Wijnschenk
Task: Help people make better decisions by letting application deal with all the data.
Approach: As an example, we’ll extend a demo asset management application for portfolio and trade compliance, using iKnow technology to translate agreements into rules that ensure portfolio compliance prior to trade execution.
 

Presenter: Benjamin De Boe
Task: Extract specialized information from your unstructured data
Approach: Combine InterSystems iKnow technology with third-party and custom text-processing tools
 

This session explains how you can easily combine ISC, third-party and custom text processing tools to get the broadest insights in your unstructured data.

Content related to this session, including slides, video and additional learning content can be found here.

Presenter: Dirk Van Hyfte
Task: Leverage unstructured data to improve how clinicians deliver care
Approach: Give real-world examples of organizations that are benefiting from using their unstructured data
 

Presenter: Misha Bouzinier
Task: Gain an understanding of natural language processing and the current state of the art
Approach: Discuss how InterSystems iKnow technology fits into the NLP ecosystem and complements the output of other components such as Lucene and Stanford NLP tools
 

Introduction - Analyzing Textual Big Data

Big Data for Enriching Analytical Capabilities - Big data is revolutionizing the world of business intelligence and analytics. Gartner predicts that big data will drive $232 billion in spending through 2016, Wikibon claims that by 2017 big data revenue will have grown to $47.8 billion, and McKinsey Global Institute indicates that big data has the potential to increase the value of the US health care industry by $300 billion and to increase the industry value of Europe's public sector administration by Ä250 billion.

Find out what sets the InterSystems Document Data Model apart in the industry.

Hi,

I created with Studio a persistent class with the following field and index:

Property DescriptionDemande As %String(MAXLEN = "");
Index IDXBASDescriptionDemande On (DescriptionDemande) As %iFind.Index.Basic(INDEXOPTION = 1, LANGUAGE = "fr", LOWER = 1);

INDEXOPTION is set to 1 for activating stemming. I'm indexing french  documents. I have set lower to 1 because I want to do non case sensitive search. 

I inserted a single french word "élément" in the field DescriptionDemande for testing purposes using this query: insert into my_table(DescriptionDemande) values(' élément')

Hi,

I created an iKnow domain, where I supplied dictionaries, blacklist, metadata and stemming. The datasource is a table.

I would like to use iFind semantic search feature. It is said in the documentation that iFind use iKnow semantic analysis. But I want iFind to use the iKnow  domain configuration I created earlier earlier. How can I do that ?

Regards,

Jack Abdo.

So calling this lookup manually from the console works as expected:

However, calling it from a method with some concatination to build the key is giving me problems:

Introduction

Experts estimate that 85% of all data exists in unstructured formats – held in e-mails, documents (contracts, memos, clinical notes, legal briefs), social media feeds, etc. Where structured data typically accounts for quantitative facts, the more interesting and potentially more valuable expert opinions and conclusions are often hidden in these unstructured formats. And with massive volumes of text being generated at unprecedented speed, there’s very little chance this information can be made useful without some process of synthesis or automation.