#InterSystems Natural Language Processing (NLP, iKnow)

Last week at the InterSystems BeNeLux Symposium, we announced the publication of the InterSystems iKnow Natural Language Processing technology to Open Source. This enables developers and researchers to take advantage of this unique technology at any level of their application stack, workflow or experiment.

Yes, yes! Welcome! You haven't made a mistake, you are in your beloved InterSystems Developer Community in Spanish.

You may be wondering what the title of this article is about, well it's very simple, today we are gathered here to honor the Inquisitor and praise the great work he performed. 

So, who or what is the Inquisitor?

Perfect, now that I have your attention, it's time to explain what the Inquisitor is. The Inquisitor is a solution developed with InterSystems technology to subject public contracts published daily on the platform  https://contrataciondelestado.es/ to scrutiny.

I am trying to help another group within our organization access a SQL Table that I have created to populate Epic Department Data within our environment and came across the ability to use SQL Seach REST Interface using iKnow.

However, I am having issues trying to get it to work via POSTMAN before I hand off the solution...

the POST URL... https://<servername>/api/iKnow/latest/TESTCLIN/table/osuwmc_Epic_Clarity.DepartmentMaster/search

where osuwmc_Epic_Clarity.DepartmentMaster is the table

In the body...

Hello All,

Is ther any way to convert RTF content into Text document/HTML docucment in Cache object scripts or by using %iKnow.Source.Converter.

Please advise.

Thanks,

Archunan K

Introduction

To achieve optimized AI performance, robust explainability, adaptability, and efficiency in healthcare solutions, InterSystems IRIS serves as the core foundation for a project within the x-rAI multi-agentic framework. This article provides an in-depth look at how InterSystems IRIS empowers the development of a real-time health data analytics platform, enabling advanced analytics and actionable insights. The solution leverages the strengths of InterSystems IRIS, including dynamic SQL, native vector search capabilities, distributed caching (ECP), and FHIR interoperability. This innovative approach directly aligns with the contest themes of "Using Dynamic SQL & Embedded SQL," "GenAI, Vector Search," and "FHIR, EHR," showcasing a practical application of InterSystems IRIS in a critical healthcare context.

InterSystems has decided to stop further development of the InterSystems IRIS Natural Language Processing, formerly known as iKnow, technology and label it as deprecated as of the 2023.3 release of InterSystems IRIS. InterSystems will continue to support existing customers using the technology, but does not recommend starting new development projects outside of the core text exploration use cases it was originally designed for. Other use cases involving natural language are increasingly well-served using novel techniques based on Large Language Models, an area InterSystems is also

As said in the previous article about the iris-fhir-generative-ai experiment, the project logs all events for analysis. Here we are going to discuss two types of analysis covered by analytics embedded in the project:

• Users prompts
• Execution errors

In order to extract useful data to apply analytics, we used the iknowpy library - an opensource library for Natural Language Processing based in the iKnow for IRIS Data Platform. It makes possible identifies entities (phrases) and their semantic context in natural language text in several languages.

Hello,

I'm looking for a way to write a stored procedure or something to return a ResultSet with Embedded Python.

My goal is the following:

I have a Goal table with a Text field that is free text.

CREATETable Goal (
    Idint,
    TextVARCHAR(5000)
);

I would like to create a procedure that returns all the entities (in the iKnow sense) in a new Entity column.

Python code, i would like to use :

This is the second article in a series on iKnow demo applications, showcasing how the concepts and context provided through iKnow's unique bottom-up approach can be used to implement relevant use cases and help users be more productive in their daily tasks. Last week's article discussed the Knowledge Portal, a straightforward tool to browse iKnow indexing results.

Hi Community,

Enjoy watching the new video on InterSystems Developers YouTube:

⏯ iKnow: Open Source NLP in InterSystems IRIS

I'm working in an application that uses %SIMILARITY to find matches among a set of documents that vary greatly in length. It's generally good but I've noticed issues with ranking short partially-matching documents over longer documents that match the search string entirely.

I'm in a process of acquiring a corpus  of documents on educational courses. 

For example there is an educational course called "OOP" and it can have documents from 2008, 2009, ... 2016 etc.
And there are a lot of these courses, each one with programs from different years (hopefully)

So 1 document is 1 programm of one course for one year.

I want to calculate how much does a course changes per year.

Hi Community,

Find out how you can use the open-source iKnow engine in Python within InterSystems IRIS.

⏯ Using the InterSystems iKnow Engine in Python

Hi Community!

How can I enable IRIS Text Analytics (formerly known as iKnow) in a given Namespace programmatically?

The first full release of our iKnow open-source NLP library is now available on GitHub and PyPI! 

Hi-

I have a SQL Query using %iFind.Highlight which returns text highlighting certain words and phrases.   %iFind.Highlight seems to remove cr/lf from the returned text.

Here's my query

ClassMethod Search(pSessionId As %String, pSearchString As %String) As %String
{
    set tTags="<span style='background-color:yellow;'>"
    &sql(
    SELECT %iFind.Highlight(Text , :pSearchString , , :tTags) into :results 
    FROM SSA_OCR.TempSearchable where sessionId = :pSessionId)
    quit results
}

The returned text looks like this:

Hi

I've been working with SQL using an iFind index to search text.   Using the %iFind.Highlight function in my SELECT statement I can get text back that highlights the found words using <b> and </b>

I am aware that using ##class(%iFind.Utils).Highlight, I can pass a parameter to override the <b> tag and use instead a <span> tag with style to change the background color of the found words.

Is there a way to override the <b> tag from a SQL statement?

Thanks

Introduction

We used the InterSystems iKnow technology to create a review assessment system called iKnow Reviews Analyzer (iKRA). Some information about the prototype of the system can be found here. iKRA analyzes users’ text reviews and automatically rates the object being reviewed. This functionality may come in very handy on e-commerce sites, forums or collections of media content – in other words, everywhere where people discuss products, places or services, for example.

What does the solution do?

This article introduces InterSystems iKnow Entity Browser, a web application which allows to visualize extracted and organized text data mined from a large number of texts, powered by InterSystems iKnow technology, which is also known as InterSystems Text Analytics in InterSystems IRIS. Feel free to play with the demo of this tool or learn more about it on InterSystems Open Exchange.

Hi,

I try to implement an iFind index.

Here is my definition class :

The problem start when I add an Relationship in my indexed class, I end up with this error :

In Cache'/Ensemble, by specifying the objectsPackage parameter, dictionaries (and other objects) get projected to tables that can be accessed by SQL queries.

But in IRIS (IRIS for UNIX (Ubuntu Server LTS for x86-64 Containers) 2019.1 (Build 507U) Mon Feb 25 2019 13:47:16 EST), when I created a dictionary with ##class(%iKnow.Matching.DictionaryAPI).CreateDictionary(), it does not get projected to a table.

The class APIs correctly retrieve information about this dictionary.

Am I missing something with IRIS, or is there any issues about this?

The InterSystems DBMS has a built-in technology for working with non-structured data called iKnow and a full-text search technology called iFind. We decided to take a dive into both and make something useful. As the result, we have DocSearch — a web application for searching in InterSystems documentation using iKnow and iFind.

Hi,

How Can I get sentiment analysis (positive or negative attribute) from a source or entity from iKnow REST API?

Hi Community!

New video "Treating Patients with REST and iKnow" is available now on  DC YouTube Channel:

Has anyone built tag cloud from iKnow domain?

If so, any advice on how to do that?

I have iKnow domain with 1 source, 1 data and 1 metadata fields. Source is a table.

Let's say individual rows are immutable, but new rows are added after the domain is built.

How do I add them to domain?

In  %SYSTEM.iKnow class IndexTable method is available:

classmethod IndexTable(pDomainName As %String, pTableName As %String, pIdField As %String, pGroupField As %String, pDataField As %String, pMetaFields As %List = "", pWhereClause As %String = "", pConfig As %String = "") as %Status

Assuming I have a table App.Text with fields:

I have iKnow domain of forum posts, their full text is an iKnow data, and each post also has a number of views as a metadata field.

I want to get a sum of views by concept. Let's say I have a concept called "TESTEST" and there are 10 sources that have this concept. Each source has some views. I want to get views total - impact of this concept so to say.

What's the best iKnow architecture for this use case?

So far I got this:

I have an iKnow domain and a blacklist called Ads.

I want to get all sources that do not contain entities from Ads list.

Currently I'm doing it like this:

Hi Everyone!

Enjoy a new video on InterSystems Developers YouTube Channel:

Saving Lives and Livelihoods With iKnow

I am trying to create an iKnow domain programmatically like:

    Set dom = ##class(%iKnow.Domain).%New("TestDom")
    Do  dom.SetParameter("DefaultConfig", "MyConfiguration")
    Set sc = dom.%Save()

   ...

Although "MyConfiguration" sets the language to "ja", i.e. Japanese, it doesn't seem to be respected, and what I see on the top right pane in Knowledge Portal is related concepts, instead of proximity profiles, which I expect to see  in Japanese language mode.

Also resulting segmentation of sentences looks to be it is in English mode.

Can someone tell me how I can do this?