#iFind

Greetings dear community members!

Many of you will remember the NLP capabilities available in IRIS under the name iKnow, which were deprecated not long ago. But...is everything deprecated? NO! A small village resists deprecation: iFind indexes!

What is Unstructured Data?
Unstructured data refers to information lacking a predefined data model or organization. In contrast to structured data found in databases with clear structures (e.g., tables and fields), unstructured data lacks a fixed schema. This type of data includes text, images, videos, audio files, social media posts, emails, and more.

Why Are Insights from Unstructured Data Important?
According to an IDC (International Data Corporation) report, 80% of worldwide data is projected to be unstructured by 2025, posing a significant concern for 95% of businesses. Forbes Article
 

InterSystems has decided to stop further development of the InterSystems IRIS Natural Language Processing, formerly known as iKnow, technology and label it as deprecated as of the 2023.3 release of InterSystems IRIS. InterSystems will continue to support existing customers using the technology, but does not recommend starting new development projects outside of the core text exploration use cases it was originally designed for. Other use cases involving natural language are increasingly well-served using novel techniques based on Large Language Models, an area InterSystems is also

Hello there!I am hoping to get some help with "SQL Search index" and JSON objects.I am storing a JSON object in a column of type %Stream.GlobalCharacter Property JSON As %Stream.GlobalCharacter;I also have a index that looks like below.Index AnalyticIdx On (JSON) As %iFind.Index.Analytic(INDEXOPTION = 0, LANGUAGE = "en", LOWER = 1);The index is updated on insert, but the property names are indexed as well, like firstName in a object like {"FirstName":"Magnus", "LastName":"Guvenal"}, Say I want "Magnus" ie FirstName och not "Güvenal" / LastName to be indexed.Is that supported, if so am I using

I'm working in an application that uses %SIMILARITY to find matches among a set of documents that vary greatly in length. It's generally good but I've noticed issues with ranking short partially-matching documents over longer documents that match the search string entirely.

I have an SQL text index defined like this

Index TextIndex On (Text) As %iFind.Index.Basic(IGNOREPUNCTUATION = 1, INDEXOPTION = 0, LANGUAGE = "en", LOWER = 1);

If I run a query like this:

SELECT
Text
FROM Post
WHERE %ID %FIND search_index("TextIndex",'ABC')

I get 20 results with posts containing the term ABC.

But each post can contain the term ABC several times.

Is there a way to get a total count of ABC term?

Is there a way to get a count of ABC term in each Post?

Hi-

I have a SQL Query using %iFind.Highlight which returns text highlighting certain words and phrases.   %iFind.Highlight seems to remove cr/lf from the returned text.

Here's my query

ClassMethod Search(pSessionId As %String, pSearchString As %String) As %String
{
    set tTags="<span style='background-color:yellow;'>"
    &sql(
    SELECT %iFind.Highlight(Text , :pSearchString , , :tTags) into :results 
    FROM SSA_OCR.TempSearchable where sessionId = :pSessionId)
    quit results
}

The returned text looks like this:

Hi

I've been working with SQL using an iFind index to search text.   Using the %iFind.Highlight function in my SELECT statement I can get text back that highlights the found words using <b> and </b>

I am aware that using ##class(%iFind.Utils).Highlight, I can pass a parameter to override the <b> tag and use instead a <span> tag with style to change the background color of the found words.

Is there a way to override the <b> tag from a SQL statement?

Thanks

Hi,

I try to implement an iFind index.

Here is my definition class :

The problem start when I add an Relationship in my indexed class, I end up with this error :

Have some free text fields in your application that you wish you could search efficiently?  Tried using some methods before but found out that they just cannot match the performance needs of your customers?  Do I have one weird trick that will solve all your problems?  Don’t you already know!?  All I do is bring great solutions to your performance pitfalls!

As usual, if you want the TL;DR (too long; didn’t read) version, skip to the end.  Just know you are hurting my feelings.

The InterSystems DBMS has a built-in technology for working with non-structured data called iKnow and a full-text search technology called iFind. We decided to take a dive into both and make something useful. As the result, we have DocSearch — a web application for searching in InterSystems documentation using iKnow and iFind.

Hi, Community!  

Please find a new session recording from Global Summit 2017:

Catching Bad Guys with iFind and REST 

Hello All,

what is the best method for search a portion of text in a non-indexed global?

I need to implement an autocomplete kind of search, in a global of >1M registers (text type, not $lb)

Maybe the best way would be use a SQL mapped class, with 'Bitmap' indexes?

Thanks in advance!

I have a class with text property, which contains html text (usually pieces, so it may be invalid), here's a sample value:

<div moreinfo="none">Word1 Word2</div><br>
<a href = "123" >Word3</a>

When I add iFind index on text, there are at least two problems:

• Words like moreinfo="none">Word1, so exact match with Word1 returns nothing
• Irrelevant results for href search

How can I pass plaintext into iFnd index?

Enterprises need to grow and manage their global computing infrastructures rapidly and efficiently while simultaneously optimizing and managing capital costs and expenses. Amazon Web Services (AWS) and Elastic Compute Cloud (EC2) computing and storage services meet the needs of the most demanding Caché based application by providing
 a highly robust global computing infrastructure.

Earlier in this series, we've presented four different demo applications for iKnow, illustrating how its unique bottom-up approach allows users to explore the concepts and context of their unstructured data and then leverage these insights to implement real-world use cases. We started small and simple with core exploration through the Knowledge Portal, then organized our records according to content with the Set Analysis Demo, organized our domain knowledge using the Dictionary Builder Demo and finally build complex rules to extract nontrivial patterns from text with the Rules Builder Demo.

I've been using iFind indices in one of my applications but this morning discovered that the index is missing data. I confirmed the issue also exists in our test environment and rebuilt the indices there. That seems to have fixed the immediate problem. However I'd like to understand how this came about to avoid it happening in the future.

Has anyone encountered this before and know the cause?

A simple and rather automated search portal leveraging iFind capabilities for rich text search in 2016.1. It has simple faceting, result ranking, highlighting of search results etc and just works off any table you point it to that has an iFind index by appending ?t=MyPackage.TableName to the URL.

See also https://github.com/bdeboe/isc-iknow-ifindportal for more details and the latest version.

Hi,

I created with Studio a persistent class with the following field and index:

Property DescriptionDemande As %String(MAXLEN = "");
Index IDXBASDescriptionDemande On (DescriptionDemande) As %iFind.Index.Basic(INDEXOPTION = 1, LANGUAGE = "fr", LOWER = 1);

INDEXOPTION is set to 1 for activating stemming. I'm indexing french  documents. I have set lower to 1 because I want to do non case sensitive search. 

I inserted a single french word "élément" in the field DescriptionDemande for testing purposes using this query: insert into my_table(DescriptionDemande) values(' élément')

Hi,

I created an iKnow domain, where I supplied dictionaries, blacklist, metadata and stemming. The datasource is a table.

I would like to use iFind semantic search feature. It is said in the documentation that iFind use iKnow semantic analysis. But I want iFind to use the iKnow  domain configuration I created earlier earlier. How can I do that ?

Regards,

Jack Abdo.