txtai – Low-Code Semantic Search with Transformers

textai transformers BERT semantic search

textai transformers BERT semantic search

A very short intro

NLP tasks gone all to low-code in the previous year with HuggingFace:

Hugging Face – On a mission to solve NLP, one commit at a time, already covered here with My AI Programming Heros During Corona Days – AI Daily News (striki.ai)

Still, if you want to quickly implement semantic search with transformers, you have a plethora of decisions to make:

  • Which BERT-like model to use for closed domain QA
  • How to index the content exposed to search and questions

For small volume of texts it is not worth to spend a time on this. You would probably like some library that implements this out-of-the-box.

Here comes txtai

GitHub – neuml/txtai: AI-powered search engine

And a few links related to ti:

Introducing txtai, an AI-powered search engine built on Transformers | by David Mezzetti | Towards Data Science

Complete Tutorial On Txtai: An AI-Powered Search Engine (analyticsindiamag.com)

textai allows not only search and answering. It also provide indexing the text that is exposed, making the search and queries and answering the questions much faster.

Bonus 0:

I found some unexpected bindings for non-Python languages:

txtai – npm (npmjs.com)

txtai – crates.io: Rust Package Registry

GitHub – neuml/txtai.go: AI-powered search engine for Go

Is AI focus highly biased to Python slowly moving towards JS and Rust? Where is Julia is this picture?

Damn, I’ll miss Python’s tons of libraries for literary speak EVERYTHING, still, don’t expect huge movements here in the next five years. Also, positively surprised with the signs of entering Rust in this area.

Bonus 1

txtai comes also with Zero-shot text classification: you simply describe your target classes with a few words, and voilà: there is a classifier out of the box. Welcome to the transformers & BERT magic. Covered here with Zero-Shot Text Classification – AI Daily News (striki.ai)

No comment

Leave a Reply

Your email address will not be published. Required fields are marked *