If you’re delving into the realm of ChatGPT, you may find yourself disappointed that it’s unable to answer questions about your own business or documentation. Don’t worry! In this easy-to-follow guide, we will show you how to utilize powerful AI models called LLMs (Large Language Models) to build your own private question-answering agent tailored to your business context. With this method, your AI will be able to understand user queries and provide accurate and helpful responses using internal document context. And best of all, this approach can be implemented 100% locally, meaning that it guarantees data privacy by design. Here’s a step-by-step guide on how to get this up and running.

LLMs in a nutshell

Before we go over the steps, let’s first find out what LLMs (Large Language Models) are. LLMs are AI models that are trained on vast amounts of text data to understand and generate human-like language. These models are designed to comprehend and generate natural language in a variety of tasks, such as text completion, translation, summarization and answering questions.

Some of the examples are:

  • GPT-3 is made by OpenAI and is one of the most powerful language models. It has been used for things like chatbots, creating content, and translating languages. It has a lot of parameters (kind of like the building blocks of the model) – 175 billion to be exact!
  • BERT is another popular language model made by Google. It’s been used for tasks like analyzing the sentiment of text, recognizing named entities (like names of people or places) and classifying text into different categories.
  • T5, also made by Google, is a versatile language model that can do many different language processing tasks. It takes an input text and turns it into an output text. It has been used successfully for tasks like translating languages, summarizing text, answering questions and generating new text.

The steps

Step 1: Define and dissect your corpus of documents

The first step is to define a corpus, which simply means gathering a collection of documents that will serve as the knowledge base for your model. These documents can be anything relevant to the information you want your model to answer questions about. Once you have your corpus, it’s time to break it down into smaller, more manageable pieces called snippets of text. Think of them as bite-sized portions of text that your model will use to generate answers. These snippets will be transformed into vectors to help the model retrieve and process information effectively.

Step 2: Converting text snippets into a usable format

Now it’s time to convert your text snippets into a format that can be easily understood and processed by AI algorithms. This process is called embedding. Think of it as translating your text into a mathematical language that the AI algorithms can understand. To do this, we convert each text snippet into a numerical representation called vectors. These vectors capture the essence of the text and allow the AI algorithms to manipulate and analyze them effectively. For this task, any form of document embedding can work. In our demonstration, we used a multilingual BERT sentence transformer locally, to ensure privacy and keep everything within our own system.

Step 3: Store your embedded corpus

After the embedding process, it’s time to store the embedded corpus in a vector database. This database acts as a place where the transformed data can be efficiently stored and retrieved. There are several options available for this purpose, including Redis, Elasticsearch and Postgres, which are production-grade solutions. In our demonstration, we used a vector database called Chroma. It is a flexible and powerful database that can run locally on your system.

Step 4: Embed the user query

Now that your vector database is ready, it’s time to work with the user’s queries. Once you receive a query from a user, it should be embedded using the same embedding method from step 2. This ensures consistency and compatibility between the embedded queries and the document snippets.

Step 5: Retrieve similar text snippets

With the embedded query, you can now retrieve the most similar text snippets from your vector store. Typically, the top four snippets are used, but this number can be adjusted based on the specific requirements of your project. The similarity is generally determined by the cosine similarity measure.

Step 6: Prompt the GPT Model

Finally, it’s time to prompt the GPT model. For this task, we incorporate the retrieved snippets into the prompt, along with the user query. For this purpose, you could use the (Azure) OpenAI API to leverage the power of models such as ChatGPT or GPT-4. Alternatively, if you want to maintain data privacy, you can run your own GPT model locally and use it to run inference on the LLM context window.

With our step-by-step approach, you can seamlessly integrate LLM question-answering into your custom context. Curious about its applications for your business? Our team is ready to help you! We prioritize your data security by making smart architectural choices, ensuring that no confidential information ever leaves your network. Harness the full potential of AI and discover how it can take your organization to the next level!

This blog is written in collaboration with Camiel Kerkhofs – Data Scientist at Mediaan Conclusion.