Context

In this workshop we aim to create a Question-Answer (QA) system using RAG architecture. We are using Solution Ops repository as the corpus on which we would like to be able to easily retrieve answers. The repo has been added as a submodule in the current repository in data.

We have identified that most of the content in Solution Ops is present in code-with-engineeer and code-with-mlops sections. Therefore, we have decided to focus only on those two parts. More information on the analysis can be found here.

We will start with a baseline RAG implementation, where we will take the most obvious choices. Afterwards we will go through two experiments Experiment 1 and Experiment 2 where we will focus on different aspects of a RAG solution, such as: embedding and chunking. Afterwards, we will build a new RAG solution, using the best choices, as resulted in the two experiments.

Finally, we will evaluate the system’s performance end to end.

After looking at different aspects of a RAG solution, experimenting and evaluating, we will look into A/B testing and post-production related topics.