Customizing OpenAI with your Data (2 / 2)

Convert Own Data to Vectors

The first step is take the sgcWebSockets pdf manual and convert to text. We'll use this file to create embeddings with useful information about how to configure the package, use the components... and this information will be used when asking to OpenAI.

Once we've the pdf manual in text format we can use the component TsgcAIOpenAIEmbeddings to obtain the Vectors from OpenAI for every section of the file and in this case we'll store into a text file (for production, you can use a Vector Database like Pinecone).

procedure ConvertFileToVector;
var
  oDialog: TOpenDialog;
  oEmbeddings: TsgcAIOpenAIEmbeddings;
  oFile: TsgcAIDatabaseVectorFile;  
begin
  oDialog := TOpenDialog.Create(nil);
  Try
    oDialog.Filter := 'TXT Files|*.txt';
    if oDialog.Execute then
    begin
      oEmbeddings := TsgcAIOpenAIEmbeddings.Create(nil);
      Try
        oFile := TsgcAIDatabaseVectorFile.Create(nil);
        Try
          oEmbeddings.Database := oFile;
          oEmbeddings.OpenAIOptions.ApiKey := '<your api key>';
          oEmbeddings.CreateEmbeddingsFromFile(oDialog.FileName);
        Finally
          oFile.Free;
        End;
      Finally
        oEmbeddings.Free;
      End;
    end;
  Finally
    FreeAndNil(oDialog);
  End;
end; 

ChatBot & Embeddings

Once we've converted all our data to vectors, we can start to build our own model, the idea behind is very simple, every time we ask the bot, first we convert the question to a vector, then we search into our database which vector is more similar to the question, and finally we use the most similar data to the question and add it as a context.

procedure AskToChatGPT(const aQuestion: string);
var
  oChatBot: TsgcAIOpenAIChatBot;
  oEmbeddings: TsgcAIOpenAIEmbeddings;
  oFile: TsgcAIDatabaseVectorFile;  
  vContext: string;
begin
  oChatBot := TsgcAIOpenAIChatBot.Create(nil);
  Try
    oChatBot.OpenAIOptions.ApiKey := '<your api key>';
    oEmbeddings := TsgcAIOpenAIEmbeddings.Create(nil);
    Try
      oChatBot.Embeddings := oEmbeddings;
      oFile := TsgcAIDatabaseVectorFile.Create(nil);
      Try
        oEmbeddings.Database := oFile;
        vContext := oChatBot.GetEmbedding(aQuestion);
        oChatBot.ChatAsUser('Answer the question based on the context below.\n\nContext:\n' + 
          vContext + '\nQuestion:' + aQuestion + '\nAnswer:');
      Finally
        oFile.Free;
      End;
    Finally
      oEmbeddings.Free;
    End;
  Finally
    FreeAndNil(oDialog);
  End;
end; 

Delphi Windows Demo

Find below a sample for Delphi using the sgcWebSockets AI Package which shows how to build a Questions & Answers application using customized data. The demo contains 2 projects:

1. sgcCreateEmbeddings: this projects loads a text file, split in chunks and convert every chunk into a vector and stores it into a database file. The demo already contains the vector database in text file format for easy of use.

2. sgcQuestionsAnswers: using the previously database records, every time the user ask a question to the chatbot, first the application searches into the database which is the context more similar to the question and adds as a contextual information to help the ChatBot to provide more accurate responses.