Motivation

The current RAG techniques can not answer the global questions about the corpus. For example, we may want to know what is the topic of the corpus. Usually, the answer does not exist in the corpus but needs to understand the whole corpus and give summarization. Such global questions are called query-focused summarization (QFS) problems in this paper¹. A naive RAG technique can not handle such a situation.

And it’s unrealistic to put all the text in the corpus in the context window of LLM. Even if we could, the LLM may miss the information in the middle of the context window.

Motivations

The model’s performance is related to the model’s parameter. The bigger the model is, the more performant it will be. However, the computational cost also increases. To mitigate this problem, various forms of conditional computation have been proposed to increase model performance without a proportional increase in computational costs¹.

Today I would like to share the Sparsely-Gated Mixture-of-Experts Layer (MoE) as proposed in this paper.¹

MoE architecture

There are $n$ experts in the MoE layer (denoted as $E_1, E_2, …, E_n$) and they are controlled by a gating network $G$. The output of the gating network $G$ is a vector with length $n$.

Intro

If you could understand this statement: Python function is a first-class function, then I believe you should have no problem understanding the Python decorator too. This statement means that the function is also a value, just like any other primitive types (int, str, float, etc), and can be passed as arguments to function or returned as function outputs.

You may heard of the technical term - high-order function, which means that its arguments contain a function or (and) it returns a function. So we know that the Python decorator is a kind of high-order function.

Motivation

A language solves 2 subproblems.

1. Mapping sentence prefixes to fixed-size representation.
2. Using these representations to predict the next token in the context.

The $k\texttt{NN-LM}$ proposed in this hypothesis that representation learning problem may be easier than the prediction problem

kNN-LM

The following graph demonstrates the idea behind the $k\texttt{NN-LM}$ model.

Data Preparation

To use the $k\texttt{NN-LM}$, we need to preprocess the documents in the corpus. The preprocessing procedure can be divided into some steps. Take the following sentence as an example.

What’ is KNN Algorithm

Syntax

The idea

In-Context RALM¹ is the RAG technology for Autoregressive LM. In summary, the RAG technology involves using a retriever during model inference to fetch relevant documents, which are then concatenated with the origin input.

In the In-Context Learning setting, some examples are placed before the user’s input, and then they are fed to LLM. Similarly, the In-Context RALM works in a similar way: it directly concatenates the most relevant retrieved document in front of the model’s input. The advantage is that there’s no need to retrain the LLM. A diagram created with Mermaid is shown below.

Introduction

Recently I was planning to learn the RAG technology so I started to read some related papers. I found a good Roadmap in the ACL 2023 Tutorial’s slide. Today’s topic is the most fundamental one: Retrieval-Augmented Language Model Pre-Training (REALM).

Intro

I have been learning and using OCaml for a while. However, I’m still quite confused about the Polymorphic variant feature. Recently, while exploring the Yojson library, I came across this feature again. After some research, I couldn’t find any comprehensive articles about polymorphic variants (the official introduction is not easy to grasp this feature IMHO), only a few related answers¹². So I decided to write an article, hoping it will be helpful to you :)

Intro

Recently, while using Git with Neovim, I noticed that my workflow isn’t as smooth as I’d like it to be. I tend to exit Neovim, and then type git commands in the terminal. Before committing code changes, I usually check the diff information using delta. To cut down the number of keystrokes, I’ve also enabled the Oh My Zsh’s git plugin, so I can use a bunch of shortcuts like ga = git add and gcmsg = git commit -m