The idea

In-Context RALM¹ is the RAG technology for Autoregressive LM. In summary, the RAG technology involves using a retriever during model inference to fetch relevant documents, which are then concatenated with the origin input.

In the In-Context Learning setting, some examples are placed before the user’s input, and then they are fed to LLM. Similarly, the In-Context RALM works in a similar way: it directly concatenates the most relevant retrieved document in front of the model’s input. The advantage is that there’s no need to retrain the LLM. A diagram created with Mermaid is shown below.

Introduction

Recently I was planning to learn the RAG technology so I started to read some related papers. I found a good Roadmap in the ACL 2023 Tutorial’s slide. Today’s topic is the most fundamental one: Retrieval-Augmented Language Model Pre-Training (REALM).

Intro

I have been learning and using OCaml for a while. However, I’m still quite confused about the Polymorphic variant feature. Recently, while exploring the Yojson library, I came across this feature again. After some research, I couldn’t find any comprehensive articles about polymorphic variants (the official introduction is not easy to grasp this feature IMHO), only a few related answers¹². So I decided to write an article, hoping it will be helpful to you :)

Intro

Recently, while using Git with Neovim, I noticed that my workflow isn’t as smooth as I’d like it to be. I tend to exit Neovim, and then type git commands in the terminal. Before committing code changes, I usually check the diff information using delta. To cut down the number of keystrokes, I’ve also enabled the Oh My Zsh’s git plugin, so I can use a bunch of shortcuts like ga = git add and gcmsg = git commit -m

Intro

I like Python’s tuple, which allows us to quickly bundle together values of different types as a single entity and manage them in an intuitive and easy-to-use way. However, I find that once the tuple has many fields, I’m forced to add a comment to indicate the meaning of each field. For example,

Intro

Recently I started learning the GitHub Actions, a feature provided by GitHub that can be used to automate a series of steps. In the process of software development, the most common use of this feature may be the building process. For static-typed programming languages such as C/C++, we are usually required to write the build scripts. The build process involves environment preparation, dependencies download, and build execution. However, automating software builds with GitHub Actions is not the focus of this post. As I was learning this feature, I thought about how I could put it into practice and realized I could use it to automate the build and deployment of my Hugo blog :)

Tail call & Tail-recursion

Assume that function A calls function B. We call function A Caller and function B Callee.

The tail call refers to when the Caller only needs to wait for the return value of the Callee, as everything else has already been completed¹.

If this is a recursive function call(e.g. the Caller and Callee are the same function), then the function is said to be tail-recursive.

Tail-call Optimization (TCO)

According to the mechanism of the function call, a stack frame will be created to store some useful information such as the return address. What if the function call is a tail-call? Could you find any optimization chance?

Intro

You probably do not know what the text-object is in Vim/Neovim. However, you may use it in your daily life. For instance, When you are writing code, you may want to change the arguments of a function. Take the following code as an example, let’s say you want to change the function call to bar(3, 2, 1), and the cursor currently stays on the ,

Background

The secret behind LLM is that it will generate tokens one by one based on all the previous tokens.

Let’s assume that we have already generated $t$ tokens, denoted by $x_{1:t}$. In the next iteration, the LLM will generate $x_{1:t+1}$. Note that the first $t$ tokens are the same.

$$x_{1:t+1}=\text{LLM}(x_{1:t})$$

The next iteration is similar.

$$x_{1:t+2}=\text{LLM}(x_{1:t+1})$$

In summary, in each iteration, we will use the output of the previous round as a new input for the LLM. Generally, this process will continue until the output reaches the maximum length we predefined or the LLM itself generates a special token, signifying the completion of the generating process.