Posts

FLASHATTENTION: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022)

Let $N$ be the sequence length. In a standard Transformer, self-attention takes $O(N^2)$ time and $O(N^2)$ memory. Approximate methods such as Reformer: The Efficient Transformer reduce FLOPs by approximating attention, but they often do not yield significant wall-clock speedups and are therefore not widely used. FLASHATTENTION instead targets memory traffic: it reduces data movement between GPU high-bandwidth memory (HBM) and on-chip SRAM, which can be a major bottleneck. HBM is much larger but has lower bandwidth than SRAM; SRAM is fast but small. FLASHATTENTION tiles the computation: it loads blocks of $\mathbf{Q}$, $\mathbf{K}$, and $\mathbf{V}$ into SRAM, updates a block of the output $\mathbf{O}$, and then writes that block back to HBM. By repeating this over all blocks, FLASHATTENTION computes exact attention.

FLASHATTENTION also reduces HBM–SRAM traffic in the backward pass. In a typical implementation, one materializes the attention score matrix $\mathbf{S}$ and the softmax output $\mathbf{P}$ so they can be reused to compute gradients of $\mathbf{Q}$, $\mathbf{K}$, $\mathbf{V}$, and $\mathbf{O}$. When $\mathbf{Q}$, $\mathbf{K}$, $\mathbf{V}\in \mathbb{R}^{N\times d}$, both $\mathbf{S}$ and $\mathbf{P}$ are $O(N^2)$ in size. FLASHATTENTION avoids storing these $N\times N$ matrices by recomputing $\mathbf{S}$ and $\mathbf{P}$ from $\mathbf{O}$ and intermediate values during backpropagation.

#Attention

December 29, 2025

Improving Language Understanding by Generative Pre-Training (2018)

Improving Language Understanding by Generative Pre-Training describes the original generative pre-trained transformer model. The paper shows that pre-training on unlabeled text followed by fine-tuning on supervised tasks leads to strong performance. During fine-tuning, the output of the pre-trained model is passed to a linear output layer.

The model described in the paper is based on a multi-layer Transformer decoder. Unlike the architecture in Attention Is All You Need, this model uses only the decoder stack. Representations for upstream tasks are constructed so that their format aligns with the unlabeled text used in pre-training. For example, an entailment example is created by concatenating the premise and the hypothesis with a delimiter. A text similarity task also concatenates two texts with a delimiter and feeds the sequence into the pre-trained model.

November 16, 2025 (Originally posted on August 7, 2020)

Deep Neural Networks for Youtube Recommendations (2016)

Deep Neural Networks for YouTube Recommendations, published in 2016, describes YouTube’s large-scale recommendation system.
The system consists of two feed-forward neural networks: a candidate generation model and a ranking model, together containing approximately one billion parameters.

The candidate generation network suggests hundreds of videos using collaborative filtering. A user’s representation includes coarse features such as the IDs of watched videos and tokens from search queries. The ranking network predicts the expected watch time of the videos suggested by the candidate generation network.

September 23, 2025

Kademlia: A Peer-to-Peer Information System Based on the XOR Metric (2002)

Mainline DHT, the distributed hash table used by BitTorrent, is based on Kademlia. In Kademlia, both nodes and keys are identified by 160-bit identifiers. The distance between two identifiers is defined as their bitwise XOR interpreted as an integer. To locate other nodes or values, a node repeatedly sends RPC requests to peers whose addresses it knows and that appear closer to the target according to the XOR metric.

#Peer-to-Peer

September 13, 2025

Multi-probe consistent hashing (2015)

Karger et al’s Consistent hashing is widely used for load balancing in content delivery networks and key-value store databases. Consistent hashing maps both keys and nodes to values in the unit interval, and assigns each key to its closest node.

#Consistent Hashing

September 6, 2025

Distributed Caches in Infinispan

Infinispan is an open-source in-memory data grid and forms the core of Red Hat Data Grid. It can also serve as a cache server for KeyCloak, which provides single sign-on and identity brokering.

The official documentation describes Infinispan as a drop-in replacement for Redis or Memcached. It also implements JSR107, which allows Java clients to interact with an Infinispan cluster via the JCache API. For example, a client can use a CacheManager to obtain a Cache—a map-like data structure—and perform updates or lookups.

August 23, 2025

Amazon.com Recommendations Item to Item Collaborative Filtering (2003)

When the journal IEEE Internet Computing was celebrating its 20th anniversary in 2017, its editorial board selected Amazon.com Recommendations: Item-to-Item Collaborative Filtering, published in 2003, as the paper that best stood the test of time. The paper views traditional collaborative filtering as the approach of finding users similar to the target user and recommending items those users have selected but the user has not. Let $N$ be the number of the items. In the traditional approach, each user is represented as an $N$-dimensional vector to measure the similarity between users, the item to item collaborative filtering constructs an $N$ x $N$ matrix, where each component represents the similarity between two items. By computing the item to item matrix offline, the proposed approach can recommend similar items in time independent of both $N$ and the number of users $M$.

#Collaborative Filtering

August 16, 2025

OUTRAGEOUSLY LARGE NEURAL NETWORKS THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER (2017)

Increasing a model’s number of parameters enhances its capacity to learn complex patterns, but it also raises computational costs. OUTRAGEOUSLY LARGE NEURAL NETWORKS: THE SPARSELY-GATED MIXTURE-OF-EXPERTS LAYER (MoE) introduces a method to scale model capacity efficiently by using a gating network that dynamically selects a sparse subset of feed-forward networks (called experts) for each input, while skipping computation for the others.

#Mixture-of-Experts

August 1, 2025

Training Language Models to Follow Instructions With Human Feedback (2022)

Increasing the number of parameters does not inherently improve a model’s ability to follow users’ intent. Large Language Models (LLMs) are typically trained to predict the next token in large-scale internet text corpora, but this objective does not explicitly encourage models to follow user instructions in a helpful and safe manner. [Training Language Models to Follow Instructions with Human Feedback](https://arxiv.org/pdf/2203.02155 introduces a method for aligning models with human intent using Reinforcement Learning from Human Feedback (RLHF).

July 24, 2025

REACT: SYNERGIZING REASONING AND ACTING IN LANGUAGE MODELS (2023)

ReAct (Synergizing Reasoning + Acting) is a prompting strategy that enables large language models (LLMs) to generate reasoning traces and actions in an interleaved manner, allowing them to more effectively combine thought and actions. ReAct provides LLMs with reasoning traces that guide their actions. Additionally, it allows models to incorporate external knowledge obtained through actions into their reasoning process.

July 11, 2025