Posts

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer(2020)

The authors of Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer performed experiments with Text-to-Text Transfer Transformer(T5), a unified framework for NLP. The basic idea underlying T5 is to treat various NLP problems as taking text as input and producing new text as output. Their goal is to explore general language learning abilities instead of providing new methods. They are interested in exploring the limits of transfer learning for NLP by scaling up models and data sets beyond what has previously been considered. To perform experiments at scale, they created Colossal Clean Crawled Corpus (C4), a dataset consisting of hundreds of gigabytes of clean English text scraped from the web. They leveraged Common Crawl to create C4. They used heuristic for cleaning up Common Crawl web-extracted text, and used langdetect to filter out any pages that are not classified as English documents.

#Transformer

August 5, 2023

Catalan Numbers(2016)

The Catalan numbers are a sequence of natural numbers that occur in various problems in combinatorial mathematics. For example, the number of expressions containing \(n\) valid pairs of parentheses is the \(n\)th Catalan number. Suppose you have a grid of \(n \times n\) squares, the \(n\)th Catalan number represents the number of paths with a length of \(2n\) that lead from the upper left corner to the lower right corner without intersecting the diagonal dotted line running from the upper left to the lower right. Later in the page, we will derive explicit formulas for the Catalan numbers in two different ways.

#Catalan Numbers

July 29, 2023

Layer Normalization (2016)

Layer Normalization, introduced in the 2016 paper Layer Normalization, is a technique used to normalize the summed inputs to the neurons in a layer by using the mean and standard deviation of those inputs within that layer.

Batch Normalization uses the mean and the standard deviation of the summed inputs of a neuron in a mini-batch to normalize the summed inputs to it. While Batch Normalization can accelerate training, it depends on mini-batch size and is challenging to implement in RNNs.

#Layer Normalization

June 17, 2023