ML Katas

### Description Vision Transformers (ViT) process images by first breaking them down into a sequence of flattened patches. The `einops` library is perfectly suited for this task, offering a...

### Description Layer Normalization is a key component in many modern deep learning models, especially Transformers. It normalizes the inputs across the feature dimension. Your task is to...

### Description Implement the softmax function, which converts a vector of numbers into a probability distribution. A naive implementation can be numerically unstable if the input values are very...

### Description Space-to-depth is an operation that rearranges blocks of spatial data into the channel dimension. For an input of shape `(B, C, H, W)` and a block size `S`, the output will be `(B,...

### Description In decoder-style Transformers (like GPT), we need a "causal" or "look-ahead" mask to prevent positions from attending to subsequent positions. This is typically a lower-triangular...

### Description Reversing the order of elements in a sequence is a common operation. While it can be done with slicing (`torch.flip`), let's practice doing it with `einops` for a different...

### Description Implement the dropout layer from scratch. During training, dropout randomly zeroes some of the elements of the input tensor with probability `p`. The remaining elements are scaled...

### Description Implement one-hot encoding for a batch of class indices. Given a 1D tensor of integer labels, create a 2D tensor where each row is a vector of zeros except for a `1` at the index...

### Description Depth-to-space is the inverse of the space-to-depth operation. It rearranges features from the channel dimension into spatial blocks, increasing spatial resolution and decreasing...

### Description Perform a batched matrix multiplication `(B, N, D) @ (B, D, M) -> (B, N, M)` using `einops` `einsum`. While `torch.bmm` is the standard, this is a good exercise to understand how...

### Description Implement the sinusoidal positional encoding from the "Attention Is All You Need" paper. This technique adds information about the position of tokens in a sequence by injecting a...

### Description `torch.squeeze` and `torch.unsqueeze` are common for removing or adding dimensions of size one. `einops.rearrange` can do this as well, often with more clarity by explicitly naming...

### Description Understand the mechanics of a 2D convolution by manually applying a 3x3 filter to a small, single-channel image. This exercise helps demystify what happens inside a...

### Description `torch.gather` is a powerful but sometimes confusing function for selecting elements from a tensor based on an index tensor. Your task is to use it to select specific elements...

### Description **Warning: `as_strided` is an advanced and potentially unsafe operation that can crash your program if used incorrectly, as it creates a view on memory without checks.** With that...

### Description The `einops.repeat` function is a powerful and readable alternative to `torch.expand` or `torch.tile` for broadcasting or repeating a tensor along new or existing dimensions. ###...

### Description `torch.scatter_add_` is used to add values into a tensor at specified indices. It's useful in cases like converting an edge list in a graph to an adjacency matrix or pooling...