All Katas - ML Katas

Transfer Learning with a Pre-trained Model

this month medium (<1 hr) | cnn transfer learning resnet fine-tuning

Fine-tune a **pre-trained model** (e.g., `resnet18` from `torchvision.models`) on a new, small image classification dataset (e.g., `CIFAR-10`). You'll need to freeze the weights of the initial...

Implementing a Siamese Network for Similarity Learning

this month hard (>1 hr) | siamese network similarity contrastive mnist

Build and train a **Siamese network** on a dataset like MNIST. The network takes pairs of images as input and learns to determine if they belong to the same class (a positive pair) or different...

Generative Adversarial Network (GAN) on MNIST

this month hard (>1 hr) | training adversarial generative mnist gan

Implement and train a simple **Generative Adversarial Network (GAN)**. The network consists of a generator and a discriminator. The generator takes a random noise vector and tries to generate a...

Distributed Data Parallel Training

this month hard (>1 hr) | training distributed parallel multi-gpu speed

Set up a **distributed data parallel training** script using `torch.nn.parallel.DistributedDataParallel` and `torch.distributed`. You'll need to use `torch.multiprocessing.spawn` to launch...

Implementing a Simple Attention Mechanism

this month medium (<1 hr) | rnn attention mechanism seq2seq weights

Implement a **simple attention mechanism** for a sequence-to-sequence model. Given a sequence of encoder outputs and a single decoder hidden state, your attention module should compute attention...

Building a Custom `Dataset` and `DataLoader`

this month medium (<30 mins) | dataset dataloader custom data pipeline

Create a **custom `torch.utils.data.Dataset` class** to load a simple, non-image dataset (e.g., from a CSV file). The `__init__` method should read the data, `__len__` should return the total...

Implementing a Graph Neural Network (GNN) Layer

this month hard (>1 hr) | network gnn graph node embedding

Implement a simple **Graph Neural Network (GNN) layer** for node classification on a small graph. The layer should aggregate features from a node's neighbors and combine them with the node's own...

Implementing Weight Initialization Schemes

this month medium (<1 hr) | training initialization weight he xavier

Implement **different weight initialization schemes** (e.g., Xavier/Glorot, He) for a simple neural network. Create a function that iterates through a model's parameters and applies a chosen...

Building a Neural Ordinary Differential Equation (NODE) Layer

this month hard (>1 hr) | node ode continuous differential equation

Implement a simple **Neural Ordinary Differential Equation (NODE) layer**. This involves defining a `torch.nn.Module` that represents the derivative function $f(t, z(t))$ and then using...

Implementing a Simple VAE for Text (Sentence VAE)

this month hard (>1 hr) | rnn vae generative text nlp

Implement a **Variational Autoencoder (VAE)** for text, often called a Sentence VAE. The encoder will be an RNN (e.g., GRU) that outputs a latent distribution, and the decoder will be another RNN...

Differentiating Through a Non-differentiable Function with `torch.autograd.Function`

this month hard (<1 hr) | autograd custom gradient function backprop

Implement a **custom `torch.autograd.Function`** for a non-differentiable operation, such as a custom quantization function. The `forward` method will perform the non-differentiable operation, and...

Implementing Layer Normalization from Scratch

this month medium (<30 mins) | training custom module normalization layer

Implement **Layer Normalization** as a custom `torch.nn.Module`. Unlike `BatchNorm`, `LayerNorm` normalizes across the features of a single sample, not a batch. Your implementation should...

Matrix Multiplication Basics

this month easy (<10 mins) | linear algebra tensors torch.mm

Implement a function in PyTorch that multiplies two matrices using `torch.mm`. ### Problem Write a function `matmul(A, B)` that takes two 2D tensors `A` and `B` and returns their matrix product. -...

ReLU Activation Function

this month easy (<10 mins) | tensors activation nn

Implement the ReLU (Rectified Linear Unit) function in PyTorch. ### Problem Write a function `relu(x)` that takes a 1D tensor and replaces all negative values with 0. - **Input:** A tensor `x` of...

Manual Gradient Descent Step

this month medium (<30 mins) | autograd optimization gradient descent

Simulate one step of gradient descent for a simple quadratic loss. ### Problem Given a scalar parameter $w$ initialized at 5.0, minimize the loss $L(w) = (w - 3)^2$ using PyTorch. - **Input:**...

Custom Dataset Class

this month medium (<30 mins) | dataloader nn datasets

Create a custom PyTorch `Dataset` for pairs of numbers and their sum. ### Problem Implement a dataset where each sample is `(x, y, x+y)`. - **Input:** A list of tuples `(x, y)`. - **Output:** For...

Implement a Simple MLP

this month medium (<30 mins) | mlp nn forward pass

Build and run a minimal Multi-Layer Perceptron (MLP) using `torch.nn`. ### Problem Construct a 2-layer MLP with ReLU activation for input of size 10 and output of size 2. - **Input:** Tensor of...

Implement a Custom Loss Function

this month medium (<30 mins) | pytorch loss functions regression implementation

Create a custom loss function called `MeanAbsolutePercentageError` (MAPE) in PyTorch. It should: 1. Take predictions and targets as input tensors. 2. Compute $$\frac{1}{n} \sum_i \frac{|y_i -...

Custom Dataset for CSV Data

this month medium (<1 hr) | pytorch dataset dataloader tabular data

Write a PyTorch `Dataset` class that loads data from a CSV file containing tabular data (features + labels). Requirements: - Use `pandas` to read the CSV. - Convert features and labels to tensors....

Batch Normalization From Scratch

this month hard (>1 hr) | pytorch implementation batchnorm deep learning

Implement 1D batch normalization manually (without using `nn.BatchNorm1d`). Steps: 1. Compute batch mean and variance. 2. Normalize inputs. 3. Scale and shift with learnable $$\gamma, \beta$$....

Gradient Clipping Example

this month medium (<30 mins) | pytorch rnn training gradients

Write code to: 1. Train a small RNN on dummy data. 2. Add gradient clipping using `torch.nn.utils.clip_grad_norm_`. 3. Print gradient norms before and after clipping. Show that exploding gradients...

Implement a Linear Regression Model

this month easy (<30 mins) | pytorch training linear regression basics

Build a simple linear regression model using `nn.Module`. Requirements: - One input feature, one output. - Train it on synthetic data $$y = 3x + 2 + \epsilon$$. - Use `MSELoss` and `SGD`. Check...

Checkpointing with torch.save

this month easy (<30 mins) | pytorch training basics checkpoints

Train a simple feedforward model for 1 epoch. Save: 1. Model state dict. 2. Optimizer state dict. 3. Epoch number. Then load the checkpoint and resume training seamlessly.

Implement Dropout Manually

this month medium (<1 hr) | pytorch implementation dropout regularization

Implement dropout as a function `my_dropout(x, p)`: - Zero out elements of `x` with probability `p`. - Scale survivors by $$1/(1-p)$$. - Ensure deterministic behavior when `torch.manual_seed` is...

Custom Activation Function

this month medium (<1 hr) | pytorch implementation activations mlp

Define a custom activation function called `Swish`: $$f(x) = x \cdot \sigma(x)$$. - Implement it as a PyTorch `nn.Module`. - Train a small MLP on random data with it. - Compare with ReLU...