All Katas - ML Katas

Differentiable Additive Synthesizer

yesterday medium (<30 mins) | pytorch generative ddsp audio

### Description Differentiable Digital Signal Processing (DDSP) is a technique that combines classic signal processing with deep learning by making the parameters of synthesizers learnable via...

Implement a Knowledge Distillation Loss

yesterday medium (<30 mins) | pytorch compression distillation

### Description Knowledge Distillation is a model compression technique where a small "student" model is trained to mimic a larger, pre-trained "teacher" model. [1] This is achieved by training...

Masked Autoencoder (MAE) Input Preprocessing

yesterday medium (<30 mins) | pytorch self-supervised mae vision transformer

### Description Masked Autoencoders (MAE) are a powerful self-supervised learning technique for vision transformers. The core idea is simple: randomly mask a large portion of the input image...

Deep Canonical Correlation Analysis (DCCA) Loss

yesterday medium (<30 mins) | pytorch dcca correlation multimodal

### Description Canonical Correlation Analysis (CCA) is a statistical method for finding correlations between two sets of variables. Deep CCA (DCCA) uses neural networks to first project two...

Gradient Reversal Layer

yesterday medium (<30 mins) | pytorch autograd gan domain adaptation

### Description Implement a Gradient Reversal Layer (GRL), a key component in Domain-Adversarial Neural Networks (DANNs). [1] The GRL acts as an identity function during the forward pass but...

Tiny Neural Radiance Fields (NeRF)

yesterday medium (<30 mins) | pytorch nerf implicit nets

### Description Implement a simplified version of a Neural Radiance Field (NeRF) to represent a 2D image. [1] A NeRF learns a continuous mapping from spatial coordinates to pixel values. Instead...

Implement Lottery Ticket Hypothesis Pruning

yesterday medium (<30 mins) | pytorch pruning lottery ticket

### Description The Lottery Ticket Hypothesis suggests that a randomly initialized, dense network contains a smaller subnetwork (a "winning ticket") that, when trained in isolation, can match the...

Simple Differentiable Renderer

yesterday medium (<30 mins) | pytorch autograd diff rendering 3d

### Description Modern 3D deep learning often relies on differentiable rendering, allowing gradients to flow from a 2D rendered image back to 3D scene parameters. [1] Your task is to implement a...

Implement a Custom Loss Function

this month medium (<30 mins) | pytorch loss functions regression implementation

Create a custom loss function called `MeanAbsolutePercentageError` (MAPE) in PyTorch. It should: 1. Take predictions and targets as input tensors. 2. Compute $$\frac{1}{n} \sum_i \frac{|y_i -...

Custom Dataset for CSV Data

this month medium (<1 hr) | pytorch dataset dataloader tabular data

Write a PyTorch `Dataset` class that loads data from a CSV file containing tabular data (features + labels). Requirements: - Use `pandas` to read the CSV. - Convert features and labels to tensors....

Gradient Clipping Example

this month medium (<30 mins) | pytorch rnn training gradients

Write code to: 1. Train a small RNN on dummy data. 2. Add gradient clipping using `torch.nn.utils.clip_grad_norm_`. 3. Print gradient norms before and after clipping. Show that exploding gradients...

Implement Dropout Manually

this month medium (<1 hr) | pytorch implementation dropout regularization

Implement dropout as a function `my_dropout(x, p)`: - Zero out elements of `x` with probability `p`. - Scale survivors by $$1/(1-p)$$. - Ensure deterministic behavior when `torch.manual_seed` is...

Custom Activation Function

this month medium (<1 hr) | pytorch implementation activations mlp

Define a custom activation function called `Swish`: $$f(x) = x \cdot \sigma(x)$$. - Implement it as a PyTorch `nn.Module`. - Train a small MLP on random data with it. - Compare with ReLU...

Weight Initialization Techniques

this month medium (<30 mins) | pytorch deep learning training initialization

Initialize a neural network's weights using different schemes: - Xavier initialization. - Kaiming initialization. Show histograms of weight distributions before and after initialization.

Custom Collate Function

this month medium (<30 mins) | pytorch dataloader sequences collate

Write a custom `collate_fn` for `DataLoader` that pads variable-length sequences with zeros. Use `torch.nn.utils.rnn.pad_sequence`. Test by batching random-length tensors.

Visualize Training with TensorBoard

this month medium (<1 hr) | pytorch training tensorboard visualization

Integrate TensorBoard into a training loop: - Log training loss and validation accuracy. - Add histograms of weights and gradients. - Write a few sample images. Open TensorBoard and verify logs.

Gradient Accumulation Example

this month medium (<30 mins) | pytorch implementation training gradients

Simulate large-batch training using gradient accumulation: - Train with microbatches of size 4. - Accumulate gradients over 8 steps. - Update optimizer after accumulation. Verify final result...

Implement Early Stopping

this month medium (<1 hr) | pytorch training basics early stopping

Add early stopping to a training loop: - Monitor validation loss. - Stop training if no improvement after 5 epochs. - Save best model checkpoint. Demonstrate on MNIST subset.

Implement Label Smoothing

this month medium (<30 mins) | pytorch training label smoothing classification

Write a function to apply label smoothing for classification: - Replace one-hot targets with $$1-\epsilon$$ for true class, $$\epsilon/(K-1)$$ for others. - Use it in cross-entropy training. Show...

Save and Load TorchScript Model

this month medium (<30 mins) | pytorch torchscript model deployment serialization

Convert a trained PyTorch model to TorchScript via tracing and scripting. Save it to disk. Reload and run inference. Compare outputs with the original model.

Mixed Precision Training with autocast

this month medium (<30 mins) | pytorch training mixed precision gpu

Modify a training loop to use `torch.cuda.amp.autocast`: - Wrap forward + loss in `autocast`. - Use `GradScaler` for backward. Compare training speed vs. full precision.