All Katas - ML Katas

Build a Simple Neural Network with Flax

this week medium (<30 mins) | mlp jax flax neural-network

Using Flax, JAX's neural network library, build a simple Multi-Layer Perceptron (MLP). The MLP should have an input layer, one hidden layer with a ReLU activation function, and an output layer....

Implement a Basic Optimizer with Optax

this week medium (<30 mins) | optimization jax optax sgd

Use Optax, JAX's optimization library, to create a simple Stochastic Gradient Descent (SGD) optimizer. You'll need to define a model, a loss function, and then use `optax.sgd` to update the...

Implement Layer Normalization

this week medium (<30 mins) | jax deep-learning layer-normalization

Implement Layer Normalization from scratch in JAX. Layer Normalization is a key component in many modern neural network architectures, like Transformers. It normalizes the inputs across the...

Implementing Dropout

this week medium (<30 mins) | dropout regularization jax deep-learning

Implement the dropout regularization technique in JAX. This involves randomly setting a fraction of input units to 0 at each update during training time. Remember that dropout should only be...

Differentiable Additive Synthesizer

this week medium (<30 mins) | pytorch generative ddsp audio

### Description Differentiable Digital Signal Processing (DDSP) is a technique that combines classic signal processing with deep learning by making the parameters of synthesizers learnable via...

Implement a Knowledge Distillation Loss

this week medium (<30 mins) | pytorch compression distillation

### Description Knowledge Distillation is a model compression technique where a small "student" model is trained to mimic a larger, pre-trained "teacher" model. [1] This is achieved by training...

Masked Autoencoder (MAE) Input Preprocessing

this week medium (<30 mins) | pytorch self-supervised mae vision transformer

### Description Masked Autoencoders (MAE) are a powerful self-supervised learning technique for vision transformers. The core idea is simple: randomly mask a large portion of the input image...

Deep Canonical Correlation Analysis (DCCA) Loss

this week medium (<30 mins) | pytorch dcca correlation multimodal

### Description Canonical Correlation Analysis (CCA) is a statistical method for finding correlations between two sets of variables. Deep CCA (DCCA) uses neural networks to first project two...

Gradient Reversal Layer

this week medium (<30 mins) | pytorch autograd gan domain adaptation

### Description Implement a Gradient Reversal Layer (GRL), a key component in Domain-Adversarial Neural Networks (DANNs). [1] The GRL acts as an identity function during the forward pass but...

Tiny Neural Radiance Fields (NeRF)

this week medium (<30 mins) | pytorch nerf implicit nets

### Description Implement a simplified version of a Neural Radiance Field (NeRF) to represent a 2D image. [1] A NeRF learns a continuous mapping from spatial coordinates to pixel values. Instead...

Implement Lottery Ticket Hypothesis Pruning

this week medium (<30 mins) | pytorch pruning lottery ticket

### Description The Lottery Ticket Hypothesis suggests that a randomly initialized, dense network contains a smaller subnetwork (a "winning ticket") that, when trained in isolation, can match the...

Simple Differentiable Renderer

this week medium (<30 mins) | pytorch autograd diff rendering 3d

### Description Modern 3D deep learning often relies on differentiable rendering, allowing gradients to flow from a 2D rendered image back to 3D scene parameters. [1] Your task is to implement a...