Autoencoders
Here are detailed notes on autoencoders, their various types, and their pros and cons.
What is an Autoencoder?
An autoencoder is an unsupervised artificial neural network designed to learn efficient, compressed representations (codings) of data. It's "unsupervised" (or more accurately, "self-supervised") because it doesn't need labeled data. Its "label" is the input data itself.
The primary goal is to learn a compressed latent-space representation (an "encoding") of the input, and then use that representation to reconstruct the original input as closely as possible.
The Core Architecture
An autoencoder always consists of three main parts:
Encoder: This part of the network compresses (encodes) the input data into a lower-dimensional latent-space representation, .
Bottleneck (or Latent Space): This is the most compressed layer where the latent representation lives. It's the "bottleneck" that forces the network to learn only the most important features.
Decoder: This part of the network tries to reconstruct (decodes) the original input (pronounced "x-hat") from the compressed latent representation .
The network is trained by minimizing a reconstruction loss (like Mean Squared Error or Binary Cross-Entropy), which measures the difference between the original input and the reconstructed output .
Types of Autoencoders
While the basic structure is simple, different "types" of autoencoders use clever constraints to force the network to learn useful features, not just "memorize" the data.
1. Undercomplete Autoencoder
This is the simplest, most "classic" type.
How it Works: The only constraint is the architecture itself. The bottleneck layer is forced to have fewer neurons than the input layer. This physical bottleneck forces the network to learn a compressed representation.
Pros:
Simple to understand and implement.
Good for a first pass at non-linear dimensionality reduction (like PCA, but more powerful).
Cons:
Can be too simple: If the network is too powerful (too many layers or neurons), it might learn to "cheat" and just pass the data through without learning meaningful patterns.
Prone to overfitting if the latent dimension is not small enough.
When to Use:
Simple dimensionality reduction.
As a learning tool to understand the basic concept.
2. Sparse Autoencoder
This type forces sparsity in the activations, not in the architecture.
How it Works: The bottleneck layer can be larger than the input. The "compression" comes from a sparsity penalty added to the loss function. This penalty punishes the network for "activating" too many neurons in the hidden layer.
How the Penalty Works:
L1 Regularisation: Adds a penalty based on the absolute value of the activations. This encourages most activations to become exactly zero.
KL Divergence: A more formal method. It forces the average activation of each neuron (e.g., 0.05) to be close to a small "sparsity parameter" .
Pros:
Feature Disentanglement: By forcing only a few neurons to fire for any given input, the model learns to associate specific neurons with specific features.
Highly Interpretable: You can look at which neurons activate for which inputs, making it great for feature learning.
Cons:
More complex to train due to the extra penalty term in the loss function.
When to Use:
Feature extraction and interpretability.
When you suspect your data is composed of many independent features (e.g., identifying faces, where features are "nose," "eyes," "mouth").
3. Denoising Autoencoder
This type is forced to learn robust features by "cleaning" noisy data.
How it Works:
Take the original input and add random noise to it, creating a "corrupted" version, . (This noise can be Gaussian noise, or "masking" where random pixels are set to zero).
Feed the corrupted input to the encoder.
The decoder outputs a reconstruction .
Crucially: The loss is calculated between the reconstruction and the original, clean input .
This forces the model to ignore the noise and learn the underlying structure or manifold of the data.
Pros:
Extremely robust feature learning.
Excellent for data denoising (e.g., removing static from audio or "snow" from images).
Cons:
Requires careful tuning of the noise level (too much noise and it can't learn; too little and it's no different from a standard AE).
When to Use:
Image denoising and restoration.
Pre-training deep networks to learn robust features.
4. Contractive Autoencoder
This type forces the latent representation to be "insensitive" to tiny, irrelevant changes in the input.
How it Works: It adds a complex penalty to the loss function that is based on the Jacobian matrix of the encoder's activations.
Simple Analogy: Imagine two input images of a dog that are almost identical (e.g., different by just one or two pixels). This autoencoder is penalized if these two inputs result in wildly different latent representations (). It is "rewarded" (has a lower loss) if it "contracts" both inputs to almost the same point in the latent space.
Pros:
Learns representations that are robust to very small, unimportant variations.
Captures the "manifold" of the data very well.
Cons:
Very computationally expensive to calculate the Jacobian matrix at every step.
Harder to implement and tune than other types.
When to Use:
When you need to be sure your model is learning the true "essence" of the data and not just superficial noise (e.g., in medical imaging).
5. Variational Autoencoder (VAE)
This is the most advanced type. It's a generative model, meaning it can create new data.
How it Works:
Probabilistic Encoder: The encoder doesn't output a single point . It outputs the parameters of a probability distribution (a mean and a log-variance ).
Latent Space: The latent space is sampled from this distribution. This means even the same input will produce a slightly different each time.
Reparameterization Trick: To allow backpropagation (which can't flow through a "sampling" node), it uses a trick: , where is a random value from a standard normal distribution. This isolates the randomness.
Generative Decoder: The decoder learns to reconstruct the input from these sampled points .
The VAE Loss (Evidence Lower Bound or ELBO):
Reconstruction Loss: Same as other AEs (e.g., MSE).
KL Divergence: A second loss term that acts as a regularizer. It forces all the distributions (, ) to be close to a standard normal distribution (mean=0, variance=1). This packs the latent space tightly and continuously.
Pros:
Generative! Once trained, you can throw away the encoder, pick a random from the latent space, and the decoder will generate a brand new, plausible output (e.g., a new "face" that it's never seen).
The latent space is continuous and structured, allowing for "latent space arithmetic" (e.g., "smiling face" vector - "neutral face" vector = "smile" vector).
Cons:
Blurry Outputs: The "averaging" effect of the probabilistic sampling often leads to blurrier, less sharp image reconstructions compared to GANs (Generative Adversarial Networks).
Much more complex to understand and implement.
When to Use:
Generative tasks: Creating new images, music, or text.
Understanding complex data distributions.
Data augmentation.
Summary: Which Autoencoder to Use?
Autoencoder Type
Key Idea
Main Use Case
Undercomplete
Physical bottleneck
Simple non-linear dimensionality reduction.
Sparse
Penalty on activations (L1/KL)
Feature learning, interpretability, disentanglement.
Denoising
Reconstruct from noisy input
Image/audio denoising, robust feature extraction.
Contractive
Penalty on latent-space "sensitivity"
Learning robust, invariant features (e.g., medical).
Variational (VAE)
Probabilistic latent space
Generative tasks (creating new, similar data).
Last updated