The Least Squares Generative Adversarial Network, or LSGAN for short, is an extension to the GAN architecture that addresses the problem of vanishing gradients and loss saturation.
It is motivated by the desire to provide a signal to the generator about fake samples that are far from the discriminator model’s decision boundary for classifying them as real or fake.
The further the generated images are from the decision boundary, the larger the error signal provided to the generator, encouraging the generation of more realistic images.
The LSGAN can be implemented with a minor change to the output layer of the discriminator layer and the adoption of the least squares, or L2, loss function.
In this tutorial, you will discover how to develop a least squares generative adversarial network.
After completing this tutorial, you will know:Discover how to develop DCGANs, conditional GANs, Pix2Pix, CycleGANs, and more with Keras in my new GANs book, with 29 step-by-step tutorials and full source code.
Let’s get started.
How to Develop a Least Squares Generative Adversarial Network (LSGAN) for Image GenerationPhoto by alyssa BLACK.
, some rights reserved.
This tutorial is divided into three parts; they are:The standard Generative Adversarial Network, or GAN for short, is an effective architecture for training an unsupervised generative model.
The architecture involves training a discriminator model to tell the difference between real (from the dataset) and fake (generated) images, and using the discriminator, in turn, to train the generator model.
The generator is updated in such a way that it is encouraged to generate images that are more likely to fool the discriminator.
The discriminator is a binary classifier and is trained using binary cross-entropy loss function.
A limitation of this loss function is that it is primarily concerned with whether the predictions are correct or not, and less so with how correct or incorrect they might be.
… when we use the fake samples to update the generator by making the discriminator believe they are from real data, it will cause almost no error because they are on the correct side, i.
e.
, the real data side, of the decision boundary— Least Squares Generative Adversarial Networks, 2016.
This can be conceptualized in two dimensions as a line or decision boundary separating dots that represent real and fake images.
The discriminator is responsible for devising the decision boundary to best separate real and fake images and the generator is responsible for creating new points that look like real points, confusing the discriminator.
The choice of cross-entropy loss means that points generated far from the boundary are right or wrong, but provide very little gradient information to the generator on how to generate better images.
This small gradient for generated images far from the decision boundary is referred to as a vanishing gradient problem or a loss saturation.
The loss function is unable to give a strong signal as to how to best update the model.
The Least Squares Generative Adversarial Network, or LSGAN for short, is an extension to the GAN architecture proposed by Xudong Mao, et al.
in their 2016 paper titled “Least Squares Generative Adversarial Networks.
” The LSGAN is a modification to the GAN architecture that changes the loss function for the discriminator from binary cross entropy to a least squares loss.
The motivation for this change is that the least squares loss will penalize generated images based on their distance from the decision boundary.
This will provide a strong gradient signal for generated images that are very different or far from the existing data and address the problem of saturated loss.
… minimizing the objective function of regular GAN suffers from vanishing gradients, which makes it hard to update the generator.
LSGANs can relieve this problem because LSGANs penalize samples based on their distances to the decision boundary, which generates more gradients to update the generator.
— Least Squares Generative Adversarial Networks, 2016.
This can be conceptualized with a plot, below, taken from the paper, that shows on the left the sigmoid decision boundary (blue) and generated fake points far from the decision boundary (pink), and on the right the least squares decision boundary (red) and the points far from the boundary (pink) given a gradient that moves them closer to the boundary.
Plot of the Sigmoid Decision Boundary vs.
the Least Squared Decision Boundary for Updating the Generator.
Taken from: Least Squares Generative Adversarial Networks.
In addition to avoiding loss saturation, the LSGAN also results in a more stable training process and the generation of higher quality and larger images than the traditional deep convolutional GAN.
First, LSGANs are able to generate higher quality images than regular GANs.
Second, LSGANs perform more stable during the learning process.
— Least Squares Generative Adversarial Networks, 2016.
The LSGAN can be implemented by using the target values of 1.
0 for real and 0.
0 for fake images and optimizing the model using the mean squared error (MSE) loss function, e.
g.
L2 loss.
The output layer of the discriminator model must be a linear activation function.
The authors propose a generator and discriminator model architecture, inspired by the VGG model architecture, and use interleaving upsampling and normal convolutional layers in the generator model, seen on the left in the image below.
Summary of the Generator (left) and Discriminator (right) Model Architectures used in LSGAN Experiments.
Taken from: Least Squares Generative Adversarial Networks.
Take my free 7-day email crash course now (with sample code).
Click to sign-up and also get a free PDF Ebook version of the course.
Download Your FREE Mini-CourseIn this section, we will develop an LSGAN for the MNIST handwritten digit dataset.
The first step is to define the models.
Both the discriminator and the generator will be based on the Deep Convolutional GAN, or DCGAN, architecture.
This involves the use of Convolution-BatchNorm-Activation layer blocks with the use of 2×2 stride for downsampling and transpose convolutional layers for upsampling.
LeakyReLU activation layers are used in the discriminator and ReLU activation layers are used in the generator.
The discriminator expects grayscale input images with the shape 28×28, the shape of images in the MNIST dataset, and the output layer is a single node with a linear activation function.
The model is optimized using the mean squared error (MSE) loss function as per the LSGAN.
The define_discriminator() function below defines the discriminator model.
The generator model takes a point in latent space as input and outputs a grayscale image with the shape 28×28 pixels, where pixel values are in the range [-1,1] via the tanh activation function on the output layer.
The define_generator() function below defines the generator model.
This model is not compiled as it is not trained in a standalone manner.
The generator model is updated via the discriminator model.
This is achieved by creating a composite model that stacks the generator on top of the discriminator so that error signals can flow back through the discriminator to the generator.
The weights of the discriminator are marked as not trainable when used in this composite model.
Updates via the composite model involve using the generator to create new images by providing random points in the latent space as input.
The generated images are passed to the discriminator, which will classify them as real or fake.
The weights are updated as though the generated images are real (e.
g.
target of 1.
0), allowing the generator to be updated toward generating more realistic images.
The define_gan() function defines and compiles the composite model for updating the generator model via the discriminator, again optimized via mean squared error as per the LSGAN.
Next, we can define a function to load the MNIST handwritten digit dataset and scale the pixel values to the range [-1,1] to match the images output by the generator model.
Only the training part of the MNIST dataset is used, which contains 60,000 centered grayscale images of digits zero through nine.
We can then define a function to retrieve a batch of randomly selected images from the training dataset.
The real images are returned with corresponding target values for the discriminator model, e.
g.
y=1.
0, to indicate they are real.
Next, we can develop the corresponding functions for the generator.
First, a function for generating random points in the latent space to use as input for generating images via the generator model.
Next, a function that will use the generator model to generate a batch of fake images for updating the discriminator model, along with the target value (y=0) to indicate the images are fake.
We need to use the generator periodically during training to generate images that we can subjectively inspect and use as the basis for choosing a final generator model.
The summarize_performance() function below can be called during training to generate and save a plot of images and save the generator model.
Images are plotted using a reverse grayscale color map to make the digits black on a white background.
We are also interested in the behavior of loss during training.
As such, we can record loss in lists across each training iteration, then create and save a line plot of the learning dynamics of the models.
Creating and saving the plot of learning curves is implemented in the plot_history() function.
Finally, we can define the main training loop via the train() function.
The function takes the defined models and dataset as arguments and parameterizes the number of training epochs and batch size as default function arguments.
Each training loop involves first generating a half-batch of real and fake samples and using them to create one batch worth of weight updates to the discriminator.
Next, the generator is updated via the composite model, providing the real (y=1) target as the expected output for the model.
The loss is reported each training iteration, and the model performance is summarized in terms of a plot of generated images at the end of every epoch.
The plot of learning curves is created and saved at the end of the run.
Tying all of this together, the complete code example of training an LSGAN on the MNIST handwritten digit dataset is listed below.
Note: the example can be run on the CPU, although it may take a while and running on GPU hardware is recommended.
Running the example will report the loss of the discriminator on real (d1) and fake (d2) examples and the loss of the generator via the discriminator on generated examples presented as real (g).
These scores are printed at the end of each training run and are expected to remain small values throughout the training process.
Values of zero for an extended period may indicate a failure mode and the training process should be restarted.
Note: your specific results will vary given the stochastic nature of the learning algorithm.
Plots of generated images are created at the end of every epoch.
The generated images at the beginning of the run are rough.
Example of 100 LSGAN Generated Handwritten Digits after 1 Training EpochAfter a handful of training epochs, the generated images begin to look crisp and realistic.
Remember: more training epochs may or may not correspond to a generator that outputs higher quality images.
Review the generated plots and choose a final model with the best quality images.
Example of 100 LSGAN Generated Handwritten Digits After 20 Training EpochsAt the end of the training run, a plot of learning curves is created for the discriminator and generator.
In this case, we can see that training remains somewhat stable throughout the run, with some very large peaks observed, which wash out the scale of the plot.
Plot of Learning Curves for the Generator and Discriminator in the LSGAN During Training.
We can use the saved generator model to create new images on demand.
This can be achieved by first selecting a final model based on image quality, then loading it and providing new points from the latent space as input in order to generate new plausible images from the domain.
In this case, we will use the model saved after 20 epochs, or 18,740 (60K/64 or 937 batches per epoch * 20 epochs) training iterations.
Running the example generates a plot of 10×10, or 100, new and plausible handwritten digits.
Plot of 100 LSGAN Generated Plausible Handwritten DigitsThis section provides more resources on the topic if you are looking to go deeper.
In this tutorial, you discovered how to develop a least squares generative adversarial network.
Specifically, you learned:Do you have any questions?.Ask your questions in the comments below and I will do my best to answer.
Develop Your GAN Models in Minutes .
with just a few lines of python codeDiscover how in my new Ebook: Generative Adversarial Networks with PythonIt provides self-study tutorials and end-to-end projects on: DCGAN, conditional GANs, image translation, Pix2Pix, CycleGAN and much more.
Finally Bring GAN Models to your Vision Projects Skip the Academics.
Just Results.
Click to learn more.. More details