From the illustration above, an autoencoder consists of two components: (1) an encoder which learns the data representation, i.e. the important features zof the data, and (2) a decoder which reconstructs the data based on its idea zof how it is structured.

Going back, we established that an autoencoder wants to find the function that maps x to x. It does so through its components. Mathematically,

The encoder h-sub-e learns the data representation z from the input features x, then the said representation serves as the input to the decoder h-sub-d in order to reconstruct the original data x.

We shall further dissect this model below.

Encoder
The first component, the encoder, is similar to a conventional feed-forward network. However, it is not tasked on predicting values or labels. Instead, it is tasked to learn how the data is structured, i.e. data representation z.

The encoding is done by passing data input x to the encoder’s hidden layer h in order to learn the data representation z = f(h(x)). We can implement the Encoder layer as follows,

We first define an Encoderclass that inherits the tf.keras.layers.Layer to define it as a layer instead of a model. Why a layer instead of a model? Recall that the encoder is a component of the autoencoder model.

Going through the code, the Encoder layer is defined to have a single hidden layer of neurons (self.hidden_layer) to learn the activation of the input features. Then, we connect the hidden layer to a layer (self.output_layer) that encodes the data representation to a lower dimension, which consists of what it thinks as important features. Hence, the “output” of the Encoder layer is the learned data representation z for the input data x.

Decoder
The second component, the decoder, is also similar to a feed-forward network. However, instead of reducing data to a lower dimension, it reconstructs the data from its lower dimension representation z to its original dimension x.

The decoding is done by passing the lower dimension representation z to the decoder’s hidden layer h in order to reconstruct the data to its original dimension x = f(h(z)). We can implement the decoder layer as follows,

We define a Decoderclass that also inherits the tf.keras.layers.Layer.

The Decoder layer is also defined to have a single hidden layer of neurons to reconstruct the input features from the learned representation by the encoder. Then, we connect its hidden layer to a layer that decodes the data representation from a lower dimension to its original dimension. Hence, the “output” of the decoder layer is the reconstructed data x from the data representation z. Ultimately, the output of the decoder is the autoencoder’s output.

Now that we have defined the components of our autoencoder, we can finally build the model.

Building the Autoencoder model
We can now build the autoencoder model by instantiating the Encoder and the Decoder layers.

As we discussed above, we use the output of the encoder layer as the input to the decoder layer. So, that’s it? No, not exactly.

To this point, we have only discussed the components of an autoencoder and how to build it, but we have not yet talked about how it actually learns. All we know to this point is the flow of data; from the input layer to the encoder layer which learns the data representation, and use that representation as input to the decoder layer that reconstructs the original data.

Like other neural networks, an autoencoder learns through backpropagation. However, instead of comparing the values or labels of the model, we compare the reconstructed data x-hat and the original data x. Let’s call this comparison the reconstruction error function, and it is given by the following equation,

In TensorFlow, the above equation could be expressed as follows,

Are we there yet? Close enough. Just a few more things to add. Now that we have our error function defined, we can finally write the training function for our model.

This way of implementing backpropagation affords us with more freedom by enabling us to keep track of the gradients, and the application of an optimization algorithm to them.

Are we done now? Let’s see.

• Define an encoder layer. Checked.
• Define a decoder layer. Checked.
• Build the autoencoder using the encoder and decoder layers. Checked.
• Define the reconstruction error function. Checked.
• Define the training function. Checked.

Yes! We’re done here! We can finally train our model!

But before doing so, let’s instantiate an Autoencoder class that we defined before, and an optimization algorithm to use. Then, let’s load the data we want to reconstruct. For this post, let’s use the unforgettable MNIST handwritten digit dataset.

We can visualize our training results by using TensorBoard, and to do so, we need to define a summary file writer for the results by using tf.summary.create_file_writer.

Next, we use the defined summary file writer, and record the training summaries using tf.summary.record_if.

We can finally (for real now) train our model by feeding it with mini-batches of data, and compute its loss and gradients per iteration through our previously-defined train function, which accepts the defined error function, the autoencoder model, the optimization algorithm, and the mini-batch of data.

After each iteration of training the model, the computed reconstruction error should be decreasing to see if it the model is actually learning (just like in other neural networks). Lastly, to record the training summaries in TensorBoard, we use the tf.summary.scalar for recording the reconstruction error values, and the tf.summary.image for recording the mini-batch of the original data and reconstructed data.

After some epochs, we can start to see a relatively good reconstruction of the MNIST images.