mcbosch

SGD-Scratch-Project

This project is aimed to build a workable python class to work with a squence of LinearLayers and activation functions and be able to go forward and train backgward with batches. It is aimed to have a scalable python class to build Variational Autoencoders (VAE).

In this documentation we'll go through the repository structure, and the methods of the python classes we bulid. If you find any mistake, don't hesitate in contribuiting with the projects.

Structure

We have 3 main python classes: LinearLayers, Sequence, and NeuralNetworks. The two first python-objects are the important ones; the third is a Sequence with a training function.

LinearLayer

A LinearLayer is a mathematical function that goes from an euclidean space of dimension n_in to an euclidean space of dimension n_out. It's the composition of a linear transformation with a non-linear activation function.

l = LinearLayer(n_cels_in = 2,
                n_cels_out = 4,
                bias = True,
                activation="ReLU")
print(l)
>>> 2 -- Fully Conected --> 4 -->  ReLU

There are three options of activation functions: ReLU, Sigmoid, and Softmax. The Linear Layer has three methods:

Linear Layer Methods
method arguments description
l.forward()
  • input
 Makes a forward run of la layer and stores the value of the neurons
l.backpropagation()
  • d: error to backpropagate
  • first_delta_compute: bool that indicates if the error to backpropagate is computed from the activated neurons or non-activated neurons.
 It backpropagates the error and stores the error of the neurons in l.cache.
l.updateparameters()
  • learning_rate: step of grad descendent
  • beta_1: parameter of the first momentum ADAM
  • beta_2: second parameter of the second momentum ADAM
  • m0: first momentum ADAM
  • m1: second momentum ADAM
  • t: counter of updates
  • adam: bool to indicate if we use ADAM to update
 Updates the parameters using grad descendent with ADAM (if activated) using the error of the neurons in l.cache.

N.B. in a future update I'll made a more visual table with css and not html table.

Sequence

A Sequence is a bunch of ordered layers connected. It's simply a list of layers with some methods. We can see an example on how to build a sequence in the following code:

l1 = LinearLayer(2,4,activation="ReLU")
l2 = LinearLayer(4,6,activation="ReLU")
l3 = LinearLayer(6,2,activation="ReLU")
seq = Sequence([l1, l2, l3])

print(len(seq))
>>> 3

print(seq[1])
>>> 4 -- Fully Connected --> 6 --> ReLU

Sequence([LinearLayer(2,2),LinearLayer(3,2)])
>>>ValueError: Incompatible layer dimensions

We can understand a Sequence as a Layer, so it has the same methods but runing over each layer in order.

Sequence Methods
method parameters description
seq.forward()
  • input
 Runs the input over all the layers in order.
seq.add()
  • layer
Adds a layer at the final of the sequence.
seq.backpropagate()
  • delta: error to backpropagate
  • step: learning rate of grad. descend
  • beta_1: parameter for first ADAM momentum
  • beta_2: parameter for second ADAM momentum
  • momentums: list of tuples of momentums for each layer.
  • t: counter of updates for ADAM bias correction
  • update_parameters: bool, if true updates parameters
  • adam: bool, if true updates parameters using adam
  • first_delta_computed: bool, indicates if the error to backpropagate is in the non-activated neurons (True value) or not
 Backpropagates the error and stores the error of the neurons in the cache of each layer. If update parameters is set as true, it updates the parameters.

The Sequence object is useful to work with concatenation of layers for any purpouse. It can also work as a NN, but it doesn't have methods to work with data and train/test. I built it this way to have a more general class for any purpose, and build an other NN class.

NeuralNetwork

A NeuralNetwork is a sequence of layers provided with training and test methods; and other methods to make work the training and test methods. We can see in the following code an example of how to use it:

N = NeuralNetwork([LinearLayer(2,4),
                  LinearLayer(4,2, activation="Softmax")])

print(N)
>>> Neural NeuralNetwork
----------------
Number of Hidden Layers: 1
Pass of Information:
    2 -- Fully Connected --> 4 --> ReLU
    4 -- Fully Connected --> 2 --> Softmax
----------------
NeuralNetwork Methods
method parameters description
N.forward()
  • input
 Runs over all the layers in order
N.train()
  • data: dataset where we want to train the model.
  • epochsNumber of epochs to train over the dataset.
  • learning_rate: step of gradient descendent.
  • batch_size: size of the batches.
  • beta_1: ADAM parameter for first momentum
  • beta_2: ADAM parameter for second momentum
  • loss: loss function which is going to be optimizied.
  • adam: bool
  • data_val: dataset of validation.
 It trains a model using SGD with batches of the size indicated, and with the indicated loss. Also, it uses a data_val if there is given.
N.test()
  • data: data for the test
  • loss: function to use to compute the loss.
 It computes the loss and acc of the dataset test.
to_one_hot
    labels
Given a batch of labels it return a batch of vectors with zeros and a one in the label positon.

Future Improvements

This python class is workable but could be improved in some ways. One that brought me problems is not having a parameter class to be able to automatically count how many times the parameter has been updated for the ADAM correction bias. Also, the code could be more clean grouping some variables.

I may improve this class, but for now my objective was to learn in having a good repository, a clean documentation, and make a workable class to make a VAE from scratch.