ML in TensorFlow: Single Neuron Linear Activation Regression

The most basic model in machine learning is a model of:

• Single neuron 
• With linear (identity) activation, 
• That does regression.

The training process loop:

• 1. Generate a batch (should not generate them all before training loop as the steps may be real high like hundreds of thousands and it takes all the RAM)
• 2. Feed the batch of samples to the model
• 3. Show average loss after certain number of steps

The steps 1 and 2 above affect the total performance, so it is suggested to use all the features of TensorFlow together for the best, ie. use tf.data.Dataset together with the model decorated by @tf.function.

Source code:

%tensorflow_version 2.x

%reset -f

#libs

import tensorflow as tf;

#constants

DSIZE = 4; #num samples in the whole dataset

BSIZE = 2; #num samples in a batch

#model

class model(tf.Module):

  #constructor

  def __init__(this):

    this.W1 = tf.Variable(tf.random.uniform([2,1], -1,1, tf.float32));

    this.B1 = tf.Variable(tf.random.uniform([  1], -1,1, tf.float32));

  #model call

  @tf.function(input_signature=[tf.TensorSpec([BSIZE,2], tf.float32)])

  def __call__(this,Inp):

    Out = tf.identity(tf.matmul(Inp,this.W1) + this.B1);

    return Out;

#PROGRAMME ENTRY POINT==========================================================

#data

X = tf.constant([[1,2],[3,4],[5,6],[7,8]], tf.float32);

Y = tf.constant([[3  ],[5  ],[7  ],[9  ]], tf.float32);

Data = tf.data.Dataset.from_tensor_slices((X,Y));

Data = Data.repeat().shuffle(DSIZE).batch(BSIZE,drop_remainder=True);

Iter = iter(Data);

#train

Model = model();

Loss  = tf.losses.MeanSquaredError();

Optim = tf.optimizers.SGD(1e-3);

Steps = 1000; #number of batches to process

Laft  = 100;  #log loss after every this num batches

Lsum  = 0;

for I in range(Steps):

  Batch = next(Iter);

  Inp   = Batch[0];

  Exp   = Batch[1];

  with tf.GradientTape() as T:

    Lval  = Loss(Exp,Model(Inp));

    Lsum += Lval.numpy();

  Grads = T.gradient(Lval, Model.trainable_variables);

  Optim.apply_gradients(zip(Grads, Model.trainable_variables));

  if I%Laft==Laft-1:

    print("Average Loss:",Lsum/Laft);

    Lsum = 0;

#eof

Result:
Average Loss: 2.1378963772580026 Average Loss: 0.21530249016243033 Average Loss: 0.19278635787957682 Average Loss: 0.17028992957601305 Average Loss: 0.15324589672891306 Average Loss: 0.13415179751318645 Average Loss: 0.12121034623822197 Average Loss: 0.10616741587153228 Average Loss: 0.09501745967809257 Average Loss: 0.08556641670929821