Case Study: Regression/Classification Solutions

These are popular ways to do ML regression/classification:

• Regress to value
• Classify to class probabilities
• Classify to class indices

Classifying to class indices is simpler:

• No transformation needed to convert classes to probabilities in Y training data
• The output layer is simpler: Only 1 neuron (sigmoid-like activation, NOT unit-step activation)

Classifying to class probabilities is a little more complex:

• Need to transform training data Y to probabilities of classes
• Output layer has number of neurons equal number of classes
• The good point in regressing to class probabilities is when fed data don't actually match a class 100%, and other classes are possible results too.

Source code:

#libs
import tensorflow        as tf;
import matplotlib.pyplot as pyplot;

#data
MAX        = 11;
Batch_Size = 5;

#solution1: regress with probabilities
#X = [[0,0],   [0,1],   [1,0],   [10,10], [MAX,MAX]];
#Y = [[1,0,0], [1,0,0], [1,0,0], [0,1,0], [0,0,1]  ]; #will regress to these probabilities
#Y_Shape = 3;

#solution2: regress with class indices
X = [[0,0],   [0,1],   [1,0],   [10,10], [MAX,MAX]];
Y = [[0],     [0],     [0],     [1],     [2]      ]; #will regress to these classes
Y_Shape = 1;

#normalise
for I in range(len(X)):
  X[I][0] = X[I][0]/MAX;
  X[I][1] = X[I][1]/MAX;
#end for

#model
Input     = tf.placeholder(dtype=tf.float32, shape=[Batch_Size,2]);
Expected  = tf.placeholder(dtype=tf.float32, shape=[Batch_Size,Y_Shape]); #3 classes

Weight1   = tf.Variable(tf.random_uniform(shape=[2,20], minval=-1, maxval=1));
Bias1     = tf.Variable(tf.random_uniform(shape=[  20], minval=-1, maxval=1));
Hidden1   = tf.nn.relu(tf.matmul(Input,Weight1) + Bias1);

Weight2   = tf.Variable(tf.random_uniform(shape=[20,10], minval=-1, maxval=1));
Bias2     = tf.Variable(tf.random_uniform(shape=[   10], minval=-1, maxval=1));
Hidden2   = tf.nn.relu(tf.matmul(Hidden1,Weight2) + Bias2);

#solution1: use sigmoid for probabilities
#Weight3   = tf.Variable(tf.random_uniform(shape=[10,3], minval=-1, maxval=1));
#Bias3     = tf.Variable(tf.random_uniform(shape=[   3], minval=-1, maxval=1));
#Output    = tf.sigmoid(tf.matmul(Hidden2,Weight3) + Bias3);

#solution2: use sigmoid for class indices
Weight3   = tf.Variable(tf.random_uniform(shape=[10,1], minval=-1, maxval=1));
Bias3     = tf.Variable(tf.random_uniform(shape=[   1], minval=-1, maxval=1));
Output    = tf.multiply(tf.sigmoid(tf.matmul(Hidden2,Weight3) + Bias3), 2);

Loss      = tf.reduce_sum(tf.square(Expected-Output));
Optimiser = tf.train.GradientDescentOptimizer(1e-1);
Training  = Optimiser.minimize(Loss);

#train
Sess = tf.Session();
Init = tf.global_variables_initializer();
Sess.run(Init);

Losses = [];
for I in range(9000):
  if (I%900==0):
    Lossvalue = Sess.run(Loss, feed_dict={Input:X, Expected:Y});
    Losses += [Lossvalue];
    print("Loss:",Lossvalue);
  #end if
  
  Sess.run(Training, feed_dict={Input:X, Expected:Y});
#end for

#result: loss
Lastloss = Sess.run(Loss, feed_dict={Input:X, Expected:Y});
Losses  += [Lastloss];
print("Loss:",Lastloss,"(Last)");

#result: eval
Evalresult = Sess.run(Output, feed_dict={Input:X, Expected:Y});
for I in range(Batch_Size):
  Evalresult[I][0] = round(Evalresult[I][0]);  
  
  if (Y_Shape==3):
    Evalresult[I][1] = round(Evalresult[I][1]);
    Evalresult[I][2] = round(Evalresult[I][2]);
  #end if
#end for
print("Eval:\n"+str(Evalresult));

#result: diagram
print("Loss curve:");
pyplot.plot(Losses);
#eof

Colab link:
https://colab.research.google.com/drive/1tSlhmqGdNad2wIa7K2tPdlnkK4_YWGgu