TensorFlow 1 Nostalgia: Create a Graph and Run in a Session

TensorFlow 2 runs by default in Eager mode in which functions return value-tensors instead of op-tensors. To build an Autograph, put @tf.function annotation in the line right before 'def'. However, there's another way to use Ops as really ops, that is making op chain inside graph as_default(), and run with tf.compat.v1.Session, see the code below.

Source code:

%tensorflow_version 2.x
%reset -f

#libs
import tensorflow as tf;

#create new graph, inside default graph, TF functions return op-tensors.
#outside of default graph, TF functions return value-tensors, as Eager mode is default on TF2
G = tf.Graph();  
with G.as_default():

  #G is now the default graph
  print("Is default graph:",tf.compat.v1.get_default_graph() is G);

  #no operations
  print("Operations:",G.get_operations());

  #add some operations
  print("\nAdding operations...");
  Inp    = tf.compat.v1.placeholder(tf.float32, [2], name="Inp");
  Times2 = tf.multiply(Inp, 2, name="Mul");

  #now having 1 placeholder, 1 operation
  print("Operations:",G.get_operations());

#feed to graph
print("\nRun graph in session, result:");
S = tf.compat.v1.Session(graph=G);
R = S.run(Times2, feed_dict={Inp:[1,2]});
print(R);
#eof

Colab vs Paperspace vs Kaggle

Colab:

• Free: Yes
• IPython-based: Yes
• Code blocks saved as-is: No (ipynb JSON)
• TensorFlow 1: Yes
• TensorFlow 2: Yes
• Save to GDrive: Yes
• Save to GitHub: Yes
• Save to GitLab: No
• Comfortable UI: Yes
• GPU: Yes

Paperspace:

• Free: Yes
• IPython-based: Yes
• Code blocks saved as-is: No (ipynb JSON)
• TensorFlow 1: Yes
• TensorFlow 2: Yes
• Save to GDrive: No
• Save to GitHub: No
• Save to GitLab: No
• Comfortable UI: No (Too big top bar)
• GPU: Yes

Kaggle:

• Free: Yes
• IPython-based: Yes
• Code blocks saved as-is: No (ipynb JSON)
• TensorFlow 1: Yes
• TessorFlow 2: No (Can't even !pip install tensorflow==2.0.0)
• Save to GDrive: No
• Save to GitHub: No
• Save to GitLab: No
• Comfortable UI: No (No left side panel)
• GPU: Yes

TensorFlow: Save and Load to Continue Training (tf.Module instead of tf.keras.Model)

After saving a tf.Module with tf.saved_model.save, the model can be loaded by tf.saved_model.load, the model can be train more by applying gradients to:

• M.Some_Layer.trainable_variables
• M.Some_Var

The list Model.trainable_variables is no longer in the model after loading.

Source code:

%tensorflow_version 2.x
%reset -f

#libs
import tensorflow as tf;
from tensorflow.keras.layers import *;

#constants
BSIZE = 4;

#model
class model(tf.Module):
  def __init__(this):
    super().__init__();
    #this.W1 = tf.Variable(tf.random.uniform([2,20], -1,1));
    #this.B1 = tf.Variable(tf.random.uniform([  20], -1,1));
    this.Layer1 = Dense(20, activation=tf.nn.leaky_relu);

    this.W2 = tf.Variable(tf.random.uniform([20,1], -1,1));
    this.B2 = tf.Variable(tf.random.uniform([   1], -1,1));

  @tf.function(input_signature=[tf.TensorSpec([BSIZE,2], tf.float32)])
  def __call__(this,Inp):
    #H1  = tf.nn.leaky_relu(tf.matmul(Inp,this.W1) + this.B1);
    H1  = this.Layer1(Inp);
    Out = tf.sigmoid(tf.matmul(H1,this.W2) + this.B2);    
    return Out;

#data (OR)
X = tf.convert_to_tensor([[0,0],[0,1],[1,0],[1,1]], tf.float32);
Y = tf.convert_to_tensor([[0],  [1],  [1],  [1]  ], tf.float32);

#train
Model = model();
Loss  = tf.losses.MeanSquaredError();
Optim = tf.optimizers.SGD(1e-1);
Steps = 10;

for I in range(Steps):
  if I%(Steps/10)==0:
    Out = Model(X);
    Lv  = Loss(Y,Out);
    print("Loss:",Lv.numpy());

  with tf.GradientTape() as T:
    Out = Model(X);
    Lv  = Loss(Y,Out);

  Grads = T.gradient(Lv, Model.trainable_variables);
  Optim.apply_gradients(zip(Grads, Model.trainable_variables));

Out = Model(X);
Lv  = Loss(Y,Out);
print("Loss:",Lv.numpy(),"(Last)");

#save
print("\nSaving model...");
Dir = "/tmp/models/test";
tf.saved_model.save(Model,Dir);

#load
print("\nLoading model...");
M = tf.saved_model.load(Dir);
print(vars(M).keys());
print(tf.keras.backend.flatten(M(X)).numpy());

#train more
print("\nContinue training...");
Steps = 1000;

for I in range(Steps):
  if I%(Steps/10)==0:
    Out = M(X);
    Lv  = Loss(Y,Out);
    print("Loss:",Lv.numpy());

  with tf.GradientTape() as T:
    Out = M(X);
    Lv  = Loss(Y,Out);

  Grads = T.gradient(Lv, M.Layer1.trainable_variables+[M.W2,M.B2]);
  Optim.apply_gradients(zip(Grads, M.Layer1.trainable_variables+[M.W2,M.B2]));

Out = M(X);
Lv  = Loss(Y,Out);
print("Loss:",Lv.numpy(),"(Last)");
print(tf.keras.backend.flatten(M(X)).numpy());
#eof

Why Unsupervised Learning & Reinforcement Learning are More Important than Supervised Learning

Supervised Learning (SL):

• Requires some training data
• This learning method only knows what learnt

Unsupervised Learning (UL, check data using formula), and Reinforcement Learning (RL, check data using environment):

• Don't require data
• Generate data and test themselves --> Some creativity!

However, the optimal method in reality is combining SL with both UL & RL to make IL: Incremental Learning.

Nginx Redirect: Return and Rewrite

Nginx redirect all using location+return:

server {

  listen ...;

  server_name ...;

  

  location ~* ^/(.*)$ {

    return 307 https://some.domain/$1;

  }

}

Nginx redirect all using rewrite:

server {

  listen ...;

  server_name ...;

  rewrite ^/(.*)$ https://some.domain/$1 redirect;

}

Nginx as proxy (to port 9999 for example):

server {

  listen ...;

  server_name ...;

  location / {

    proxy_pass http://localhost:9999;

  }

}

TensorFlow: Load Model to Continue Training

TensorFlow low-level model based on tf.Module is easy to save just as Keras model but low-level model is hard to continue training after loading back as custom functions must be created to assign weight values. The following is an example to save/load Keras model to continue training.

Source code:

%tensorflow_version 2.x
%reset -f

#libs
import tensorflow as tf;
from tensorflow.keras.layers import *;

#constants
BSIZE = 4;

#model
class model(tf.keras.Model):
  def __init__(this):
    super().__init__();
    this.W1 = tf.Variable(tf.random.uniform([2,20], -1,1));
    this.B1 = tf.Variable(tf.random.uniform([  20], -1,1));

    this.W2 = tf.Variable(tf.random.uniform([20,1], -1,1));
    this.B2 = tf.Variable(tf.random.uniform([   1], -1,1));

  #@tf.function(input_signature=[tf.TensorSpec([BSIZE,2])])
  def call(this,Inp):
    H1  = tf.nn.leaky_relu(tf.matmul(Inp,this.W1) + this.B1);
    Out = tf.sigmoid(tf.matmul(H1,this.W2) + this.B2);
    return Out;

#data
X = tf.convert_to_tensor([[0,0],[0,1],[1,0],[1,1]], tf.float32);
Y = tf.convert_to_tensor([[0],  [1],  [1],  [0]  ], tf.float32);

#train
Model = model();

#hard to resume training with this low-level training procedure:
'''
Loss  = tf.losses.MeanSquaredError();
Optim = tf.optimizers.SGD(1e-1);
Steps = 100;

for I in range(Steps):
  if I%(Steps/10)==0:
    Out = Model(X);
    Lv  = Loss(Y,Out);
    print("Loss:",Lv.numpy());

  with tf.GradientTape() as T:
    Out = Model(X);
    Lv  = Loss(Y,Out);

  Grads = T.gradient(Lv, Model.trainable_variables);
  Optim.apply_gradients(zip(Grads, Model.trainable_variables));

Out = Model(X);
Lv  = Loss(Y,Out);
print("Loss:",Lv.numpy(),"(Last)");
'''

#easier to resume training with keras
Model.compile(loss=tf.losses.MeanSquaredError(), optimizer=tf.optimizers.SGD(1e-1));
Model.fit(X,Y, batch_size=4, epochs=10, verbose=0);
print("Test:");
print(Model.predict(X, batch_size=4, verbose=0));

#save
print("\nSaving model...");
tf.keras.models.save_model(Model,"/tmp/models/test");

#load
print("\nLoading model to train more...");
M = tf.keras.models.load_model("/tmp/models/test");
print(M.predict(X, batch_size=4, verbose=0));

#continue training
M.fit(X,Y, batch_size=4, epochs=5000, verbose=0);
print("\nTest:");
print(M.predict(X, batch_size=4, verbose=0));
#eof

3D Plotting with Numpy

Source code:

%tensorflow_version 2.x
%reset -f

#core
import math;

#libs
import numpy      as np;
import tensorflow as tf;
from tensorflow.keras.layers import *;

import matplotlib.pyplot as pp;
from mpl_toolkits import mplot3d;

#plot
pp.figure(figsize=[10,10]);
pp3d = pp.axes(projection="3d",elev=45,azim=20);
X    = np.linspace(-10,10, 50);
Y    = np.linspace(-10,10, 50);
X,Y  = np.meshgrid(X,Y);
Z    = np.sin(X/math.pi/2)*np.cos(Y/math.pi/2)*(X-Y);

#pp3d.plot_wireframe(X,Y,Z);
pp3d.plot_surface(X,Y,Z, cmap="coolwarm");
#eof

ReLU as a Function of Single Infinite Domain

In machine learning, ReLU = max(0,x); this leads to a function with 2 domains:

• f = 0 for x<0
• f = x for x>=0

However, ReLU can be expressed as a function in infinite domain:

ReLU = f(x) = [ abs(x) + x ] / 2

ML Activation Functions and Their Uses

Linear Activation (both negative and positive to infinity):

Tanh Activation (limited negative, limited positive):

Sigmoid Activation (no negative, limited positive):

ReLU (no negative, positive to infinity):

Softplus aka Smooth ReLU (rectifier, smooth value transition):

Basically, depends on what output values should be and an activation function is selected:

• Identity activation: No negative limit, no positive limit.
• Tanh: Limited negative, limited positive.
• Sigmoid: No negative, limited positive.
• ReLU: No negative, limited positive.
• Etc.

Any custom function can be used as activation function, for example, a function similar to Tanh:

f(x) = [ sign(x)*(abs(x) - ln(cosh(x))) ] / 0.7

Softsign is a sigmoid-like activation function that utilise sign(x) and abs(x) too:

softsign(x) = x / (abs(x)+1)

Python 3D Plotting with matplotlib

The following is a plot of a sample function f(x,y) = sin(x)*cos(y)*(x-y).

Source code:

%reset -f

#core
import math;

#libs
import numpy             as np;
import matplotlib.pyplot as pp;

from mpl_toolkits import mplot3d;

#function to plot
Pi = math.pi;
f  = lambda x,y: math.sin(x/Pi/2)*math.cos(y/Pi/2)*(x-y);

#make data
Len = 50;
X   = np.linspace(-10,10, Len);
Y   = np.linspace(-10,10, Len);
X,Y = np.meshgrid(X,Y);
Z   = [];

for I in range(Len):
  Z += [[]];
  for J in range(Len):
    Z[I] += [f(X[I][J],Y[I][J])]; #any function of x,y  

#plot
pp.figure(figsize=[10,10]);
pp3d = pp.axes(projection="3d", elev=50, azim=70);

pp3d.set_title("Function z = f(x,y)");
pp3d.set_xlabel("x values");
pp3d.set_ylabel("y values");
pp3d.set_zlabel("z values");
pp3d.plot_wireframe(X,Y,np.array(Z));
#pp3d.plot_surface(X,Y,np.array(Z), cmap="coolwarm");
#eof