Non-sequential DNN with Calculation Flow Combination in the Middle of Network

A typical DNN usually comes with single flow from single batch input to output called sequential DNN. When there are more than 1 batch input and having calculation flow combination in the middle of the DNN, it is still DNN but not sequential DNN as it's not a chain of functions on results of immediate previous functions.

Sequential example:
U = layer3(layer2(layer1(X))

Non-sequential example:
H1a = layer1a(X1)
H1b = layer1b(X2)
U   = layer3(layer2(H1a,H1b))

This kind of multi-input (multiple X(s), there are already multiple small x(s) in each big X) is commonly used in text translation model where:

• Translation done as: 
• A left language sentence --> A right language sentence
• The first input is a sentence of left language
• For example: i love you
• The second input is a sequence preceding the sentence in right language
• For example: 
• NULL ti   amo  --> ti   amo  NULL
• ti   amo  NULL --> amo  NULL NULL
• amo  NULL NULL --> NULL NULL NULL
• The output is a sentence of right language
• For example: ti amo

The following source code doesn't demonstrate the translation model, but it shows how to make calculation flow combination in the middle of the network using tf.concat. Note that there are many other methods for calculation flow combination in the middle of network.

Source Code:

%tensorflow_version 2.x

%reset -f

#Libs

import tensorflow              as     tf;

from   tensorflow.keras.layers import *;

#Constants

DSIZE  = 4;

BSIZE  = 4;

EPOCHS = 1000;

LAFTE  = 100;

#Model

class model(tf.keras.Model):

  def __init__(this):

    super(model,this).__init__();

    this.Dense1a = Dense(units=1, activation=tf.nn.leaky_relu);

    this.Dense1b = Dense(units=1, activation=tf.nn.leaky_relu);

    this.Dense2  = Dense(units=1, activation=tf.sigmoid);

  @tf.function

  def call(this,X):

    Firstcall = X[0].shape[0]!=None;

    this.X  = X;

    this.X1 = X[0];

    this.X2 = X[1];

    if Firstcall: print(f"X1  shape: {this.X1.shape}");

    if Firstcall: print(f"X2  shape: {this.X2.shape}");

    this.H1a = this.Dense1a(this.X1);

    if Firstcall: print(f"H1a shape: {this.H1a.shape}");

    this.H1b = this.Dense1b(this.X2);

    if Firstcall: print(f"H1b shape: {this.H1b.shape}");

    this.H2 = tf.concat([this.H1a,this.H1b],1);

    if Firstcall: print(f"H2  shape: {this.H2.shape}");

    this.U = this.Dense2(this.H2);

    if Firstcall: print(f"U   shape: {this.U.shape}");

    return this.U;

#Keras callback

class callback(tf.keras.callbacks.Callback):

  def on_epoch_end(this,Epoch,Logs):

    L,A = Logs["loss"],Logs["accuracy"];

    if Epoch%LAFTE==LAFTE-1:

      print(f"Average batch loss: {L:.9f}, accuracy: {A:.9f}");

    if Epoch==EPOCHS-1:

      print(f"Finavg  batch loss: {L:.9f}, accuracy: {A:.9f}");

#Custom accuracy

@tf.function

def accuracy(Ys,Us):

  Max_Loss = 1*BSIZE;

  return (Max_Loss - tf.reduce_sum(tf.abs(Us-Ys))) / Max_Loss;

#PROGRAMME ENTRY POINT==========================================================

#Data (XOR)

Inp1 = tf.constant([[0,0],[0,1],[1,0],[1,1]], tf.float32);

Inp2 = tf.constant([[0,0],[0,1],[1,0],[1,1]], tf.float32);

Exp  = tf.constant([[0  ],[1  ],[1  ],[0  ]], tf.float32);

#Train

Model = model();

Model.compile(loss=tf.losses.MeanSquaredError(), optimizer=tf.optimizers.Adam(1e-1),

              #metrics=[tf.metrics.BinaryAccuracy(name="accuracy")]);

              metrics=[accuracy]);

L,A = Model.evaluate(x=[Inp1,Inp2],y=Exp, batch_size=BSIZE, verbose=0);

print(f"\nInitavg batch loss: {L:.9f}, accuracy: {A:.9f}");

Model.fit(x=[Inp1,Inp2],y=Exp, epochs=EPOCHS,shuffle=True,batch_size=BSIZE,

          callbacks=[callback()], verbose=0);

#EOF

Result:

X1  shape: (4, 2)
X2  shape: (4, 2)
H1a shape: (4, 1)
H1b shape: (4, 1)
H2  shape: (4, 2)
U   shape: (4, 1)

Initavg batch loss: 0.240871489, accuracy: 0.524801254
Average batch loss: 0.000486979, accuracy: 0.979479671
Average batch loss: 0.000210377, accuracy: 0.986456275
Average batch loss: 0.000119435, accuracy: 0.989769578
Average batch loss: 0.000077473, accuracy: 0.991768181
Average batch loss: 0.000054577, accuracy: 0.993095338
Average batch loss: 0.000040651, accuracy: 0.994043112
Average batch loss: 0.000031508, accuracy: 0.994757116
Average batch loss: 0.000025154, accuracy: 0.995317698
Average batch loss: 0.000020553, accuracy: 0.995768666
Average batch loss: 0.000017106, accuracy: 0.996140778
Finavg  batch loss: 0.000017106, accuracy: 0.996140778