CRLF on Various Systems and Web Programming

Carriage Return (CR) and Line Feed (LF) are common character in computer programming. Different systems use them differently.

CRLF = 0D,0A = 13,10 = \r\n

New line on different systems:

• Windows: \r\n
• Mac: \r
• Linux: \n
• HTML textarea tag: \n
• expect: \r

Check JavaScript event Enter key

HTML ('event' is lower case to work in Firefox):

<input onkeyup="check_enter(event);"/>

JS code:

function check_enter(Event){

  if (Event.keyCode==13)

    console.log("Enter key pressed");

}

Minimal Yet Advanced Code Base for Webapps, Websites

Front-end:

• Use jQuery
• Use $.load and $.getScript to load static files
  $("#Some-Id").load("/static/file.html",()=>{
    //Load JS here to have HTML already ready
    $.getScript("/static/file.js");
  });
• Load screen static files based on the URL in address bar
• Load component static files in screen .js file

Back-end:

• Use Express.js to serve static files only, no other URL handling.
• Use Socket.io to serve JSON data in the  webapp style.
• APIs  run through Socket.io, not HTTP/s protocol.

A Non-sequential Model for Machine Text Translation

A typical sequential DNN has 1 input head and 1 output head. However, the common text translation model with encoder-decoder, or encoder-decoder with attention has another input at middle of the DNN, so it's still DNN but not sequential. The model is fed as following:

Encoder_Out = encoder(Xleftlang)
Decoder_Out = decoder(Encoder_Out,Xrightlang)

Minimal Yet Complete Dockerfile

1) Create a project directory

2) Create file 'build.sh' with commands to build Docker image

3) Create file 'start.sh' with commands to exec after image build

Note:

ravik694/c7-systemd-sshd is a good base Docker image to start from. It comes with systemd, sshd.

Dockerfile:

FROM ravik694/c7-systemd-sshd

COPY . /project

RUN  cd /project && bash build.sh

CMD  cd /project && bash start.sh

Run Docker without Sudo

Docker of old versions by default must run with 'sudo', do these to run as normal user (required that the user is in 'wheel' or 'sudo' group):

Optional: Install Docker if not yet
sudo yum install docker -y

Optional: Start Docker service
sudo systemctl start docker
sudo systemctl enable docker

1) Create 'docker' group
sudo groupadd docker

2) Add current sudoer user to the group
sudo usermod -aG docker $USER

3) Logout of terminal and login back

4) Restart docker service (Important!)
sudo systemctl restart docker

5) Test docker without sudo
docker image ls
docker container ls

Create and View HTML Files in Google Drive

HTML files can be created directly in Google Drive (https://drive.google.com) using these Google Drive apps:

• Text Editor for Google Drive (Text mode)
• HTML Editor for Google Drive (WYSIWYG)

Right-click on the HTML file, click 'Get Shareable Link', a link is copied to clipboard similar to this:

https://drive.google.com/open?id=XXX

However that link is View UI of Google Drive, change to the following to download:

https://drive.google.com/uc?id=XXX

However, HTML files shouldn't be downloaded, should be viewed in browser, use the following URL (Google Drive Toolkit), where XXX is the Id of the file.

http://gdfile.tk/XXX

Example URL to HTML file on Google Drive:
http://gdfile.tk/1JBRTL3d7mcmdUqqLs1bEORedRbvJRXpx

Create HTML Files in Google Drive and View in Browser

HTML files can be created in Google Drive using these Google Drive apps:

• Text Editor for Google Drive
• HTML Editor for Google Drive

When getting link for the HTML file from Google Drive by clicking 'Get Shareable Link', the URL is:

https://drive.google.com/open?id=XXX

The above link is to View UI of Google Drive. Change it to get the direct link to the file:

https://drive.google.com/uc?id=XXX

Images work okay! but for HTML files, Google purposely deny users from viewing in browser by forcing a download, Google servers set a response header:

Content-Disposition: attachment;

Now, how to view the HTML content in browser anyway? Nginx proxy_pass doesn't work, Google blocks it. Nginx rewrite & return doesn't work as there are multiple HTTP 302 redirects and can't remove the above header.

Solution: 

Create a web server just to load the HTML files and send to browser!

Source Code:

//Libs

import express from "express";

import request from "request";

//Make a lock

function new_lock(){

  var Unlock,Lock=new Promise((Resolve,Reject)=>{ Unlock=Resolve; });

  return [Lock,Unlock];

}

//PROGRAMME ENTRY POINT=========================================================

var Server = express();

Server.get("/gd/*",async (Req,Res)=>{

  var Url     = Req.url;

  var Id      = Url.replace("/gd/","");

  var Gd_Url  = `https://drive.google.com/uc?id=${Id}`;

  //make request to Google Drive

  var Content       = "no-contents";

  var [Lock,Unlock] = new_lock();

  request(Gd_Url,(Err,Gdres,Body)=>{

    Content = Body;

    Unlock();

  });

  await Lock;

  Res.send(Content);

});

Server.listen(8080);

//EOF

How to Get Direct View/Download Link for a Google Drive File

Google Drive let users store files online. When click 'Get Shareable Link' on context menu, an URL is copied to clipboard for pasting, it is always:

https://drive.google.com/open?id=XXX

This link opens the View UI of Google Drive for the file, however, sometimes we need the direct link to images to show on web page, or direct link to trigger download immediately, here it is, just change "/open" to "/uc" where 'uc' means 'user content':

https://drive.google.com/uc?id=XXX

Note that the 'uc' link always trigger download or show in web pages if it is UC of images, as Google doesn't want to have Google Drive turned into web hosting. So, if the UC link is a link to .html file, instead of showing the web contents, it forces you to download with this HTTP response header:

Content-Disposition: attachment;

Human Vision (Object Detection) and YOLO

How we humans detect and recognise object with our eyes:

1. Location: Turn our eyes to some direction (left, right, top, bottom, centre, or some random point)
2. Boundary: Detect the boundary
3. Classification: Any object? classify it.

YOLO (You Only Look Once) network does somehow similarly:

1. Location: Grid up an image (or video frame) to NxN, for example 3x3, that is also left, right, top, bottom, centre, etc.
2. Boundary: From a point in a grid cell, predict a rectangle, that is boundary
3. Classification: Any object? classify it.

Links:

YOLO: https://pjreddie.com/darknet/yolo

OpenCV: https://opencv.org

Lambda Expression and Lambda Function in Python, JavaScript (also Node.js)

Lambda expression and lambda function in programming are no-name (anonymous). Both Python and JavaScript (also Node.js) have this feature.

Example utility function:
f = x^2 + x + 1

Example lambda expression in Python:
F = lambda X: f(X);

Example lambda function in Python, note the commas and no 'return':
F = lambda X: (
  statement1,
  statement2,
  Result
);

Example lambda expression in JavaScript:
F = X => f(X);

Example lambda function in JavaScript, note the semicolons and 'return':
F = X => {
  statement1;
  statement2; 
  return Result;
};

Generate a function with constant C in Python:
F = (lambda T: lambda X: f(X)+C)(C);

Generate a function with constant C in JavaScript:
F = (C => X => f(X)+C)(C);

Generate Functions with Different Constants in Python, JS

Source Code (Python):

#Any function

def f(X):

return X**2;

#Constants for generating functions

Ts = [1,2,3];

#Generate functions

Fs = [(lambda X,T=T: f(X)+T) for T in Ts];

#Call generated functions with default constants

print(Fs[0](3));

print(Fs[1](3));

print(Fs[2](3));

#Call generated functions with specific constants

print(Fs[0](3,1));

print(Fs[1](3,2));

print(Fs[2](3,3));

#EOF

Result:
10
11
12
10
11
12

Source Code (JavaScript, Node.js):

//Shortcuts

log = console.log;

//Any function

function f(X){

return X*X;

}

//Constants to generate functions

Ts = [1,2,3];

Fs = [];

//Function generator loop

for (I in Ts) { //Get index list

..T = Ts[I];

..F = (T => { //A function instance

....return function(X){

......return f(X)+T;

....}

..})(T);

..Fs.push(F);

}

//Test generated functions

log(Fs[0](3));

log(Fs[1](3));

log(Fs[2](3));

//EOF

Result:
The same

Domain Levels and Subdomains

Global top level  domains (TLDs):

• Traditional: .com, .org, .net, .info, .biz
• New TLDs: .shop, .site, .game, .fun, etc.

Country top level domains (Country TLDs):

• .uk, .it, .de, .fr, .vn, etc.

Country second level domains (Country SLDs):

• .me.uk, .co.uk, .org.uk, .co.kr, .com.vn, etc.

Note that although second level domains consist of 2 components but they are still domains, not subdomains.

Subdomains are additional components after TLDs, country TLDs, or country SLDs, for example:

• Subdomains without dots:
• abc.xyz.com
• abc.xyz.uk
• abc.xyz.co.uk
• Subdomains with dots:
• foo.abc.xyz.com
• foo.abc.xyz.uk
• foo.abc.xyz.co.uk

Incremental Learning and the Four Factors

Traditional ML comes with Supervised Learning, Unsupervised Learning, and Reinforcement Learning, but Incremental Learning with the follow factors would wrap it all:

• Factor 1: Any training data?
• Factor 2: Generate training data?
• Factor 3: Any formula for checking results?
• Factor 4: Any environment for checking results?

Ways to Run Processes in Terminal That Won't be Terminated

Commonly, processes run in terminal will be terminated when logging out of terminal. There are multiple ways to keep them running after closing terminal.

1) forever
https://www.npmjs.com/package/forever

2) pm2
https://pm2.keymetrics.io/docs/usage/quick-start/

3) nohup
Regular command: mycmd --arg value
Nohup command: nohup mycmd --arg value

4) setsid
Create mycmd.sh as:
#!/bin/bash
mycmd --arg value

Run:
setsid ./mycmd.sh

5) daemonize
Similar to setsid, create a shell file to run mycmd.

6) disown
Raw and powerful to separate command from terminal.
mycmd --arg value & disown $!

Auto-encoder vs Encoder-Decoder

Auto-encoder:

• Input and Output are the same
• Used for making embedding layer

Encoder-Decoder:

• Input and Output are different sequences
• Used for text translation

Keras Accuracy Functions Test

Source Code:

%tensorflow_version 2.x

%reset -f

#Libs

import tensorflow as tf;

#PROGRAMME ENTRY POINT==========================================================

a  = tf.metrics.Accuracy();

ba = tf.metrics.BinaryAccuracy();

ca = tf.metrics.CategoricalAccuracy();

#Y is expected, U is output

#Accuracy (use with Heaviside or integer outputs)

Y = tf.constant([[0],[1],[1],[0  ]], tf.float32);

U = tf.constant([[0],[1],[1],[0.1]], tf.float32);

print("Accuracy:",a(Y,U)); #Perfectly equal only, 0.1 is NOT 0

#BinaryAccuracy (use with single Sigmoid output)

Y = tf.constant([[0],[1],[1],[0  ]], tf.float32);

U = tf.constant([[0],[1],[1],[0.1]], tf.float32); 

print("BinaryAccuracy:",ba(Y,U)); #Threshold 0.5, 0.1 is still 0

#CategoricalAccuracy (use with multi-prob Softmax output)

Y = tf.constant([[0,1,0],[0, 0, 1  ]], tf.float32);

U = tf.constant([[0,1,0],[0, 0, 0.1]], tf.float32); 

#Considered as:           0, 0, 1 

#2 zeros prefix 0.1, so 0.1 is still max prob and considered as 1, means correct

print("CategoricalAccuracy:",ca(Y,U));

Y = tf.constant([[0,1,0],[0, 0,   1  ]], tf.float32);

U = tf.constant([[0,1,0],[0, 0.3, 0.1]], tf.float32); 

#Considered as:           0, 1,   0

#0.2 is max prob and considered as 1, so the second sample is partially wrong

#0.1 is considered as 0 coz 0.1 < 0.3/2

print("CategoricalAccuracy:",ca(Y,U));

#EOF

Result:

Accuracy: tf.Tensor(0.75, shape=(), dtype=float32)
BinaryAccuracy: tf.Tensor(1.0, shape=(), dtype=float32)
CategoricalAccuracy: tf.Tensor(1.0, shape=(), dtype=float32)
CategoricalAccuracy: tf.Tensor(0.75, shape=(), dtype=float32)

Useful Apps for Google Drive

Google Drive has all built-in Google apps for drive, however, some useful apps are missing. The following are those useful apps:

• Draw.io - Draw diagrams
• Text Editor for Google Drive - Text file editor
• HTML Editor for Google Drive - HTML file editor

Right-click in Google Drive, then choose:

• More
• Connect more apps

Cheating in Chrome Dinosaur Game

Open the game:
chrome://dino

Press F12, click Console, past and press Enter this line:
Runner.prototype.gameOver = function(){}

Press Space to run and never game over!

Available Loss Functions and Optimisers in TensorFlow 2

TensorFlow 2 has a pack of loss functions:

• Basic and popular:
• MeanAbsoluteError: The most basic loss function
• MeanSquaredError: The faster and popular loss function
• Smoothed MAE:
• LogCosh
• Entropy loss functions:
• BinaryCrossentropy
• CategoricalCrossentropy
• SparseCategoricalCrossentropy
• KLDivergence
• Related to SVM:
• Hinge
• SquaredHinge
• CategoricalHinge
• Others:
• CosineSimilarity
• MeanAbsolutePercentageError
• MeanSquaredLogarithmicError
• Poisson
• Huber

TensorFlow 2 also has a pack of optimisers (in tf.optimizers.*) to use to optimise the variables (weights and biases):

• Basic:
• SGD The most basic optimiser, Stochastic Gradient Descent.
• Momentum:
• RMSprop Similar to momentum optimiser
• Adaptives:
• Adam The fastest optimiser with super-convergence, Adaptive Momentum.
• Adamax Adam with infinity norm
• Adagrad Adaptive gradient
• Adadelta Adaptive delta
• Others
• Ftrl Follow-the-regularised-leader

Adam (Adaptive Momentum) is the fastest one to optimise variables, super-convergence, practically tested.

See also: https://www.fast.ai/2018/07/02/adam-weight-decay/

DNN Gradient Descent Backpropagation with Mean Squared Error

Gradient descent is applied in the backpropagation (dynamic programming) to get gradients of loss with respect to weights. Mean Squared Error (L2) is commonly used instead of Mean Absolute Error (L1).

MSE = mean[ sum[ (Out-Y)² ] ]

Where Out-Y is delta, and mean squared error is also called mean squared loss; not the loss is squared, deltas are squared and summed up.

The algorithm:

• Notation:
• H: Hidden layer output
• U: Output layer output
• u: Single node output (at output layer)
• B: Backpropagation intermediate value
• f: Activation function
• fd: Activation function derivative
• X: Input to network
• Y: Labels or expected results
• y: Single node expected result
• R: Learning rate
• Dot without arguments is dot in feedforward
• Feedforward:
• First hidden layer:
• H = f(dot(X,W))
• Other hidden layers:
• H = f(dot(H_prev,W))
• Output layer:
• U = f(dot(H_prev,W))
• Backpropagate:
• Output layer:
• B = 2*(u-y)/N * fd(dot)
• Other  hidden layers:
• B = dot(B_right,W) * fd(dot)
• Update weights (Optimise after having all B(s))
• G = B*H_prev
• W -= R*G

How to Add a Programme to Start-up in Windows 10

Add a programme to Start-up in Windows 10:

• Press: Window+R
• Type: shell:startup
• A folder is opened, right-click and add a shortcut to the programme.

Feedforward and Backpropagation in DNN

x1, x2, x3 = Input values
D          = Dot product of inputs and weights (forward)
H          = Output value of a neuron
d1, d2     = Differences compared to expected values = abs(out-y)
L          = Summarised loss = d1+d2
G          = Gradient of loss function
D2         = Dot product of gradients and weights (backward)
activate   = Activation function
dActivate  = Derivative of activation function
B          = Backpropagation intermediate value at a neuron

Gradient of loss with respect to W:
Gw = B * Input_to_the_Weight

Apply gradient:
W -= Learning_Rate * Gw

First layer outputs:  H1, H2, H3 (from top down)
Second layer outputs: H4, H5 (from top down)

Gradient for last layer
Last layer is not the same, for hidden layer: Gw is like above.
For the last layer (let L = Mean squared error):
Gw = dL/dw = dMSE/dw = dMSE/dAct * dAct/dDot * dDot/dw

3D Programming in Web Browser

Modern browsers support WebGL and WebAssembly. These new technologies enable 3D embeds in HTML5 canvas.

3D applications, games in browser canvas:

• Solution 1:
• Programming language: JavaScript
• Graphics technology: WebGL
• Draw into canvas
• Solution 2:
• Programming language: C++ (compile to WebAssembly)
• Graphics technology: OpenGL
• Draw into canvas

History of Types in C/C++

16-bit Machines (Max 32-bit):
Note: short and int are the same, float is 16-bit

• char        8-bit
• short       16-bit
• int         16-bit
• long        32-bit (Max)
• long long   No such type
• float       16-bit
• double      32-bit (Max)
• long double No such type

32-bit Machines (Max 64-bit):
Note: int and long are the same, float is 32-bit

• char        8-bit
• short       16-bit
• int         32-bit
• long        32-bit
• long long   64-bit (Max)
• float       32-bit
• double      64-bit (Max)
• long double No such type

64-bit Machines (Max 128 bit):
Note: int is 32-bit, long is 64-bit, long double is added

• char        8-bit
• short       16-bit
• int         32-bit
• long        64-bit
• long long   128-bit (Max)
• float       32-bit
• double      64-bit
• long double 128-bit (Max)

How to Use Google Colab to Run C++ Code

Create a notebook on Colab, and create these 2 code blocks to run C++ code right inside Colab; it is interesting as Colab is not just for Python.

Code block 1 (C++ source code):

%%writefile test.cpp

//Main function

int main(int Argc,char* Args[]){

  return 0;

}

Code block 2 (Compile and run):

%%script bash

g++ test.cpp -o test

ls -laX

./test