¹²For example, let’s express LSTMs and Gated Recurrent Unit networks as networks built from sigmoid neurons, linear neurons, and neurons performing multiplication for gating following Appendix C of [9], and let’s assume that we are writing this in terms of neurons processing scalar streams (streams of numbers). Then each of the two inputs for each neuron performing multiplication of scalar streams is connected with weight 1 from a single output of an appropriate neuron.