Feedforward Neural Networks in Depth

For a more in depth explanation of Feedforward Neural Networks, one of our DLS mentors has written some articles (optional reading) on them. If these interest you, do check them out.

Clarification For: What does this have to do with the brain?

Note that the formulas shown in the next video have a few typos. Here is the correct set of formulas.

𝑑𝑍[𝐿]=𝐴[𝐿]−𝑌dZ[L]=A[L]−Y

𝑑𝑊[𝐿]=1𝑚𝑑𝑍[𝐿]𝐴[𝐿−1]𝑇dW[L]=m1dZ[L]A[L−1]T

𝑑𝑏[𝐿]=1𝑚𝑛𝑝.𝑠𝑢𝑚(𝑑𝑍[𝐿],𝑎𝑥𝑖𝑠=1,𝑘𝑒𝑒𝑝𝑑𝑖𝑚𝑠=𝑇𝑟𝑢𝑒)db[L]=m1np.sum(dZ[L],axis=1,keepdims=True)

𝑑𝑍[𝐿−1]=𝑊[𝐿]𝑇𝑑𝑍[𝐿]∗𝑔′𝐿−1dZ[L−1]=W[L]TdZ[L]∗g′L−1

Note that * denotes element-wise multiplication)

⋮

𝑑𝑍[1]=𝑊[2]𝑇𝑑𝑍[2]∗𝑔′1dZ[1]=W[2]TdZ[2]∗g′1

𝑑𝑊[1]=1𝑚𝑑𝑍[1]𝐴[0]𝑇dW[1]=m1dZ[1]A[0]T

Note that 𝐴[0]𝑇A[0]T is another way to denote the input features, which is also written as 𝑋𝑇XT

𝑑𝑏[1]=1𝑚𝑛𝑝.𝑠𝑢𝑚(𝑑𝑍[1],𝑎𝑥𝑖𝑠=1,𝑘𝑒𝑒𝑝𝑑𝑖𝑚𝑠=𝑇𝑟𝑢𝑒)db[1]=m1np.sum(dZ[1],axis=1,keepdims=True)

Deep L-layer Neural Network

⇒ Technically logistic regression is a one layer neural network, we could then, but over the last several years the AI, on the machine learning community, has realized that there are functions that very deep neural networks can learn that shallower models are often unable to.

Although for any given problem, it might be hard to predict in advance exactly how deep in your network you would want.

So it would be reasonable to try logistic regression, try one and then two hidden layers, and view the number of hidden layers as another hyper parameter that you could try a variety of values of, and evaluate on all that across validation data, or on your development set.