⇒ In a neural network that you train with supervised learning, the training set contains values of the inputs x as well as the target outputs y. So the term hidden layer refers to the fact that in the training set, the true values for these nodes in the middle are not observed. That is, you don't see what they should be in the training set. You see what the inputs are. You see what the output should be. But the things in the hidden layer are not seen in the training set. So that kind of explains the name hidden layer.
⇒ What you see is that it is like logistic regression, but repeated a lot of times.
Now, we've said before that logistic regression, the circle in logistic regression, really represents two steps of computation rows. You compute z as follows, and a second, you compute the activation as a sigmoid function of z. So, a neural network just does this a lot more times.