One interesting thing to think about is how neural network learning models interact with discrete learning models. Why? Because discrete learning models are well studied and easy to intuit about. So perhaps by making these analogies, we can discover the missing pieces of NNs.
Lets start with basic decision trees, and a basic random forest model. A very high level overview of these with a comparison with neural nets can be found here.
Now, in decision tree world, we randomly walk through a series of sequential spaces