Source: Forbes, Jun 2016
The thing that fascinates me about the brain is that it has hugely more parameters than it has training data. So it is very unlike the neural nets that are currently being very successful. What is happening at present is we have neural nets with millions of weights and we train them on millions of training examples and they do very well. Sometimes billions of weights and billions of examples. But we typically do not have hugely more parameters than training data, and that is not true with the brain.
The brain has about ten thousand parameters for every second of experience. We do not really have much experience about how systems like that work or how to make them be so good at finding structure in data.
Most people in AI, particularly the younger ones, now believe that if you want a system that has a lot of knowledge in, like an amount of knowledge that would take millions of bits to quantify, the only way to get a good system with all that knowledge in it is to make it learn it. You are not going to be able to put it in by hand.
Hinton: Yes. Many of us had the belief for many years that if we had powerful enough computation and if we had enough data, then our techniques would eventually work. We have reached the point where that is true now. Now, our techniques scale up: you make computation more powerful, we can make you better models; you give us bigger data sets, we can make you better models. That is not true if you program everything. So this stuff scales better than AI ever did in the past.
Hinton: No one person has to be expert in all those areas. It is too difficult. What you need is an expert in an area who also understands what the main project is so that you can ask them a question and get the answer and save yourself reading through the literature. If you have an expert in neuroscience, you can ask questions like “When there is a forward projection from one area to the next area, and then there is a backward projection, how many neurons are there in that loop? How many synapses does the information have to go through before it gets back to where it started?” That is the kind of thing that will take you a while reading the literature, and also you do not know which papers you can trust. If you have an expert, they can just tell you there and then. It is very useful having very good people who understand the project because you can save yourself a lot of time and they can tell you from their perspective what is silly and what is not.