Reinforcement learning for the birds
Much like human speech, bird song learning is social; perhaps we'll discover machine learning is social, too.
Baby cardinal (source: Mike Loukides)
I just read a fascinating article about an experiment in bird psychology. We’ve known for a long time that bird songs aren’t innate; they’re learned. If you listen carefully to your back yard birds in the spring, you can hear the young birds learning their songs; you’ll probably hear a few that can’t get it right, and that gradually get better as summer progresses.
We’ve also known that bird songs (as distinct from other bird calls) are strictly a male behavior: they’re part of the mating rituals. Getting its song right is an important step in a male bird’s education. Female birds don’t sing. They stay quiet and choose the mate whose song they like best. (For a fascinating discussion of birds, mating rituals, and aesthetics, see The Evolution of Beauty.)
Learn faster. Dig deeper. See farther.
The common sense understanding of how birds learn their songs has been that it’s simply imitation: the baby bird tries to sing like its father. But it’s not that simple. The mother plays a crucial role. She gives tiny signals to her children that show them whether they’re getting the song right. It’s basically a birdie “thumbs up”: fluffing her feathers or twitching a wing to show that she likes the song. We haven’t noticed because a bird’s mental processing is much faster than ours. We’re too slow to see the minute twitches that the mother birds use to signal to their offspring. They only became evident when scientists used high-speed cameras and slowed down the video.
So, my first thought (well, actually, third or fourth thought) was, how does this relate to machine learning? And I realized this is essentially reinforcement learning: the mother is rewarding her offspring as it progresses toward a better song. In this context, “better” presumably means “more attractive to female birds,” and the mother trains her male offspring to go forth and find mates. Get yourself good singing lessons, and you’re all set.
Neural networks are about imitation. You get your tagged training set, see how well the model can imitate the tags in the training set, and when it’s good enough, you try real-world data. That’s gotten us a long way, but it has limitations: it requires a lot of training data, and lengthy training through many iterations. Bird learning, like reinforcement learning, is all about rewards, and that’s fundamentally different. There’s relatively little training data: just a mother that gives cues about whether the child is getting it right. (And even as I write this, I think, just a mother? Even considered purely as data, a mother probably represents more data than the largest training sets imaginable.) Is that terribly surprising? Human babies don’t learn to speak on their own; they’re constantly getting feedback from their parents.
Now we know that’s how birds do it, too. And maybe that’s how our machines will do it. Learning is social; perhaps we’ll find out that, in the end, machine learning is also social.