Neuroevolution: A different kind of deep learning

Neuroevolution is making a comeback. Prominent artificial intelligence labs and researchers are experimenting with it, a string of new successes have bolstered enthusiasm, and new opportunities for impact in deep learning are emerging. Maybe you haven’t heard of neuroevolution in the midst of all the excitement over deep learning, but it’s been lurking just below the surface, the subject of study for a small, enthusiastic research community for decades. And it’s starting to gain more attention as people recognize its potential.

Put simply, neuroevolution is a subfield within artificial intelligence (AI) and machine learning (ML) that consists of trying to trigger an evolutionary process similar to the one that produced our brains, except inside a computer. In other words, neuroevolution seeks to develop the means of evolving neural networks through evolutionary algorithms.

When I first waded into AI research in the late 1990s, the idea that brains could be evolved inside computers resonated with my sense of adventure. At that time, it was an unusual, even obscure field, but I felt a deep curiosity and affinity. The result has been 20 years of my life thinking about this subject, and a slew of algorithms developed with outstanding colleagues over the years, such as NEAT, HyperNEAT, and novelty search. In this article, I hope to convey some of the excitement of neuroevolution as well as provide insight into its issues, but without the opaque technical jargon of scientific articles. I have also taken, in part, an autobiographical perspective, reflecting my own deep involvement within the field. I hope my story provides a window for a wider audience into the quest to evolve brains within computers.

The success of deep learning

If you’ve been following AI or ML recently, you’ve probably heard about deep learning. Thanks to deep learning, computers can accomplish tasks like recognizing images and controlling autonomous vehicles (or even video game characters) at close to or sometimes surpassing human performance. These achievements have helped deep learning and AI in general to emerge from the obscurity of academic journals into the popular press and news media, inspiring the public imagination. So, what is actually behind deep learning that has enabled its success?

In fact, underneath the hood in deep learning is the latest form of a decades-old technology called artificial neural networks (ANNs). Like many ideas in AI, ANNs are roughly inspired by biology; in this case, by the structure of the brain. We choose the brain as an inspiration for AI because the brain is the unequivocal seat of intelligence; while we’re pursuing AI, it makes sense that, at some level, it should resemble the brain. And one of the key building blocks of brains is the neuron, a tiny cell that sends signals to other neurons over connections. When many neurons are connected to each other in a network (as happens in brains), we call that a neural network. So, an ANN is an attempt to simulate a collection of neuron-like components that send signals to each other. That’s the underlying mechanism behind the “deep networks” in deep learning.

Researchers in ANNs write a program that simulates these neurons and the signals that travel between them, yielding a process vaguely reminiscent of what happens in brains. Of course, there are also many differences. The challenge is that simply connecting a bunch of neuron-like elements to each other and letting them share signals does not yield intelligence. Intelligence, instead, arises from precisely how the neurons are connected.

For example, a neuron that strongly influences another neuron is said to connect to its partner with a strong weight. In this way, the weights of connections determine how neurons influence each other, yielding a pattern of neural activation across a neural network in response to inputs to the network (which could be, for example, from the eyes). To get an intelligent network, the consequent challenge is to decide what these connection weights should be.

In general, almost no one tries to decide connection weights by hand. (With millions of connections in modern ANNs, you can imagine why that is not a viable approach.) Instead, the problem of finding the right weights to perform a task is viewed as the problem of learning. In other words, researchers have invested a lot of effort in devising methods to allow ANNs to learn the best weights for a task on their own. The most common approach to learning weights is to compare the output of an ANN (e.g., “That looks like a dog”) to the ground truth (it’s actually a cat) and then shift the weights through a principled mathematical formula to make the correct output more likely in the future.

After numerous such training examples (maybe millions), the network starts to assign the right weights to answer all kinds of questions correctly. Often, it can even generalize to questions it has never seen, as long as they are not too different from those it saw in training. At this point, the ANN has basically learned to perform the desired task. A common approach to such weight shifting is called stochastic gradient descent, which is the aforementioned formula popular throughout deep learning. The realization in deep learning in recent years is that it’s possible to train massive ANNs with many layers of neurons (which is why they’re called “deep”) through this approach thanks to the availability of powerful modern computer hardware.

But there’s an issue we haven’t yet addressed: how do you determine what connects to what in the first place? In other words, the behavior of your brain is not only determined by the weights of its connections, but by the overall architecture of the brain itself. Stochastic gradient descent does not even attempt to address this question. Rather, it simply does its best with the connections it is provided.

So, where did these connections come from? In deep learning, the answer is that they generally come from a human researcher who decides, based on some level of experience, what the architecture should be. In contrast, the answer for natural brains is evolution. The 100-trillion-connection architecture of our human brain evolved through a Darwinian process over many millions of years.

The architecture of our brains is exceedingly powerful. After all, human intelligence is literally seated there, which means, in effect, that the evolution of brains in nature is the only example of any known process in the universe actually producing something strongly intelligent. The goal of neuroevolution is to trigger a similar evolutionary process inside a computer. In this way, neuroevolution is the only branch of AI with an actual proof of concept: brains did evolve, so we know that’s one way to produce intelligence.

To be clear, deep learning traditionally focuses on programming an ANN to learn, while the concern in neuroevolution focuses on the origin of the architecture of the brain itself, which may encompass what is connected to what, the weights of those connections, and (sometimes) how those connections are allowed to change. There is, of course, some overlap between the two fields—an ANN still needs connection weights suited to its task, whether evolved or not, and it’s possible that evolved ANNs might leverage the methods used in deep learning (for instance, stochastic gradient descent) to obtain those weights. In fact, deep learning might even be viewed as a sibling of neuroevolution that studies how weights are learned within either an evolved or preconceived architecture.

However, it’s also conceivable that the mechanism of learning itself could be evolved, potentially transcending or elaborating the conventional techniques of deep learning as well. In short, the brain—including its architecture and how it learns—is a product of natural evolution, and neuroevolution can probe all the factors that contribute to its emergence, or borrow some from deep learning and let evolution determine the rest.

How it works

Now that we have a sense of what neuroevolution is about, we can talk about how it works. The first neuroevolution algorithms appeared in the 1980s. At the time, its small group of practitioners thought it might be an alternative to the more conventional ANN training algorithm called backpropagation (a form of stochastic gradient descent). In these early systems, neuroevolution researchers would (as in deep learning today) decide on the neural architecture themselves—which neurons connect to which—and simply allow evolution to decide the weights instead of using stochastic gradient descent. Because the architecture could not be changed by evolution, this approach came to be known as fixed-topology neuroevolution.

These systems are a bit different from nature in that the genes of the evolving ANNs in fixed-topology neuroevolution literally encode their weights, which are frozen from birth. In this way, the ANNs are “born” knowing everything they will ever know and cannot learn anything further during their “lifetime.” This scenario may be a little confusing because we generally consider learning something we do during our lifetime, but if you think about it, the breeding happening in these systems in effect is the learning. That is, when parents produce children better adapted to a task, a kind of “learning” over generations is happening.

So, how would a system like that really be set up? How do you evolve an artificial brain to solve a problem? In fact, it’s a lot like animal breeding. Suppose you want to evolve a neural network to control a robot to walk. In this kind of task, we would typically have on hand a simulator because neuroevolution takes a lot of trials, which are much faster and less risky to run in simulation. So, we’ll start with a robot body in a physics simulator.

Now we need some ANNs to get everything started. At the beginning, we don’t know how to solve the task, so we just generate a population (say 100) of random ANNs. In the case of fixed-topology ANNs, the weights of the predetermined architecture would be randomized in each of the 100 individuals in the population. Now, we just need to perform selection, which means breeding the better candidates to produce offspring.

To evaluate our current population, we first take an ANN from the population and, in effect, hand over to it control of the simulated robot’s body. We let the ANN tell the body how to move, which is called the output of the network. The ANN might also receive receive input from the body, such as a sense of when each foot hits the ground. Then the computer just watches and sees what the ANN does when it’s in control. Every ANN in the population is tested in this way and given a score called its “fitness,” based on the quality of its performance.

It’s pretty clear that the randomly generated networks in the initial population are unlikely to do well. They’re more likely to flail around than anything else. (After all, their brains are random.) But that’s okay, because the key is not that one ANN is particularly good, but rather that some are better than others, even if just by a little bit. Perhaps one network manages to get the robot to limp a little farther than another. The engine that drives improvement in neuroevolution is that those who are slightly better will be selected as parents of the next generation. The algorithm will construct their offspring by slightly altering their ANNs (such as by slightly changing the weights of their connections). While some offspring are worse than their parents, some will function slightly better (say by flailing a bit less), and those will then become parents of the next generation, and so on. In this way, the overall approach is to keep selecting increasingly fit individuals as parents. In effect, the process of neuroevolution is a kind of automated breeding farm for ANNs, where the computer selects the parents to breed based on their fitness.

The core idea of neuroevolution, then, is simple: it’s essentially just breeding. But beyond that, things get a lot more interesting. Over the decades since the first fixed-topology neuroevolution algorithms began to appear, researchers have continually run into the frustrating reality that even as the algorithms create new possibilities, the brains they can evolve remain far from what evolved in nature. There are many reasons for this gap, but a fascinating aspect of the field is that every so often a surprising new insight into the workings of natural evolution emerges, resulting in a leap in the capability of neuroevolution algorithms. Often, these insights are counter-intuitive, overturning previous assumptions and highlighting the mysteriousness of nature. As we gradually chip away at these mysteries, we discover how to fashion increasingly powerful algorithms for evolving brains.

Increasing complexity

What does it mean to make progress in neuroevolution? In general, it involves recognizing a limitation on the complexity of the ANNs that can evolve and then introducing an approach to overcoming that limitation. For example, the fixed-topology algorithms of the ’80s and ’90s exhibit one glaring limitation that clearly diverges from nature: the ANNs they evolve can never become larger. In contrast, brains in nature have increased in size and complexity in many lineages, and our distant ancestors had orders of magnitude fewer neurons in their cranium than the 100 billion neurons (and 100 trillion connections) in ours. Clearly, if the topology of the ANN is fixed from the start, no such complexity can ever evolve.

Consequently, researchers began experimenting with topology and weight evolving ANNs (TWEANNs). In this more flexible variant, the architecture (topology) of a parent ANN can be slightly changed in its offspring, such as by adding a new connection or a new neuron. While the idea is relatively simple, the implications are significant because it means that the brain can evolve to become larger. A variety of TWEANNs began to appear in the 90s, though the problems they tackled remained relatively simple, such as learning to give the right answer in simple mathematical and control problems. But the excitement of this era was not necessarily for the problems themselves but rather the vast untapped potential of artificially evolving the architecture and weights of a brain-like structure. The limitations of such a system were then unknown, and anything seemed possible.

To give you a sense of the problems these ANNs (both fixed-topology and TWEANNs) were being evolved to solve during this era, one popular benchmark problem was called pole balancing. The problem is to control a simulated cart that can only move in two directions with a pole affixed to its top by a hinge. The longer the cart (controlled by an ANN) can keep the pole in the air, the higher its fitness. This problem is a lot like trying to balance a pencil on the palm of your hand—it requires tight coordination and quick reactions. As learning methods improved, ANNs could solve increasingly difficult versions of pole balancing, such as balancing two poles at once. The succession of such tasks served to mark progress in the field, and researchers could promote their algorithms as the first to solve one variant or another.

If you’re interested in the technical details of the algorithms of this era, Xin Yao published an excellent 1999 survey of many of its neuroevolution algorithms. Much of the pioneering work of this period came from Stefano Nolfi and Dario Floreano, who captured many of its best ideas in their 2000 classic book, Evolutionary Robotics. Another book that helped to popularize the idea of evolving neural networks around the turn of the century is David Fogel’s classic, Blondie24: Playing at the Edge of AI (from Morgan Kaufmann Publishers), which told the story of a neuroevolution algorithm that learns to play master-level checkers through games with real humans.

In 1999, I started to think seriously about neuroevolution as an early Ph.D. student at the University of Texas at Austin in the research group of Professor Risto Miikkulainen, where several important fixed-topology algorithms had already been invented (such as the SANE and ESP algorithms from David Moriarty and Faustino Gomez, both in collaboration with Risto Miikkulainen). These algorithms were already quite clever, investigating the idea that the individual neurons of a network might be evolved in their own subpopulations to cooperate with other neurons, and then grouped with neurons from other subpopulations to form full working networks, an idea generally known as “cooperative coevolution.”

Arriving on the scene in the midst of these developments, I was enchanted with the idea that the evolution of brains could happen inside a computer. I appreciated the importance of evolving ANNs to solve problems, but my real passion lay with the idea of increasing complexity over evolution. I wanted to build an algorithm that would enable ANNs to explode in complexity inside the computer, echoing the evolution of brains in nature. For some reason, even though existing TWEANN algorithms could change the architecture of evolving ANNs over time, they seemed to lack the strong propensity toward increasing complexity that I hoped to capture.

I spent a lot of time studying existing TWEANN algorithms and thinking about why they might not be working as well as they could. Some problems were well known, such as the competing conventions problem, which refers to the fact that it is hard to combine two parent ANNs to create an offspring (an operation called “crossover”). More specifically, different networks might express the same functionality with different combinations of connections weights, making it hard to know how to combine them as parents.

Other problems I soon confronted were not yet recognized, such as the tendency for new architectures to go extinct from an evolving population before they could realize their potential. The problem in this case is that the initial impact of changing architecture from one generation to the next is usually negative, even if future evolution might eventually take advantage of the new weights (by optimizing them further) given sufficient generations.

The result of my time seeking a better TWEANN algorithm (in collaboration with my Ph.D. advisor Risto Miikkulainen) was an algorithm called NeuroEvolution of Augmenting Topologies, or NEAT, which quickly became the most popular and widely used algorithm in neuroevolution. NEAT’s appeal is in its ability to evolve increasingly complex ANNs through a series of innovations that sidestep the typical TWEANN challenges. While earlier algorithms (like Inman Harvey’s SAGA algorithm) provided hints that complexity can evolve, the appeal of NEAT is in its explicit solutions to the various problems of neuroevolution of its day. For example, NEAT marks genes with something called a “historical marking” to ensure a coherent result of crossover. It also implements a particular kind of speciation calibrated for TWEANNs that allows innovative new structures more breathing room to optimize before being prematurely eliminated from the population.

NEAT obtained record pole-balancing benchmark results, but it was the potential to start simply and then increase complexity over generations, the trademark behavior of NEAT, that inspired my imagination more than benchmark results. In fact, although it was invented before the advent of the term “deep learning,” NEAT has the intriguing capability to evolve increasingly deep networks.

NEAT has been applied by myself and others to innumerable applications, from controlling robots to controlling video game agents. It has also found a niche within computational creativity, where (using examples from my own lab at UCF) it evolves networks that output art (such as Picbreeder) and music (e.g., MaestroGenesis ). Its creative applications also extend to video games such as Galactic Arms Race, where it provides a force for generating new content.

Among its most significant impacts, NEAT was the optimization method used at the Tevatron particle accelerator to find the the most accurate mass estimate of the top quark to date (see here for the main result and here for more details on the application of NEAT). A video by YouTube star Sethbling recently helped to educate over four million viewers on the inner workings of NEAT through evolving a controller for Mario in Super Mario Bros.

Indirect encoding

While it might sound like NEAT is the final word on neuroevolution, a captivating aspect of research in this field is that major advances often reveal limitations and flaws that were not previously clear. These revelations, in turn, produce new questions and new generations of algorithms, and the cycle continues. In effect, over time we are uncovering something approximating nature’s secrets—the secrets to how brains evolved—which are endlessly surprising.

For example, one limitation of NEAT that increasingly occupied my thinking is that it uses a kind of “artificial DNA” called direct encoding, which means that every connection in the ANN is described in the genome by a single corresponding gene. While that works out fine when you have several hundred connections or so, it starts to get pretty unwieldy if you’re aiming for larger brains. As an example, the 100 trillion connections in the human brain would require 100 trillion genes to encode in NEAT. In other words, while NEAT can indeed expand the size of the ANNs it evolves by adding connections and neurons through mutations, adding 100 trillion connections would take on the order of 100 trillion mutations. Even if somehow it managed to evolve a new connection every second (which means a new generation would need to be born every second), the sun would be long extinguished by the time the size of brains reached 100 trillion. So, NEAT must still be missing some property present in nature that makes it possible to evolve such massive structures.

In response to this challenge, neuroevolution researchers have explored a different class of genetic encodings called indirect encodings (as opposed to direct), where the number of genes can be much fewer than the number of connections and neurons in the brain. In other words, the “DNA” is a compressed representation of the brain. Pioneering work in indirect encodings (both in encodings ANNs and physical forms) by researchers such as Josh Bongard, Greg Hornby, and Jordan Pollack helped to highlight the power of indirect encodings to evolve much more impressive and natural-looking structures. These include evolved artificial creatures that exhibit regularities reminiscent of natural organisms (some of Bongard’s are shown at here) as well as a remarkable set of evolved tables (the kind with elevated, flat surfaces) with interesting structural patterns from Hornby and Pollack at Brandeis (see them at here). Among the most eye-catching demonstrations of indirect encoding (from before they caught on in neuroevolution) are the virtual creatures of Karl Sims, whose video from over 20 years ago remains captivating and unforgettable.

Indirect encodings within neuroevolution (focusing now on encoding ANNs in particular) have made it possible to evolve much larger ANNs than with direct encodings like NEAT. One of the most popular such indirect encodings is called compositional pattern-producing networks (CPPNs), which was invented in my lab, the Evolutionary Complexity Research Group at the University of Central Florida, in response to the limitations of NEAT as a direct encoding. A CPPN is basically a way of compressing a pattern with regularities and symmetries into a relatively small set of genes. This idea makes sense because natural brains exhibit numerous regular patterns (i.e., repeated motifs) such as in the receptive fields in the visual cortex. CPPNs can encode similar kinds of connectivity patterns.

When CPPNs are used to generate the connectivity patterns of evolving ANNs, the resulting algorithm, also from my lab, is called HyperNEAT (Hypercube-based NEAT, co-invented with David D’Ambrosio and Jason Gauci) because under one mathematical interpretation, the CPPN can be conceived as painting the inside of a hypercube that represents the connectivity of an ANN. Through this technique, we began to evolve ANNs with hundreds of thousands to millions of connections. Indirectly encoded ANNs have proven useful, in particular, for evolving robot gaits because their regular connectivity patterns tend to support the regularity of motions involved in walking or running. Researchers like Jeff Clune have helped to highlight the advantages of CPPNs and HyperNEAT through rigorous studies of their various properties. Other labs also explored different indirect encoding in neuroevolution, such as the compressed networks of Jan Koutník, Giuseppe Cuccu, Jürgen Schmidhuber, and Faustino Gomez.

Novelty search

Now you can begin to see a trajectory of ideas. The field started with fixed-topology networks, moved to complexifying networks, and then began to focus on indirectly encoded networks. These shifts in perspective and capabilities will continue as we gain more insight into the evolution of complexity. One other recent influential idea in the field, also from our lab (and echoed in seminal studies by Jean-Baptiste Mouret and Stéphane Doncieux), is the idea that parents should not always be selected based on their objective performance, but rather based on their novelty. In other words, perhaps surprisingly, sometimes it is actually detrimental to the long-term success of evolution to select which parents will reproduce based on fitness. For example, consider trying to evolve an ANN for a walking robot. You might think that to get the best walker, the current best walkers should be the parents of the next generation. However, that turns out not to be the case.

In fact, early in evolving the ability to walk, the best ANNs for controlling walkers may rely merely upon lurching forward unreliably. Perhaps other candidates do something interesting like oscillating the robot’s legs in a regular pattern, but suppose these interesting robots simply fall down—if fitness is our guide, we would prefer the ones who manage to travel farther and thereby drop the oscillators from further consideration. But here’s the problem: it could be that the more interesting concept (oscillating) is more likely to lead to robust walking than the currently best strategy (lurching). Yet fitness selects against oscillation simply because at the current stage it isn’t so great. In other words, the stepping stones to the best walker may be deceptive—in which case, simply choosing from among the best is likely to move to a dead end.

This insight (among other considerations) led me to the idea for the novelty search algorithm (co-invented with my then-Ph.D. student Joel Lehman). Novelty search is a selection scheme based on the idea that you choose the most novel as parents instead of choosing the most fit.

This idea may sound strange at first, so let’s back up for a moment. In evolution, there are two primary factors that impact its performance—one is the representation or encoding, which in nature is DNA. The work on increasing complexity and indirect encoding falls into this category. The other big factor is how the system determines who will reproduce, which is called selection. Almost all the effort on selection over the history of evolutionary computation has focused on selecting parents based on their fitness. That doesn’t mean we necessarily always or only choose the most fit, because that would quickly drain diversity from the population, but it does mean that the more fit tend to have a higher chance of reproducing than the less fit.

Novelty search turns this fitness-driven paradigm on its head, dropping the usual notion of fitness (i.e., quality of behavior) entirely from its selection scheme. Instead, now, the computer looks at the behavior of an individual candidate (say a walking robot) and compares it to behaviors seen in prior generations. In novelty search, if the new candidate’s behavior is significantly different from what has come before, it has a high chance of being selected to reproduce, regardless of how fit it is.

This idea stirred up some controversy because people like the idea of selecting based on their criteria for success, which is captured by fitness, but it was hard to ignore that novelty sometimes works. That is, sometimes you could actually select for novelty and get a better solution in faster time than you would have if you selected for fitness. Consider the example of the walking robot. Unlike fitness-based evolution, novelty search is likely to select the oscillator that falls down as a parent simply because it is doing something new, while fitness-based search would almost certainly ignore it for falling over so soon. But as we discussed, it turns out that oscillation is an essential stepping stone to robust walking. So, counterintuitively (because of deception), it can actually be better to ignore your objective and select simply for novelty rather than for improving objective performance.

This insight shifted how we think about neuroevolution once again, and has led to a whole new research area called “quality diversity” and sometimes “illumination algorithms.” This new class of algorithms, generally derived from novelty search, aims not to find a single optimal solution but rather to illuminate a broad cross-section of all the high-quality variations of what is possible for a task, like all the gaits that can be effective for a quadruped robot. One such algorithm, called MAP-Elites (invented by Jean-Baptiste Mouret and Jeff Clune), landed on the cover of Nature recently (in an article by Antione Cully, Jeff Clune, Danesh Tarapore, and Jean-Baptiste Mouret) for the discovery of just such a large collection of robot gaits, which can be selectively called into action in the event the robot experiences damage.

If you think about it, novelty is not so much about solving a particular problem as it is about finding all the interesting possibilities that exist within the search space. The fact that it still sometimes gives you a state-of-the-art solution is just an intriguing side effect of this broader process. Quality diversity algorithms (the next step beyond novelty search) add the notion of fitness back in, but in a careful way so as not to undermine the delicate push for novelty. The result is a class of algorithms that diverge through the search space, while still pulling out the best of what is possible.

This new direction is important for neuroevolution, but it’s interesting to note that it goes beyond neuroevolution, machine learning, or even computer science. Consider: the principle that sometimes the best way to achieve something noteworthy is to avoid explicitly trying to achieve it is actually so general that it even can apply to life and society. For example, the best way to raise student achievement may not be through delivering a continual stream of tests. The fact that this insight from neuroevolution has such potentially broad implications is no accident—it is natural (and exciting) that a concerted attempt to improve the ability of algorithms to explore the space of the possible would lead to insights about innovation and achievement in general.

In other words, as we crack the puzzle of neuroevolution, we are learning not just about computer algorithms, but about how the world works in deep and fundamental ways. That’s one reason that myself and my co-author Joel Lehman wrote the book, Why Greatness Cannot Be Planned: The Myth of the Objective. We wanted to share some of the broader implications of developments in our field with the larger public. We believe the implications are indeed important well outside computer science, for institutions focused on innovation and creativity.

The comeback

I began this article with the claim that neuroevolution is back, but what does that actually mean? In reality, the field never really went away, and many of its top researchers have remained in the trenches all along. However, the ascent of deep learning has certainly refocused a lot of the attention that might have gone to neuroevolution in the past. Now that’s starting to change.

One of the main reasons for deep learning’s success is the recent increase in available hardware processing power. ANNs turn out to benefit from parallel processing (e.g., in GPUs), whose availability has exploded over the last few years. While many have viewed this marriage between ANNs and powerful new hardware in deep learning as a unique and serendipitous union, it is not lost on some that neuroevolution may be on the cusp of a similar story. That is, neuroevolution is just as eligible to benefit from massive hardware investment as conventional deep learning, if not more. The advantage for neuroevolution, as with all evolutionary algorithms, is that a population of ANNs is intrinsically and easily processed in parallel—if you have 100 ANNs in the population and 100 processors, you can evaluate all of those networks at the same time, in the time it takes to evaluate a single network. That kind of speed-up can radically expand the potential applications of the method.

One consequence is that labs with access to large-scale computing clusters can see that they might be sitting on a neuroevolution goldmine, prompting a new generation of researchers and next-generation neuroevolution experiments to grow out of labs largely otherwise invested in conventional deep learning. There’s also an underlying worry about potentially missing the next big thing.

For example, Google Brain (an AI lab within Google) has published large-scale experiments encompassing hundreds of GPUs on attempts to evolve the architecture of deep networks. The idea is that neuroevolution might be able to evolve the best structure for a network intended for training with stochastic gradient descent. In fact, the idea of architecture search through neuroevolution is attracting a number of major players in 2016 and 2017, including (in addition to Google) Sentient Technologies, MIT Media Lab, Johns Hopkins, Carnegie Mellon, and the list keeps growing. (See here and here for examples of initial work from this area.)

Another area where neuroevolution is making a comeback is in reinforcement learning, which focuses on control and decision-making problems (like playing Atari games or controlling a biped robot). A team at OpenAI recently reported an experiment where a variant of neuroevolution matches the performance of more conventional deep-learning-based reinforcement learning on a slew of benchmark tasks, including Atari games originally conquered in widely publicized work by Google’s DeepMind AI think tank in deep learning.

The field of neuroevolution also continues to move along its own unique paths. For example, much research over the years and continuing to this day has focused on how to evolve plastic ANNs, which means networks with connection weights that change over the lifetime of the network. That is, evolution in this case is not just deciding the architecture and weights, but also the rules that guide how and when particular weights change. That way, the resultant networks are, in principle, more like biological brains, which change over their lifetimes in reaction to their experience. Much of my own thinking on the topic of plastic neural networks is influenced by the early works of Dario Floreano and later ideas on neuromodulation, which allows some neurons to modulate the plasticity of others, from Andrea Soltoggio.

Another interesting topic (and a favorite of mine) well suited to neuroevolution is open-endedness, or the idea of evolving increasingly complex and interesting behaviors without end. Many regard evolution on Earth as open-ended, and the prospect of a similar phenomenon occurring on a computer offers its own unique inspiration. One of the great challenges for neuroevolution is to provoke a succession of increasingly complex brains to evolve through a genuinely open-ended process. A vigorous and growing research community is pushing the boundaries of open-ended algorithms, as described here. My feeling is that open-endedness should be regarded as one of the great challenges of computer science, right alongside AI.

Video games also remain a popular application, not just for controlling characters in games, but for evolving new content. If you consider all the prolific variety of life on Earth, you can see how evolutionary algorithms could be a natural conduit for generating diverse new content. Georgios Yannakakis at the University of Malta and Julian Togelius at New York University, both pioneers in the field themselves, present a broad overview of many applications of neuroevolution (and other AI algorithms) to gaming in their new book, Artificial Intelligence and Games.

Getting involved

If you’re interested in evolving neural networks yourself, the good news is that it’s relatively easy to get started with neuroevolution. Plenty of software is available (see here), and for many people, the basic concept of breeding is intuitive enough to grasp the main ideas without advanced expertise. In fact, neuroevolution has the distinction of many hobbyists running successful experiments from their home computers, as you can see if you search for “neuroevolution” or “NEAT neural” on YouTube. As another example, one of the most popular and elegant software packages for NEAT, called SharpNEAT, was written by Colin Green, an independent software engineer with no official academic affiliation or training in the field.

Considering that evolution is indeed the only genuine means we know by which intelligence at the human level has ever been produced, the potential in the field of neuroevolution is great for further advances and new insights with the increasing computational power available today.