Training Ernest 7

This video demonstrates that Ernest develops different behaviors depending on the experience he has during his youth. Here, we have two instances of Ernest: Ernest 1 (brown) is initially kept in the small loop and released on step 290. Ernest 2 (bleu) is confronted to the complex environment right from his birth.

Ernest 1 develops more sophisticated behaviors than Ernest 2 because he is trained to touch both of its sides when he faces a wall. Consequently, after being released, he has a more exploratory behavior than Ernest 2.

Ernest 2 learns to preferably turn to the right when he faces a wall. Consequently, he tends to keep spinning in limited areas of the environment. Ernest 2's learning is limited by the fact that the environment is initially too complex for him to notice sophisticated sequences that involve touching to both sides.

The importance of training is an interesting property of Ernest because it accounts for theories of developmental learning.


(Demo implemented with Ernest r296 and Vacuum r203)

Ernest's source code

Ernest's source code is available here with the instructions to use it. The cleaned-up and tested recommended revision is r296. This revision demonstrates the exact behavior of Ernest 7 reported in the Small Loop Experiment.

Interactional Motivation

 Olivier L. Georgeon, James B. Marshall, and Simon L. Gay 2012. Interactional Motivation in Artificial Systems: Between Extrinsic and Intrinsic Motivation.  In proceedings of the 2nd Joint IEEE International Conference on Development and Learning and on Epigenetic Robotics (EPIROB 2012), San Diego, pp. 1-2.

This paper presents the notion of interactional motivation that drives Ernest, and compares it to reinforcement learning as it is traditionally implemented in Partially Observable Markov Decision Processes (POMDPs).

Ernest 11.5 constructs goals

Like Ernest 11.4, Ernest 11.5 can recognize objects by the possibilities of interaction that they afford. Additionally, Ernest 11.5 has a specific inborn taste for stepping on flowers.

Ernest 11.5 simulates different possible sequences of interactions in spatial memory before selecting the best sequence to enact. These simulations are represented in the bottom-right area of the video. Simulations that produce predictable results (due to information available in spatial memory) are represented with orange outlines. Simulations that produce unpredictable results (due to the lack of information in spatial memory) are represented in blue. The video shows that Ernest learns to simulate increasingly elaborated sequences of interactions as time goes on (see blue squares and triangles spreading in all directions around Ernest from step 253 on).

The high value associated with stepping on flowers favors simulations that lead to even more stepping on flowers. As a result, Ernest learns to make a u-turn to return to a flower when he passes one (see Ernest keeping stepping on the flower from step 260 on).

We find this experiment interesting because it illustrates how an inborn drive can give raise to an explicit goal. Ernest's inborn tendency to step on flowers makes Ernest identify flowers as an interesting goal to reach. Once this goal is recognized, Ernest performs a rudimentary problem-solving computation to reach it. Perhaps the skill to choose a desirable point in space and find a sequence of operations to reach this point underlays higher-level problem-solving skills.

Ernest 11.4 recognizes objects

This video shows Ernest 11.4 learning to interact with different objects in this new version of the Small Loop.

At the beginning, Ernest learns to interact with empty places and with dark-green walls. From step 76 on, he learns to interact with cyan walls. On step 220, we introduce alga, and he starts to learn to interact with them.

Note the funny hesitation on step 234 when Ernest touches an alga for the first time, turns back, and then returns to the alga. Once this new kind of objects is learned, Ernest moves through them without hesitation.

Ernest's previous management of bundles (Ernest 11.2) no longer works in this environment because objects can no longer be identified by disjoint bundles of interactions. Some interactions (e.g., bump) are afforded by different objects (dark-green walls and cyan walls). Ernest 11.4, however, does not actually need to fully recognize objects. He adapts to this environment by only learning "compresences" of pairs of interactions. We borrowed the term compresence from the bundle theory of objects to designate the tie between two interactions that are afforded by the same location in space. In this video, compresences are represented by gray circles that contain interactions (in sequential and spatial memory, top and bottom right areas of the video).

The question of identifying objects by bundles of interactions that are consistently compresent remains an open and difficult question. The notion of compresence seems  to be still controversial in philosophy of objects. Identifying objects raises the question of making analogies between objects, and learning categories of objects based on similarities in the interactions that they afford.

In this experiment:

Touching a cyan wall ahead generates a specific feeling (cyan squares). Touching a cyan wall on the side generates the same feeling as touching a dark-green wall on the side (dark-green squares). Bumping into a cyan wall feels the same as bumping into a dark-green wall (red triangles). Once learned, touching walls ahead "evokes" bumping ahead (light-red triangles in spatial memory, bottom right area of the video). As previously, the evocation of bumping refrains Ernest from trying to move forward towards walls.

Touching an alga ahead generates a specific feeling (light-green squares). Touching an alga on the side generates the same feeling as touching an empty square on the side (white squares). Moving to an alga feels the same as moving to an empty square (white triangles).

(Demo implemented with Ernest r261 and Vacuum r186)

Ernest 11.3 in e-puck

Ernest 11.3 is an adaptation of Ernest 11.2 for the e-puck robot.

This video shows the e-puck robot in the "Box Environment" (left). The possibilities of interaction and the LED signals are the same as with Ernest 7 in e-puck.

The top-right part of the video shows the sequential trace with the same symbols as previously.

The bottom-center shows the new spatial memory (in an egocentric referential with the robot's front oriented towards the right). When Ernest enacts an interaction, the area that is concerned by this interaction is marked by a halo in spatial memory. Interactions that concern empty places are in white, interactions that concern walls are in green. The superimposition of different interactions in the same spatial location reveals occurrences of empty place phenomena (white halos) and wall phenomena (green halos).

Note that wrong associations can occur due to false detections. For example: false detection of a wall on the left on steps 221 and 222 (time 2:23). (We turned on additional light to provoke more false detections from step 100, time 0:59.)

The bottom-right part of the video represents coefficients of spatial overlapping between interactions (red segments). The more consistent the overlapping, the shorter the segment. Over time, interactions that concern the same kind of phenomena become grouped together because they consistently overlap. Two bundles of interactions emerge: white interactions form a bundle that represents empty places, green interactions form a bundle that represents walls.

On step 229, the false detection made on step 221 and 222 yields to a wrong association between the two bundles (mixed white and green halo in the center of spatial memory on time 2:25, and long red segment between the two bundles). This wrong association, however, does not impact Ernest's behavior too much because it remains weak.

This experiment demonstrates that Ernest 11.3 handles the imprecision in the robot’s displacements and in the sensors by keeping track of probabilities of presence of phenomena in Ernest's surrounding space. Simultaneously, Ernest gradually learns the notions of empty space phenomenon and wall phenomenon by associating the interactions that they afford. In turn, the recognition of phenomena helps Ernest organize its behavior by prompting interactions adapted to the phenomena that surround him.

A Challenge for Emergent Cognition

The Small Loop Problem: A challenge for artificial emergent cognition. Olivier L. Georgeon, James B. Marshall. In proceedings of BICA2012, Annual Conference on Biologically Inspired Cognitive Architectures. Palermo, Italy, pp. 137-144. (October 31, 2012).

This paper presents the Small Loop Problem and how Ernest 11.2 handles it.

Ernest 7 in e-puck

Here is Ernest 7 in an e-puck robot.

We implemented "touch" with the infrared sensors available on the front, left, and right side of the robot. The range of these sensors was set to approximately 5cm. When Ernest "touches", the corresponding led flashes. When the touching detects a wall, the two additional leds on the rear flash. When it bumps into a wall, all the leds flash.The symbols in the trace are the same as previously.

This video shows that Ernest learns to touch ahead before moving forward to avoid bumping, and learns to turn when it reaches a wall.