Train your Ernest

Thanks to Olivier Voisin, you can now train your own Ernest online in 3D: click here to access Ernest 8.3-3D (Preferably with Chrome, but works also with Firefox).

Poor Ernest 10.3

We hesitate calling this experiment poor Ernest 10.3 or happy Ernest 10.3. Maybe this is the first implementation of artificial dizziness :-).

After the initial learning phase, we introduce fish that move (step 158). We were happily surprised to see that Ernest 10.3 was not that bad at catching moving fish. We find it fairly honorable for a purely feed-forward adaptive system.

Ernest 10.3's traces

This video shows the trace generated by Ernest 10.3 during the experiment reported in the previous post. Playing the two videos synchronously helps understand Ernest's activity.

As in Ernest 10.1's traces, the tape at the bottom represents the visual field. The twelve visual pixels are represented vertically as little rectangles when Ernest is moving forward, and as little trapezoids when Ernest is turning.

The second tape above the visual field represents the peripersonal map (known as the local map in Ernest 10.1). The grid cell in front of Ernest is represented in the center of the tape (in red when Ernest is bumping). The three cells on the left side of Ernest are represented in the upper part of the tape. The three cells on the right side of Ernest are represented in the lower part of the tape. The cell where Ernest is standing and the cell in the back of Ernest are not represented. The peripersonal map integrates stimulations from different sensory modalities: tactile (light gray, intermediary gray, and black), visual (colored), and kinematic (red). These different stimulations may be bundled together (See the discussion on bundles in Ernest 10.1).

The central part of the video represents Ernest's primitive interaction patterns (primitive enacted acts) that relate to Ernest's decisions (i.e., enacted acts both result from Ernest's previous decision and impact Ernest's next decision). The central line takes the color of the sensory salience that attracts Ernest's current attention (gray in the case of a tactile salience). Triangles that point outwards from the central line indicate that the salience is moving outwards—to the left when the triangle is above the line, and to the right when the triangle is below. Triangles that point inwards to the line indicate that the salience is moving inwards—from the left when the triangle is above the line, and from the right when the triangle is below. Little squares on the central line indicate that the salience of current attention is enlarging in the central area of the visual field. Large squares on the central line represent eating a fish (from gustatory stimulation).

Above the decisional tape is the motivational tape. The motivational tape represents Ernest's current satisfaction value as a little histogram. Positive satisfactions are displayed in green and negative satisfactions in red. See how Ernest enjoys eating fish :-).

Above the motivational tape, orange stripes and circles represent composite schemas that Ernest tries to enact as a whole sequence (from step 128 on). These sequences correspond to regularities of interaction that Ernest has discovered and learned; and Ernest is beginning to exploit these sequences. We can see that these sequences may contain steps with negative satisfaction values (steps 185, 195, 218...). This demonstrates that Ernest learns to knowingly decide to enact unsatisfying interaction in order to gain subsequent satisfying interaction.

Ernest 10.3

In his management of space, Ernest 10.3 distinguishes between two concentric zones: the peripersonal space and the extrapersonal space.

The peripersonal space is the realm of proximal perception: touch, kinematic perception (bumping), and taste (eating). The peripersonal space covers the 3x3 matrix of grid cells surrounding Ernest.

In contrast, the extrapersonal space is the realm of distal perception: sight (that also works as a kind of olfaction for Ernest). The extrapersonal space covers the rest of the surrounding world beyond the peripersonal space.

Researchers in neuropsychology have argued that these two spaces were handled by partially distinct parts of the brain in the vertebrate. Moreover, behaviors would be driven by different motivational systems. Behaviors in the peripersonal space would be related to consumatory motivation associated with the noradrenergic system. Behaviors in the extrapersonal space would be related to incentive associated with the dopaminergic system (Previc, 1998, p124).

Following these views, we have endowed Ernest with a (partially) distinct motivational system associated with each space. This video shows that this new dual-space motivational system makes Ernest much better at avoiding bumping and at catching fish in shoal. See steps 93-98 when Ernest hesitates between moving toward another shoal or keeping eating the same shoal.

Similar to Ernest 10.1, Ernest 10.3 constructs a local map of his peripersonal space (displayed in the upper-right corner). This local map is a place of multimodal integration that facilitates behavioral consistency across the two spaces, peripersonal and extrapersonal.

After the initial learning phase (roughly up to step 80 in this video), Ernest 10.3 exhibits a more coherent spatial behavior than poor Ernest 9.3, in particular, Ernest now avoids bumping as well as useless turning toward walls.

Also worthy of noting is that Ernest adopts a head bobbing behavior (like birds) consisting of turning left then right after eating (from step 128 on). We explain this strategy by that Ernest's sensory system is mostly sensitive to movement (see the discussion on Ernest 8.1 strategy learning).

Reference

Previc F.H, 1998, The neuropsychology of 3-D space. Psychological Bulletin, 124 (2). pp 124-164.

Ernest 10.2 simulates spatial movements

Ernest 10.2 anticipates the consequences of his actions in his local map. When the local map predicts that Ernest would bump into a wall, the action of moving forward is inhibited (not enacted) to prevent the bumping.

The local map is now displayed with the shape of a shark in square k6. Unlike in the previous experiment, this display shows the anticipation made during the previous step. For example, on step 16, the local map shows that Ernest was unable to anticipate (from step 15) the appearance of the yellow square in front of him. On step 17, however, he was able to anticipate (from step 16) that the yellow square would shift to the right when he turns to the left (the local map is most often incomplete and sometimes wrong).

When the local map predicts that Ernest would bump into a wall, the local map shows a red circle on Ernest's nose. This occurs for the first times on step 10, 32, and 58. On step 58, the fact that the bumping was prevented is shown by the fact that the wall in front of Ernest does not flash red.

As before, Ernest creates bundles that associate different sensory stimulations together. For example, on step 9, Ernest predicts that moving forward would make him stand on a wall. This false prediction is due to the fact that Ernest does not know yet that walls cause bumping. When Ernest actually experiences the bumping on step 9, he bundles together the tactile stimulation of walls with the bumping stimulation (the dark green color is not associated with this bundle because Ernest's visual attention was distracted by the yellow alga, which caused him to not see the wall). On the contrary, during step 90, Ernest did see the turquoise wall, which caused him to learn the "turquoise wall bundle".

We expect that Ernest's new coupling between his intrinsically-motivated sequential learning system and his spatial representation system can lead to valuable new developments. Yet, many issues remain. In particular, the current implementations of the local map and the bundle mechanism are based on some ad-hoc routines that we have hard-coded. For Ernest to adapt to other environments, these mechanisms need to be learned rather than hard-coded. We believe that such learning could be implemented with statistical methods, such as those used in robotics studies (e.g., Kuipers 2000).

References

Kuipers, B. (2000) The Spatial Semantic Hierarchy. Artificial Intelligence. 119 (1-2), pp 191-233.