Ernest 10.3

In his management of space, Ernest 10.3 distinguishes between two concentric zones: the peripersonal space and the extrapersonal space.

The peripersonal space is the realm of proximal perception: touch, kinematic perception (bumping), and taste (eating). The peripersonal space covers the 3x3 matrix of grid cells surrounding Ernest.

In contrast, the extrapersonal space is the realm of distal perception: sight (that also works as a kind of olfaction for Ernest). The extrapersonal space covers the rest of the surrounding world beyond the peripersonal space.

Researchers in neuropsychology have argued that these two spaces were handled by partially distinct parts of the brain in the vertebrate. Moreover, behaviors would be driven by different motivational systems. Behaviors in the peripersonal space would be related to consumatory motivation associated with the noradrenergic system. Behaviors in the extrapersonal space would be related to incentive associated with the dopaminergic system (Previc, 1998, p124).

Following these views, we have endowed Ernest with a (partially) distinct motivational system associated with each space. This video shows that this new dual-space motivational system makes Ernest much better at avoiding bumping and at catching fish in shoal. See steps 93-98 when Ernest hesitates between moving toward another shoal or keeping eating the same shoal.

Similar to Ernest 10.1, Ernest 10.3 constructs a local map of his peripersonal space (displayed in the upper-right corner). This local map is a place of multimodal integration that facilitates behavioral consistency across the two spaces, peripersonal and extrapersonal.

After the initial learning phase (roughly up to step 80 in this video), Ernest 10.3 exhibits a more coherent spatial behavior than poor Ernest 9.3, in particular, Ernest now avoids bumping as well as useless turning toward walls.

Also worthy of noting is that Ernest adopts a head bobbing behavior (like birds) consisting of turning left then right after eating (from step 128 on). We explain this strategy by that Ernest's sensory system is mostly sensitive to movement (see the discussion on Ernest 8.1 strategy learning).

Reference

Previc F.H, 1998, The neuropsychology of 3-D space. Psychological Bulletin, 124 (2). pp 124-164.

Ernest 10.2 simulates spatial movements

Ernest 10.2 anticipates the consequences of his actions in his local map. When the local map predicts that Ernest would bump into a wall, the action of moving forward is inhibited (not enacted) to prevent the bumping.

The local map is now displayed with the shape of a shark in square k6. Unlike in the previous experiment, this display shows the anticipation made during the previous step. For example, on step 16, the local map shows that Ernest was unable to anticipate (from step 15) the appearance of the yellow square in front of him. On step 17, however, he was able to anticipate (from step 16) that the yellow square would shift to the right when he turns to the left (the local map is most often incomplete and sometimes wrong).

When the local map predicts that Ernest would bump into a wall, the local map shows a red circle on Ernest's nose. This occurs for the first times on step 10, 32, and 58. On step 58, the fact that the bumping was prevented is shown by the fact that the wall in front of Ernest does not flash red.

As before, Ernest creates bundles that associate different sensory stimulations together. For example, on step 9, Ernest predicts that moving forward would make him stand on a wall. This false prediction is due to the fact that Ernest does not know yet that walls cause bumping. When Ernest actually experiences the bumping on step 9, he bundles together the tactile stimulation of walls with the bumping stimulation (the dark green color is not associated with this bundle because Ernest's visual attention was distracted by the yellow alga, which caused him to not see the wall). On the contrary, during step 90, Ernest did see the turquoise wall, which caused him to learn the "turquoise wall bundle".

We expect that Ernest's new coupling between his intrinsically-motivated sequential learning system and his spatial representation system can lead to valuable new developments. Yet, many issues remain. In particular, the current implementations of the local map and the bundle mechanism are based on some ad-hoc routines that we have hard-coded. For Ernest to adapt to other environments, these mechanisms need to be learned rather than hard-coded. We believe that such learning could be implemented with statistical methods, such as those used in robotics studies (e.g., Kuipers 2000).

References

Kuipers, B. (2000) The Spatial Semantic Hierarchy. Artificial Intelligence. 119 (1-2), pp 191-233.

Ernest 10.1's activity traces

Thanks to Pierre-Yves Ronot, we can now see Ernest's activity traces in real time.

In this video, the lower part of the trace shows Ernest's visual field. Visual pixels are represented as little rectangles when Ernest is moving forward, and as little trapezoids when Ernest is turning. Vertical red lines across the visual field indicate that Ernest is bumping into a wall. Blue circles indicate that he is eating a fish. The trace shows the different items in the environment going back and forth through the visual field as Ernest is moving.

We can see the initial phase of sensorymotor learning (steps 0-90) during which Ernest is often bumping into walls. Notably, his visual apparatus does not inform him about his distance to walls.

The line at the upper part of the trace takes the color of the item that recieves Ernest's current attention. This item of visual attention is also indicated by the "pie" in the local map as reported in the previous post.

For example, before eating his first fish, Ernest is distracted by other items (see steps 90 to 99). After having tasted a fish, however, Ernest focuses on fish whenever he sees them (steps 148 to 166).

Traces help us understand how Ernest sees his world. Ernest receives no other information from the world than that displayed in the trace. Traces also represent Ernest's experience that he encodes in his episodic memory. The encoding of experience in the form of activity episodes constitutes the base of Ernest's learning.

Ernest 10.1 constructs a local map of his environment

Ernest 10.1 associates co-occurring sensory stimulations together into bundles. A bundle is a set of sensory stimulations that denotes an object in the world. By constructing bundles, Ernest starts to perform multi-modal sensory integration.

We assume that a bundle can represent a unique kind of object in the world. We drew this assumption from David Hume's bundle theory of objects. This theory postulates that objects consist only of the set (bundle) of their observable properties.

Additionally, Ernest 10.1 constructs a local map of the bundles surrounding him. In this video, the local map is displayed in cell k6 (next to the cycle counter). The local map represents Ernest's awareness of his local surrounding. The top of the map corresponds to the front of Ernest. Ernest's current visual stimulation is represented as a "pie" over the map.

The local map is based on Ernest's somatotopic map. A bundle is learned by associating the tactile stimulation in front of Ernest (the center-top cell in the somatotopic map) with the simultaneous visual stimulation (the "pie"). As he moves, Ernest is then able to follow the newly-created (or recognized) bundle in the somatotopic map.

For example, Ernest creates a bundle for the yellow square on step 27. Then, on step 28, this bundle moves to the center of the map; on step 30, to the center-rear of the map; and, on step 31, to the left-rear corner of the map (the local map is displayed with a delay of one cycle).

When Ernest eats, he associates his gustatory stimulation with the bundle that represents where he is standing, that is, the "fish bundle" that was previously constructed from visual and tactile stimulations. This delicious taste associated with fish then makes Ernest's prefer pursuing fish rather than alga in subsequent activity.

Apart from his still-unexploited local map, and from the fact that his visual field was reduced to a single row of 12 pixels, Ernest 10.1 is the same as poor Ernest 9.3. We now need to investigate how Ernest would use his local map.

Ernest 10.0 has a somatotopic map

Ernest 10.0 is a shark. Sharks are archaic vertebrates whose brain didn't evolve much over the last 450 million years or so. Yet, a shark's brain contains the same set of basic anatomical components as modern vertebrates' brains (Brain, Wikipdia).

In particular, Ernest 10.0 has a somatotopic map—a brain area that represents Ernest's body in an isomorphic way. In humans, this area would correspond to the postcentral gyrus, also known as the primary somatosensory cortex.

In this video, Ernest's somatotopic map is represented as a grayscale grid over Ernest's body. This grid has 9 cells that represent what Ernest touches in the 8 surrounding squares plus the square where he is standing.

Each cell in the somatotopic map can reflect three different kinds of tactile feelings:
- light gray: only water.
- medium gray: something soft, an alga or a fish that Ernest can swim over.
- black: something hard, a wall or the aquarium's side (the central cell is never black because Ernest cannot stand on a wall).

We hypothesize that the somatotopic map will be useful for acquiring a sense of space, although we don't yet know exactly how. We drew this hypothesis from the idea that our sense of space comes from the mere fact that our body occupies space. We, however, could not find much arguments in the literature to support this hypothesis.

Apart from his still-unexploited somatotopic map, and from the fact that he is always hungry for fish, Ernest 10.0 is the same as poor Ernest 9.3. This video shows how miserable he is. He has no sense of space and he is even unable to "simplify" a sequence consisting or turning six times 45° clockwise into a sequence consisting of turning twice 45° counterclockwise (see steps 267 or 279).

Poor Ernest 9.3

On step 406, we removed the yellow-green landmark that Ernest was using to find his way towards the southeast field. This leaves poor Ernest spinning in place, miserably goalless.

To explore space in the absence of landmarks, Ernest will need some skills to construct a spatial representation of the environment. Such skills are exhibited by most vertebrates and are believed to involve some basic components of the vertebrate's brain. We now need to explore the role of these components.

Ernest 9.3

The colliculus activation algorithm has been improved. This gives Ernest 9.3 a smoother behavior than Ernest 9.2 (e.g., see the elegant curve taken on the way back to the hive on steps 356-388).

In addition, Ernest's thorax now takes the color of the most motivating landmark in the visual field—the landmark that raises the most activation in the colliculus. This landmark is the object of Ernest's current attention and drives Ernest's homing tendency.

At the beginning, Ernest is seeking to gather pollen but he does not know how to distinguish flowers from other landmarks. The highest activation is only given by the largest landmark in the visual field (i.e., the closest landmark). This causes Ernest to visit all landmarks randomly. When a landmark is visited, this landmark's activation is lowered for some time, meaning that Ernest temporarily looses interest in this landmark. This causes Ernest to move on to another landmark.

On step 38, Ernest finds a landmark from which he can gather the pollen (the blue flower). This switches his motivation to make him seek for the hive. In the hive-seeking motivational mode, landmarks raise an additional activation that is proportional to their proximity to the hive (or to Ernest's place of birth) as much as Ernest remembers from his way out. This activation mechanism causes Ernest to go back to the yellow square by traveling from known landmarks to known landmarks. After reaching the yellow square (step 83) he finds no more motivating landmarks and starts a random exploration again until he finds the hive for the first time on step 94.

When on the hive, Ernest drops the pollen and switches back to the pollen-seeking motivational mode. Now, the highest activation is generated by landmarks that Ernest remembers as being the closest to the pollen. This leads Ernest back to the northeast flower field. On the second way back, Ernest is now able to recognize the hive, which causes him to turn directly towards the hive on step 159 (by-passing the yellow square).

When the northeast flower field is empty, Ernest fumbles again until he sees new landmarks to explore (green landmarks in the southeast field on step 311). From then on, he starts exploiting the southeast field the same way he exploited the northeast field.

Ernest 9.2 has a superior colliculus

Ernest 9.2 has a visual resolution of 2 rows of 12 pixels. As before, Ernest can see above colored landmarks. The closest landmarks are seen in the first row, and possible landmarks behind these are seen above in the second row. Ernest 9.2's visual angular span equals 180°. As before, Ernest 9.2 cannot see through walls.

In this video, Ernest's half-circular head represents what Ernest sees. The first row is represented inside the half circle. The second row is represented on the half circle's crown. For example, on step 34, the inside of the half circle takes a light-green color because Ernest sees the square where it is standing, and the crown reflects the flower and the other squares that Ernest sees over the light-green square.

The environment is the same as with Ernest 9.1. The hive and the flowers are now represented as icons but Ernest distinguishes them only by their color and taste as before.

The most significant improvement is that Ernest 9.2 has a superior colliculus—a brain region also known as the tectum in the invertebrate. The superior colliculus maintains an internal retinotopic representation of the animal's surrounding environment. The superior colliculus is used to orient motivation and behavior towards a specific direction in the animal's egocentric referential (in some rudimentary vertebrates, like the hagfish, the superior colliculus constitutes the biggest brain region).

The effects of Ernest's colliculus are first seen on step 110. At this point, Ernest was heading toward the light-green square that was already known as leading to the flower field. On step 110, a flower appeared in the right side of the visual field. Because the flower was more motivating than the light-green square, the flower generated more activation in the colliculus's right side than that generated by the light-green square in the colliculus's center. This activation triggered a signal sent to the sequential system that caused Ernest to turn to the right towards the flower.

Ernest 9.2's initial phase of sensorymotor contigency learning lasts longer than before because of the increased complexity of the visual system. In this video, this initial learning phase roughly goes up to step 100. Once sensorymotor contingencies are learned, the pollen gathering is faster and steadier than with Ernest 9.1. Ernest 9.2 finishes gathering the five flowers on step 414, whereas Ernest 9.1 took 598 steps in the previous example run.