Poor Ernest 2.0

So, Ernest 2.0 can adapt to two different environment: the so called ABAB and BBBB environments. But what if Ernest is put into a ABAB environment that turns into a BBBB environment after a while? Let's call this third environment the AB--ABBB--BB environment.

Again, this is catastrophic. The schemas that were learned in the ABAB situation are no longer working in the BBBB situation. Ernest learns new schemas corresponding to the BBBB situation, but these new schemas are contradictory to the previous ones. As poor Ernest is unable to chose between concurrent schemas, he is irremediably lost when the environment turns to BBBB.

What Ernest needs is a way to loose confidence in schemas that do not meet expectations, and to reinforce confidence in schemas that meet expectations. More generally, Ernest needs a way to attribute a confidence value to his schemas, because in complex environments, he cannot assume them to work all the time.

Fortunately, the newly-released Soar version 9 offers reinforcement learning facilities. So far, Ernest was designed with Herbal and implemented with the Jess rule engine. We now need to implement Ernest with Soar to take advantage of reinforcement learning.