Representing Qualitative Action Models for Learning in Complex Virtual Worlds

Abstract

This thesis addresses the problem of representing and learning qualitative models of behaviour in complex virtual worlds. It presents a novel representation, ‘Q-Systems’, that integrates two existing representation frameworks: qualitative process models and action description languages. QSystems combines the expressive power of both frameworks to allow actions and world dynamics to be modelled in a common way using a representation based on non-deterministic and probabilistic finite state machines. The representation supports learning and planning by using a modular approach that partitions world behaviour into ‘systems’ of objects with specific contexts and a related behaviour. Q-Systems was developed and tested using an agent in a rich simulated world that was created as part of the thesis. The simulation uses a rigid body physics engine to produce complex realistic interactions between objects. An action system and a qualitative vision system were also developed to allow the agent to observe and act in the simulated world. The thesis includes a proposed two stage learning process comprising an initial stage in which ‘histories’ (contextually and temporally restricted sequences of observations) are extracted from interactions with the simulation, and a second stage in which the histories are generalised to create a knowledge base of system models. An algorithm for generating histories is presented and a number of heuristics are implemented and compared. A system for learning generalised models is presented and it is used to assess the suitability of Q-Systems with respect to learning in complex environments. Planning with Q-Systems is demonstrated in an agent which reasons with generalised models to work out how to achieve goals in the simulated world. A simple planning algorithm is described and a variety of issues are explored. Planning with a single system is shown to be relatively straightforward due to the modular nature of Q-Systems. This thesis demonstrates that Q-Systems successfully integrate two different representation frameworks and that they can be used in learning and planning in complex environments. The initial results are promising, but further investigation is required to fully understand the advantages and disadvantages of the Q-System approach compared with existing learning systems. This would involve the development of benchmark problems (currently there are none for this particular domain).