Sign in to your account

Steve James


In this work, we investigate ways an agent can combine existing skills to create novel ones in a manner that is both principled and optimal. We find that by constraining the reward function and transition dynamics, skills composition can be achieved in both entropy-regularised and standard RL. Our approach allows an agent to generate new skills without further learning, and can be applied to high-dimensional environments and deep RL methods.


In this post, we look to answer the following question: given a set of existing skills, can we compose them to produce novel behaviours? For example, imagine we have learned skills like running and jumping. Can we build more complex skills by simply combining them in interesting ways? Of course, it is likely that there are many ad-hoc ways to do this, but it would be really nice if we could do it in a way that is both provably correct and that requires no further learning.

To illustrate what we're after, we use pretrained skills generated by DeepMimic [Peng, 2018]. As we can see, in the first two animations below our agent has learned to backflip and spin. But if we try to combine these skills together (in this case, by taking the maximum of the policy network output), we get complete bupkis! The question is: is there some way we can make this work?

Attempting to compose two skills - backflips and spins - results in an agent that can injure itself optimally.

Unsurprisingly, the answer is "yes". More...

Orhan Can Görür

Through Social Cobots: Robots with Human-Like Collaboration Skills

This study is a part of a collaboration with DAI-Labor of TU Berlin, Germany. We envision the future of collaborative robots (cobots) in industry through their fully autonomous human-like collaboration with human partners.

Our research aims to address the question: "How do we build cobots with human-like natural collaboration skills?". Existing intention-aware planning approaches often make the assumptions that a human collaborator's actions are always relevant to the collaborated task and that the human always accepts the cobot's assistance when offered.

We believe that these assumptions are a significant barrier against having social cobots in the real world. In reality, a human's dynamic desires and emotional states could result in stochastic human intentions, especially in repeated tasks. A cobot with these assumptions may misinterpret the human actions, which may result in intrusive and unreasonable robot behaviors (e.g. a human gazing at an object might be interpreted as she needs it, yet behind this gaze, she could be evaluating to take it herself or thinking of something irrelevant to the task like staring into space).

Our goal is to offer a new model design as an alternative to the conventional intention-aware models by removing these assumptions. The result is our novel robot decision-making model, a partially observable Markov decision process (POMDP), that is capable of handling these dynamic...

Ofir Marom


An important area of study in the field of Reinforcement Learning (RL) is Transfer Learning where the aim is to leverage previous experiences to accelerate learning in a new unseen tasks. While it is clear that living organisms apply transfer learning throughout their lives, it is often unclear how this transfer mechanism exhibited by living organisms can be incorporated into autonomous agents

As a concrete example, consider a simple Sokoban task as below.


Once a human completes this task they have learned core concepts about the underlying structure of the Sokoban domain. For example, they would learn that the warehouse-keeper:

  1. cannot walk through walls,
  2. cannot push a box that is adjacent to another box
  3. cannot push a box that is adjacent to a wall

Now suppose the human would be given this new more complex task to solve.


Clearly, the human would re-use the rules they had previously learned in the simple task to gain an advantage in this new task. Unfortunately, most state-of-the art RL algorithms would not leverage such knowledge and would instead re-learn everything from scratch on the more complex task. Such wastefulness of prior experience is clearly inefficient!

Object-Oriented Representation

One idea that has shown promise in transfer learning is the notion of object-oriented representation. With this approach we view a task as being instances of objects classes. For example, any Sokoban task can be thought of as made objects that are instances...