Knowing that a reward is coming is not the same as knowing when it will arrive. A small reward expected soon and a larger reward expected much later can look similar when the future is reduced to a single number, like in common reinforcement learning approaches. Inspired by learning in humans (see our behavioral work for more on this), we are developing models that preserve this timing information by representing expected reward along a future timeline. Consistent with data from humans, our timeline represents the near future in greater detail and the distant future more approximately. This allows it to cover both short and long delays without requiring the same precision at every point in time.
Replacing a scalar estimate of expected cumulative reward with the reward timeline allows the same learned representation to be evaluated in different ways. With little time available, an agent may favor a nearby reward. With a longer time horizon, it may choose a larger reward that is farther away.
This means that the agent can adjust to a new deadline or a different way of weighing near and distant outcomes without learning the task again from the beginning.
The videos below show the game on the left and the predicted future reward on the right. The horizontal axis represents future steps.