Trust Region Policy Optimisation (TRPO)

Trust Region Policy Optimisation, or TRPO, is a policy gradient method in reinforcement learning that avoids parameter updates that change the policy too much with a KL-divergence constraint on the size of the policy update at each iteration.

Take the case of off-policy reinforcement learning, where the policy $\beta$ for collecting trajectories on rollout workers is different from the policy $\pi$ to optimise for. The objective function in an off-policy model measures the total advantage over the state visitation distribution and actions, while the mismatch between the training data distribution and the true policy state distribution is compensated with an importance sampling estimator:

$$\begin{array}{rl}J(\theta )& =\sum _{s\in S}{p}^{{\pi }_{{\theta }_{old}}}\sum _{a\in A}\left({\pi }_{\theta }(a|s){\hat{A}}_{{\theta }_{old}}(s,a)\right)\\ J(\theta )& =\sum _{s\in S}{p}^{{\pi }_{{\theta }_{old}}}\sum _{a\in A}\left(\beta (a|s)\frac{{\pi }_{\theta }(a|s)}{\beta (a|s)}{\hat{A}}_{{\theta }_{old}}(s,a)\right)\\ J(\theta )& ={\mathbb{E}}_{s\sim p^{{\pi }_{{\theta }_{old}}},a\sim \beta }\left(\frac{{\pi }_{\theta }(a|s)}{\beta (a|s)}{\hat{A}}_{{\theta }_{old}}(s,a)\right)\end{array}$$

When training on policy, theoretically the policy for collecting data is same as the policy that we want to optimise. However, when rollout workers and optimisers are running in parallel asynchronously, the behaviour policy can get stale. TRPO considers this subtle difference: It labels the behaviour policy as ${\pi }_{{\theta }_{old}}(a|s)$ and thus the objective function becomes:

$$J(\theta )={\mathbb{E}}_{s\sim p^{{\pi }_{{\theta }_{old}}},a\sim {\pi }_{{\theta }_{old}}}\left(\frac{{\pi }_{\theta }(a|s)}{{\pi }_{{\theta }_{old}}(a|s)}{\hat{A}}_{{\theta }_{old}}(s,a)\right)$$

TRPO aims to maximise the objective function $J(\theta )$ subject to a trust region constraing which enforces the distance between old and new policies measured by KL-divergence to be small enough, within a parameter $\delta$:

$${\mathbb{E}}_{s\sim p^{{\pi }_{{\theta }_{old}}}}\left[D_{KL}({\pi }_{{\theta }_{old}}(.|s)||{\pi }_{\theta }(.|s)\right]\le \delta$$

References

• Papers With Code
• Trust Region Policy Optimization