Bibliographic Details
| Title: |
On Many-Actions Policy Gradient |
| Authors: |
Nauman, Michal, Cygan, Marek |
| Publication Year: |
2022 |
| Collection: |
ArXiv.org (Cornell University Library) |
| Subject Terms: |
Computer Science - Machine Learning |
| Description: |
We study the variance of stochastic policy gradients (SPGs) with many action samples per state. We derive a many-actions optimality condition, which determines when many-actions SPG yields lower variance as compared to a single-action agent with proportionally extended trajectory. We propose Model-Based Many-Actions (MBMA), an approach leveraging dynamics models for many-actions sampling in the context of SPG. MBMA addresses issues associated with existing implementations of many-actions SPG and yields lower bias and comparable variance to SPG estimated from states in model-simulated rollouts. We find that MBMA bias and variance structure matches that predicted by theory. As a result, MBMA achieves improved sample efficiency and higher returns on a range of continuous action environments as compared to model-free, many-actions, and model-based on-policy SPG baselines. ; Comment: ICML 2023 |
| Document Type: |
text |
| Language: |
unknown |
| Relation: |
http://arxiv.org/abs/2210.13011 |
| Availability: |
http://arxiv.org/abs/2210.13011 |
| Accession Number: |
edsbas.74D1A26B |
| Database: |
BASE |
