论文标题
时间延迟在SIM2REAL转移中的增强学习系统中的作用
The Role of Time Delay in Sim2real Transfer of Reinforcement Learning for Cyber-Physical Systems
论文作者
论文摘要
本文分析了具有分数延迟的增强学习（RL）网络物理系统中对现实差距的模拟（即，采样期的延迟延迟）。对分数延迟的考虑对所考虑的网络物理系统的性质具有重要意义。具有延迟的系统是非马克维亚的，并且需要扩展系统状态向量以使Markovian系统。我们表明，当延迟出现在输出中时，这是不可能的，并且问题始终是非马克维亚人。基于此分析，提出了一种采样方案，该方案会导致有效的RL训练和在逼真的多旋动无人驾驶飞机模拟中表现良好的试剂。我们证明，所得代理不会产生过多的振荡，而不考虑模型中时间延迟的RL代理并非如此。
This paper analyzes the simulation to reality gap in reinforcement learning (RL) cyber-physical systems with fractional delays (i.e. delays that are non-integer multiple of the sampling period). The consideration of fractional delay has important implications on the nature of the cyber-physical system considered. Systems with delays are non-Markovian, and the system state vector needs to be extended to make the system Markovian. We show that this is not possible when the delay is in the output, and the problem would always be non-Markovian. Based on this analysis, a sampling scheme is proposed that results in efficient RL training and agents that perform well in realistic multirotor unmanned aerial vehicle simulations. We demonstrate that the resultant agents do not produce excessive oscillations, which is not the case with RL agents that do not consider time delay in the model.
