基于改进深度强化学习的HEV能量分配策略研究

吴忠强; 马博岩

doi:10.3969/j.issn.1000-1158.2023.12.12

PDF(821 KB)

计量学报 ›› 2023, Vol. 44 ›› Issue (12) : 1863-1871. DOI: 10.3969/j.issn.1000-1158.2023.12.12

电磁学计量

基于改进深度强化学习的HEV能量分配策略研究

吴忠强,马博岩

作者信息 +

Research on HEV Energy Distribution Strategy Based on Improved Deep Reinforcement Learning

WU Zhong-qiang,MA Bo-yan

Author information +

文章历史 +

摘要

以并联式混合动力汽车(HEV)为研究对象,建立整车需求功率及动力系统模型,提出一种基于改进深度强化学习(DRL)的能量分配策略。通过改进DRL中的双延迟深度确定性策略梯度(TD3)算法,引入双重回放缓冲区,提出DRB-TD3算法以提升原算法的采样效率。设计了基于规则的约束控制器并嵌入到DRL结构中,以消除不合理的转矩分配。在UDDS行驶工况下,以基于动态规划(DP)的能量分配策略性能作为基准进行仿真实验。实验结果表明,与深度确定性策略梯度(DDPG)算法以及传统TD3算法相比,DRB-TD3算法收敛性能最佳,收敛效率分别提高了61.2%和31.6%;所提出的能量分配策略相比于基于DDPG和基于TD3的能量分配策略,平均燃油消耗分别降低了3.3%和2.3%,燃油经济性达到基于DP的95.2%,效果最佳,且电池荷电状态(SOC)能够保持在一个较好的水平,有助于延长电池的使用寿命。

Abstract

A parallel hybrid vehicle was studied to establish the demand power and power system model of the whole vehicle and proposed an energy distribution strategy based on improved Deep Reinforcement Learning (DRL). The DRB-TD3 algorithm was proposed to improve the sampling efficiency of the original algorithm by improving the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm in DRL and introduced dual replay buffers. A rule-based constraint controller was designed and embedded into the algorithm structure to eliminate unreasonable torque allocation. The performance of the Dynamic Planning (DP)-based energy distribution strategy was used as a benchmark for simulation experiment under UDDS driving conditions. The experimental results show that the DRB-TD3 algorithm has the best convergence performance compared with the Deep Deterministic Policy Gradient (DDPG) algorithm and the conventional TD3 algorithm, with 61.2% and 31.6% improvement in convergence efficiency, respectively. The proposed energy distribution strategy reduces the average fuel consumption by 3.3% and 2.3% compared with the DDPG-and TD3-based energy distribution strategies, respectively. The fuel performance reaches 95.2% of DP-based, which with the best fuel economy, and the battery state of charge (SOC) can be maintained at a better level, which helps to extend the battery life.

导出引用

吴忠强,马博岩. 基于改进深度强化学习的HEV能量分配策略研究[J]. 计量学报. 2023, 44(12): 1863-1871 https://doi.org/10.3969/j.issn.1000-1158.2023.12.12

WU Zhong-qiang,MA Bo-yan. Research on HEV Energy Distribution Strategy Based on Improved Deep Reinforcement Learning[J]. Acta Metrologica Sinica. 2023, 44(12): 1863-1871 https://doi.org/10.3969/j.issn.1000-1158.2023.12.12

中图分类号： TB971

参考文献

［1］吴忠强, 尚梦瑶, 申丹丹, 等. 基于 BSA-ＲELM的纯电动汽车锂离子电池SOC估计［J］. 计量学报, 2019, 40 (4): 693-699.
Wu Z Q, Shang M Y, Shen D D, et al. Estimation of SOC of Li-ion Battery in Pure Electric Vehicle by BSA-ＲELM ［J］. Acta Metrologica Sinica, 2019, 40 (4): 693-699.
［2］吴忠强, 王国勇, 谢宗奎, 等. 基于WOA-UKF算法的锂电池容量与SOC联合估计［J］. 计量学报, 2022, 43 (5): 649-656.
Wu Z Q, Wang G Y, Xie Z K, et al. Joint estimation of the capacity and SOC of lithium battery based on WOA-UKF algorithm ［J］. Acta Metrologica Sinica, 2022, 43 (5): 649-656.
［3］Wu T Z, Wang Y Y, Xu Y S, et al. Energy optimal control strategy of HEV with PMP algorithm ［J］. Acta Automatica Sinica, 2018, 44 (11): 2092-2102.
［4］Ahmadi S, Bathaee S, Hosseinpour A H. Improving fuel economy and performance of a fuel-cell hybrid electric vehicle (fuel-cell, battery, and ultra-capacitor) using optimized energy management strategy ［J］. Energy Conversion and Management, 2018, 160: 74-84.
［5］Gao Y, Ehsani M. Design and control methodology of plug-in hybrid electric vehicles ［J］. IEEE Transactions on Industrial Electronics, 2010, 57 (2): 633-640.
［6］Zhang S, Xiong R. Adaptive energy management of a plug-in hybrid electric vehicle based on driving pattern recognition and dynamic programming ［J］. Applied Energy, 2015, 155: 68-78.
［7］林歆悠, 孙冬野, 秦大同, 等.混联式混合动力客车全局优化控制策略研究［J］. 中国机械工程, 2011, 22 (18): 2259-2263.
Lin X Y, Sun D Y, Qin D T, et al. Development of power-balancing global optimization control strategy for a series-parallel hybrid electric city bus ［J］. China Mechanical Engineering, 2011, 22 (18): 2259-2263.
［8］倪润宇, 赵治国, 高晓杰. 新型插电式混合动力轿车能量管理策略优化［J］. 同济大学学报 (自然科学版), 2019, 47 (S1): 104-109.
Ni R Y, Zhao Z G, Gao X J. Development and optimization of energy management strategy for a new plug-in hybrid electric car ［J］. Journal of Tongji University (Natural Science), 2019, 47 (S1): 104-109.
［9］Guo J, He H, Peng J, et al. A novel MPC-based adaptive energy management strategy in plug-in hybrid electric vehicles ［J］. Energy, 2019, 175: 378-392.
［10］张风奇, 胡晓松, 许康辉, 等. 混合动力汽车模型预测能量管理研究现状与展望［J］. 机械工程学报, 2019, 55 (10): 86-108.
Zhang F Q, Hu X S, Xu K H, et al. Current status and prospects for model predictive energy management in hybrid electric vehicles ［J］. Journal of Mechanical Engineering, 2019, 55 (10): 86-108.
［11］Liu T, Tan W, Tang X, et al. Driving conditions-driven energy management for hybrid electric vehicles: a review ［J］. Renewable and Sustainable Energy Reviews, 2021, 151(C), DOI: 10.1016/j.rser.2021.111521.
［12］胡晓松, 陈科坪, 唐小林, 等. 基于机器学习速度预测的并联混合动力车辆能量管理研究［J］. 机械工程学报, 2020, 56 (16): 181-192.
Hu X S, Chen K P, Tang X L, et al. Machine learning velocity prediction-based energy management of parallel hybrid electric vehicle ［J］. Journal of Mechanical Engineering, 2020, 56 (16): 181-192.
［13］Liu Y G, Li J, Gao J, et al. Prediction of vehicle driving conditions with incorporation of stochastic forecasting and machine learning and a case study in energy management of plug-in hybrid electric vehicles ［J］. Mechanical Systems and Signal Processing, 2021, 158: 107765.
［14］石琴, 仇多洋, 吴冰, 等. 基于粒子群优化支持向量机算法的行驶工况识别及应用［J］. 中国机械工程, 2018, 29 (15): 1875-1883.
Shi Q, Qiu D Y, Wu B, et al. DCR and applications based on PSO-SVM algorithm ［J］. China Mechanical Engineering, 2018, 29 (15): 1875-1883.
［15］Chen Z, Hu H, Wu Y, et al. Energy management for a power-split plug-in hybrid electric vehicle based on reinforcement learning ［J］. Applied Sciences, 2018, 8 (12),DOI:10.3390/app8122494.
［16］Sun H C, Fu Z M, Tao F Z, et al. Data-driven reinforcement-learning-based hierarchical energy management strategy for fuel cell/battery/ultracapacitor hybrid electric vehicles ［J］. Journal of Power Sources, 2020, 455(15): 227964.1-227964.12.
［17］Du G, Zou Y, Zhang X, et al. Deep reinforcement learning based energy management for a hybrid electric vehicle ［J］. Energy, 2020, 201(C),DOI: 10.1016/j.energy.2020.117591.
［18］张昊, 范钦灏, 王巍, 等. 基于强化学习的多燃烧模式混合动力能量管理策略［J］. 汽车工程, 2021, 43 (5): 683-691.
Zhang H, Fan Q H, Wang W, et al. Reinforcement learning based energy management strategy for hybrid electric vehicles using multi-mode combustion ［J］. Automotive Engineering, 2021, 43 (5): 683-691.
［19］He W L, Huang Y. Real-time energy optimization of hybrid electric vehicle in connected environment based on deep reinforcement learning ［J］. IFAC Papers OnLine, 2021, 54 (10): 176-181.
［20］Lillicrap T P, Hunt J J, Pritzel A, et al. Continuous control with deep reinforcement learning ［J］. Computer Science, 2015, 928: 136-141.
［21］Fujimoto S, Hoof H V, Meger D. Addressing function approximation error in actor-critic methods［C］//Proceedings of the 35th International Conference on Machine Learning. Stockholm, Sweden, 2018.
［22］Volodymyr M, Koray K, David S, et al. Human-level control through deep reinforcement learning ［J］. Nature, 2015, 518 (7540): 529-533.
［23］黄浩, 胡智群, 王鲁晗, 等. 基于Sumtree DDPG的智能交通信号控制算法［J］. 北京邮电大学学报, 2021, 44 (1): 97-103.
Huang H, Hu Z Q, Wang L H, et al. Intelligent traffic signal control algorithm based on Sumtree DDPG ［J］. Journal of Beijing University of Posts and Telecommunications, 2021, 44 (1): 97-103.
［24］牛礼民, 杨洪源, 周亚洲, 等. 混合动力汽车动力总成多智能体集成控制策略［J］. 机械工程学报, 2019, 55 (12): 168-177+188.
Niu L M, Yang H Y, Zhou Y Z, et al. Hybrid electric vehicle integrated control strategy based on multi-agent ［J］. Journal of Mechanical Engineering, 2019, 55 (12): 168-177+188.