基于改进深度强化学习的HEV能量分配策略研究

doi:10.3969/j.issn.1000-1158.2023.12.12

Abstract
Figure/Table
References (24)
Related Citation (5)

Download: PDF (821 KB) HTML (1 KB)
Export: BibTeX | EndNote (RIS)

Abstract A parallel hybrid vehicle was studied to establish the demand power and power system model of the whole vehicle and proposed an energy distribution strategy based on improved Deep Reinforcement Learning (DRL). The DRB-TD3 algorithm was proposed to improve the sampling efficiency of the original algorithm by improving the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm in DRL and introduced dual replay buffers. A rule-based constraint controller was designed and embedded into the algorithm structure to eliminate unreasonable torque allocation. The performance of the Dynamic Planning (DP)-based energy distribution strategy was used as a benchmark for simulation experiment under UDDS driving conditions. The experimental results show that the DRB-TD3 algorithm has the best convergence performance compared with the Deep Deterministic Policy Gradient (DDPG) algorithm and the conventional TD3 algorithm, with 61.2% and 31.6% improvement in convergence efficiency, respectively. The proposed energy distribution strategy reduces the average fuel consumption by 3.3% and 2.3% compared with the DDPG-and TD3-based energy distribution strategies, respectively. The fuel performance reaches 95.2% of DP-based, which with the best fuel economy, and the battery state of charge (SOC) can be maintained at a better level, which helps to extend the battery life.

Key words： parallel hybrid electric vehicle energy distribution strategy deep reinforcement learning TD3 algorithm SOC

Received: 24 October 2022 Published: 27 December 2023

PACS:

TB971

	Service

	E-mail this article
	Add to my bookshelf
	Add to citation manager
	E-mail Alert
	RSS
	Articles by authors
	WU Zhong-qiang
	MA Bo-yan

Cite this article:

WU Zhong-qiang,MA Bo-yan. Research on HEV Energy Distribution Strategy Based on Improved Deep Reinforcement Learning[J]. Acta Metrologica Sinica, 2023, 44(12): 1863-1871.

URL:

http://jlxb.china-csm.org:81/Jwk_jlxb/EN/10.3969/j.issn.1000-1158.2023.12.12 OR http://jlxb.china-csm.org:81/Jwk_jlxb/EN/Y2023/V44/I12/1863

［14］	石琴, 仇多洋, 吴冰, 等. 基于粒子群优化支持向量机算法的行驶工况识别及应用［J］. 中国机械工程, 2018, 29 (15): 1875-1883.
［1］	吴忠强, 尚梦瑶, 申丹丹, 等. 基于 BSA-ＲELM的纯电动汽车锂离子电池SOC估计［J］. 计量学报, 2019, 40 (4): 693-699.
［8］	倪润宇, 赵治国, 高晓杰. 新型插电式混合动力轿车能量管理策略优化［J］. 同济大学学报 (自然科学版), 2019, 47 (S1): 104-109.
［10］	张风奇, 胡晓松, 许康辉, 等. 混合动力汽车模型预测能量管理研究现状与展望［J］. 机械工程学报, 2019, 55 (10): 86-108.
［20］	Lillicrap T P, Hunt J J, Pritzel A, et al. Continuous control with deep reinforcement learning ［J］. Computer Science, 2015, 928: 136-141.
	Wu Z Q, Shang M Y, Shen D D, et al. Estimation of SOC of Li-ion Battery in Pure Electric Vehicle by BSA-ＲELM ［J］. Acta Metrologica Sinica, 2019, 40 (4): 693-699.
	Wu Z Q, Wang G Y, Xie Z K, et al. Joint estimation of the capacity and SOC of lithium battery based on WOA-UKF algorithm ［J］. Acta Metrologica Sinica, 2022, 43 (5): 649-656.
［4］	Ahmadi S, Bathaee S, Hosseinpour A H. Improving fuel economy and performance of a fuel-cell hybrid electric vehicle (fuel-cell, battery, and ultra-capacitor) using optimized energy management strategy ［J］. Energy Conversion and Management, 2018, 160: 74-84.
［6］	Zhang S, Xiong R. Adaptive energy management of a plug-in hybrid electric vehicle based on driving pattern recognition and dynamic programming ［J］. Applied Energy, 2015, 155: 68-78.
	Ni R Y, Zhao Z G, Gao X J. Development and optimization of energy management strategy for a new plug-in hybrid electric car ［J］. Journal of Tongji University (Natural Science), 2019, 47 (S1): 104-109.
［7］	林歆悠, 孙冬野, 秦大同, 等.混联式混合动力客车全局优化控制策略研究［J］. 中国机械工程, 2011, 22 (18): 2259-2263.
	Lin X Y, Sun D Y, Qin D T, et al. Development of power-balancing global optimization control strategy for a series-parallel hybrid electric city bus ［J］. China Mechanical Engineering, 2011, 22 (18): 2259-2263.
［11］	Liu T, Tan W, Tang X, et al. Driving conditions-driven energy management for hybrid electric vehicles: a review ［J］. Renewable and Sustainable Energy Reviews, 2021, 151(C), DOI: 10.1016/j.rser.2021.111521.
	Hu X S, Chen K P, Tang X L, et al. Machine learning velocity prediction-based energy management of parallel hybrid electric vehicle ［J］. Journal of Mechanical Engineering, 2020, 56 (16): 181-192.
［13］	Liu Y G, Li J, Gao J, et al. Prediction of vehicle driving conditions with incorporation of stochastic forecasting and machine learning and a case study in energy management of plug-in hybrid electric vehicles ［J］. Mechanical Systems and Signal Processing, 2021, 158: 107765.
［15］	Chen Z, Hu H, Wu Y, et al. Energy management for a power-split plug-in hybrid electric vehicle based on reinforcement learning ［J］. Applied Sciences, 2018, 8 (12),DOI:10.3390/app8122494.
［17］	Du G, Zou Y, Zhang X, et al. Deep reinforcement learning based energy management for a hybrid electric vehicle ［J］. Energy, 2020, 201(C),DOI: 10.1016/j.energy.2020.117591.
［2］	吴忠强, 王国勇, 谢宗奎, 等. 基于WOA-UKF算法的锂电池容量与SOC联合估计［J］. 计量学报, 2022, 43 (5): 649-656.
［9］	Guo J, He H, Peng J, et al. A novel MPC-based adaptive energy management strategy in plug-in hybrid electric vehicles ［J］. Energy, 2019, 175: 378-392.
［24］	牛礼民, 杨洪源, 周亚洲, 等. 混合动力汽车动力总成多智能体集成控制策略［J］. 机械工程学报, 2019, 55 (12): 168-177+188.
［3］	Wu T Z, Wang Y Y, Xu Y S, et al. Energy optimal control strategy of HEV with PMP algorithm ［J］. Acta Automatica Sinica, 2018, 44 (11): 2092-2102.
	Zhang F Q, Hu X S, Xu K H, et al. Current status and prospects for model predictive energy management in hybrid electric vehicles ［J］. Journal of Mechanical Engineering, 2019, 55 (10): 86-108.
［18］	张昊, 范钦灏, 王巍, 等. 基于强化学习的多燃烧模式混合动力能量管理策略［J］. 汽车工程, 2021, 43 (5): 683-691.
	Zhang H, Fan Q H, Wang W, et al. Reinforcement learning based energy management strategy for hybrid electric vehicles using multi-mode combustion ［J］. Automotive Engineering, 2021, 43 (5): 683-691.
［21］	Fujimoto S, Hoof H V, Meger D. Addressing function approximation error in actor-critic methods［C］//Proceedings of the 35th International Conference on Machine Learning. Stockholm, Sweden, 2018.
［22］	Volodymyr M, Koray K, David S, et al. Human-level control through deep reinforcement learning ［J］. Nature, 2015, 518 (7540): 529-533.
	Huang H, Hu Z Q, Wang L H, et al. Intelligent traffic signal control algorithm based on Sumtree DDPG ［J］. Journal of Beijing University of Posts and Telecommunications, 2021, 44 (1): 97-103.
	Niu L M, Yang H Y, Zhou Y Z, et al. Hybrid electric vehicle integrated control strategy based on multi-agent ［J］. Journal of Mechanical Engineering, 2019, 55 (12): 168-177+188.
［5］	Gao Y, Ehsani M. Design and control methodology of plug-in hybrid electric vehicles ［J］. IEEE Transactions on Industrial Electronics, 2010, 57 (2): 633-640.
［12］	胡晓松, 陈科坪, 唐小林, 等. 基于机器学习速度预测的并联混合动力车辆能量管理研究［J］. 机械工程学报, 2020, 56 (16): 181-192.
［19］	He W L, Huang Y. Real-time energy optimization of hybrid electric vehicle in connected environment based on deep reinforcement learning ［J］. IFAC Papers OnLine, 2021, 54 (10): 176-181.
	Shi Q, Qiu D Y, Wu B, et al. DCR and applications based on PSO-SVM algorithm ［J］. China Mechanical Engineering, 2018, 29 (15): 1875-1883.
［16］	Sun H C, Fu Z M, Tao F Z, et al. Data-driven reinforcement-learning-based hierarchical energy management strategy for fuel cell/battery/ultracapacitor hybrid electric vehicles ［J］. Journal of Power Sources, 2020, 455(15): 227964.1-227964.12.
［23］	黄浩, 胡智群, 王鲁晗, 等. 基于Sumtree DDPG的智能交通信号控制算法［J］. 北京邮电大学学报, 2021, 44 (1): 97-103.