Abstract A parallel hybrid vehicle was studied to establish the demand power and power system model of the whole vehicle and proposed an energy distribution strategy based on improved Deep Reinforcement Learning (DRL). The DRB-TD3 algorithm was proposed to improve the sampling efficiency of the original algorithm by improving the Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm in DRL and introduced dual replay buffers. A rule-based constraint controller was designed and embedded into the algorithm structure to eliminate unreasonable torque allocation. The performance of the Dynamic Planning (DP)-based energy distribution strategy was used as a benchmark for simulation experiment under UDDS driving conditions. The experimental results show that the DRB-TD3 algorithm has the best convergence performance compared with the Deep Deterministic Policy Gradient (DDPG) algorithm and the conventional TD3 algorithm, with 61.2% and 31.6% improvement in convergence efficiency, respectively. The proposed energy distribution strategy reduces the average fuel consumption by 3.3% and 2.3% compared with the DDPG-and TD3-based energy distribution strategies, respectively. The fuel performance reaches 95.2% of DP-based, which with the best fuel economy, and the battery state of charge (SOC) can be maintained at a better level, which helps to extend the battery life.
WU Zhong-qiang,MA Bo-yan. Research on HEV Energy Distribution Strategy Based on Improved Deep Reinforcement Learning[J]. Acta Metrologica Sinica, 2023, 44(12): 1863-1871.
Lillicrap T P, Hunt J J, Pritzel A, et al. Continuous control with deep reinforcement learning ［J］. Computer Science, 2015, 928: 136-141.
Wu Z Q, Shang M Y, Shen D D, et al. Estimation of SOC of Li-ion Battery in Pure Electric Vehicle by BSA-ＲELM ［J］. Acta Metrologica Sinica, 2019, 40 (4): 693-699.
Wu Z Q, Wang G Y, Xie Z K, et al. Joint estimation of the capacity and SOC of lithium battery based on WOA-UKF algorithm ［J］. Acta Metrologica Sinica, 2022, 43 (5): 649-656.
［4］
Ahmadi S, Bathaee S, Hosseinpour A H. Improving fuel economy and performance of a fuel-cell hybrid electric vehicle (fuel-cell, battery, and ultra-capacitor) using optimized energy management strategy ［J］. Energy Conversion and Management, 2018, 160: 74-84.
［6］
Zhang S, Xiong R. Adaptive energy management of a plug-in hybrid electric vehicle based on driving pattern recognition and dynamic programming ［J］. Applied Energy, 2015, 155: 68-78.
Ni R Y, Zhao Z G, Gao X J. Development and optimization of energy management strategy for a new plug-in hybrid electric car ［J］. Journal of Tongji University (Natural Science), 2019, 47 (S1): 104-109.
Lin X Y, Sun D Y, Qin D T, et al. Development of power-balancing global optimization control strategy for a series-parallel hybrid electric city bus ［J］. China Mechanical Engineering, 2011, 22 (18): 2259-2263.
［11］
Liu T, Tan W, Tang X, et al. Driving conditions-driven energy management for hybrid electric vehicles: a review ［J］. Renewable and Sustainable Energy Reviews, 2021, 151(C), DOI: 10.1016/j.rser.2021.111521.
Hu X S, Chen K P, Tang X L, et al. Machine learning velocity prediction-based energy management of parallel hybrid electric vehicle ［J］. Journal of Mechanical Engineering, 2020, 56 (16): 181-192.
［13］
Liu Y G, Li J, Gao J, et al. Prediction of vehicle driving conditions with incorporation of stochastic forecasting and machine learning and a case study in energy management of plug-in hybrid electric vehicles ［J］. Mechanical Systems and Signal Processing, 2021, 158: 107765.
［15］
Chen Z, Hu H, Wu Y, et al. Energy management for a power-split plug-in hybrid electric vehicle based on reinforcement learning ［J］. Applied Sciences, 2018, 8 (12),DOI:10.3390/app8122494.
［17］
Du G, Zou Y, Zhang X, et al. Deep reinforcement learning based energy management for a hybrid electric vehicle ［J］. Energy, 2020, 201(C),DOI: 10.1016/j.energy.2020.117591.
Wu T Z, Wang Y Y, Xu Y S, et al. Energy optimal control strategy of HEV with PMP algorithm ［J］. Acta Automatica Sinica, 2018, 44 (11): 2092-2102.
Zhang F Q, Hu X S, Xu K H, et al. Current status and prospects for model predictive energy management in hybrid electric vehicles ［J］. Journal of Mechanical Engineering, 2019, 55 (10): 86-108.
Zhang H, Fan Q H, Wang W, et al. Reinforcement learning based energy management strategy for hybrid electric vehicles using multi-mode combustion ［J］. Automotive Engineering, 2021, 43 (5): 683-691.
［21］
Fujimoto S, Hoof H V, Meger D. Addressing function approximation error in actor-critic methods［C］//Proceedings of the 35th International Conference on Machine Learning. Stockholm, Sweden, 2018.
［22］
Volodymyr M, Koray K, David S, et al. Human-level control through deep reinforcement learning ［J］. Nature, 2015, 518 (7540): 529-533.
Huang H, Hu Z Q, Wang L H, et al. Intelligent traffic signal control algorithm based on Sumtree DDPG ［J］. Journal of Beijing University of Posts and Telecommunications, 2021, 44 (1): 97-103.
Niu L M, Yang H Y, Zhou Y Z, et al. Hybrid electric vehicle integrated control strategy based on multi-agent ［J］. Journal of Mechanical Engineering, 2019, 55 (12): 168-177+188.
［5］
Gao Y, Ehsani M. Design and control methodology of plug-in hybrid electric vehicles ［J］. IEEE Transactions on Industrial Electronics, 2010, 57 (2): 633-640.
He W L, Huang Y. Real-time energy optimization of hybrid electric vehicle in connected environment based on deep reinforcement learning ［J］. IFAC Papers OnLine, 2021, 54 (10): 176-181.
Shi Q, Qiu D Y, Wu B, et al. DCR and applications based on PSO-SVM algorithm ［J］. China Mechanical Engineering, 2018, 29 (15): 1875-1883.
［16］
Sun H C, Fu Z M, Tao F Z, et al. Data-driven reinforcement-learning-based hierarchical energy management strategy for fuel cell/battery/ultracapacitor hybrid electric vehicles ［J］. Journal of Power Sources, 2020, 455(15): 227964.1-227964.12.
2023, Vol. 44 
