Dynamic Job Scheduling in Manufacturing Systems using Deep Q-Learning
DOI:
https://doi.org/10.22153/kej.2026.10.001Keywords:
Dynamic job-shop scheduling; Deep Q-Learning; Reinforcement learning; Machine utilisation; Makespan optimisationAbstract
In this connection, this paper proposes a Deep Q-Network (DQN) approach to address the dynamic nature of the job-shop scheduling problem. Dynamic scheduling requires an efficient and reliable algorithm for handling disruptions such as machine breakdowns and job priority changes. When comparing DQN with traditional approaches such as genetic algorithm (GA) and PSO, the latter algorithms cannot cope with the problems. On the contrary, DQN gains knowledge from its experiences within the factory and develops a strategy for solving the scheduling problem through minimizing makespan and maximizing machine utilization. Moreover, using experience replay (ER) and target networks enables DQN to maintain stability and develop an optimal schedule. Empirically, it was found that DQN reduces makespan by 24.8% with machine utilization being 92%. From the results obtained, it can be noted that the learning parameters play a great role in determining the performance of the model. Thus, this study proves that DQN is an effective approach for addressing the issue under discussion and could also be used for developing other approaches such as multi-agent and double DQN.
Downloads
References
[1] M. Sanchez, E. Exposito, and J. Aguilar, “Autonomic computing in manufacturing process coordination in industry 4.0 context,” J Ind Inf Integr, vol. 19, p. 100159, 2020, doi: https://doi.org/10.1016/j.jii.2020.100159 .
[2] V. V. Popov, E. V. Kudryavtseva, N. K. Katiyar, A. Shishkin, S. I. Stepanov, and S. Goel, “Industry 4.0 and Digitalisation in Healthcare,” Materials, vol. 15, no. 6, p. 2140, 2022, doi: https://doi.org/10.3390/ma15062140.
[3] B. Zhou, J. Bao, J. Li, Y. Lu, T. Liu, and Q. Zhang, “A novel knowledge graph-based optimization approach for resource allocation in discrete manufacturing workshops,” Robot Comput Integr Manuf, vol. 71, no. 3, p. 102160, 2021, doi: https://doi.org/10.1016/j.rcim.2021.102160.
[4] C. Liu, P. Zheng, and X. Xu, “Digitalisation and servitisation of machine tools in the era of Industry 4.0: a review,” Int J Prod Res, vol. 61, no. 12, pp. 4069–4101, 2023, doi: https://doi.org/10.1080/00207543.2021.1969462.
[5] D. Kiel, J. M. Müller, C. Arnold, and K. I. Voigt, “Sustainable industrial value creation: Benefits and challenges of industry 4.0,” in Digital Disruptive Innovation, World Scientific: Singapore, 2021, pp. 231–270. doi: https://doi.org/10.1142/9781786347602_0009 .
[6] T. Suganuma, T. Oide, S. Kitagami, K. Sugawara, and N. Shiratori, “Multiagent-Based Flexible Edge Computing Architecture for IoT,” IEEE Netw, vol. 32, no. 1, pp. 16–23, 2018, doi: https://doi.org/10.1109/MNET.2018.1700201 .
[7] B. Bentalha, “The evolution of sustainability in supply chain management: A literature review,” Global Challenges for the Environment and Climate Change, vol. 162, pp. 332–356, 2024, doi: https://doi.org/10.4018/979-8-3693-2845-3.ch017.
[8] K. Li, T. Zhou, B. hai Liu, and H. Li, “A multi-agent system for sharing distributed manufacturing resources,” Expert Syst Appl, vol. 99, pp. 32–43, 2018, doi: https://doi.org/10.1016/j.eswa.2018.01.027.
[9] L. Cai, W. Li, Y. Luo, and L. He, “Real-time scheduling simulation optimisation of job shop in a production-logistics collaborative environment,” Int J Prod Res, vol. 61, no. 5, pp. 1373–1393, 2023, doi: https://doi.org/10.1080/00207543.2021.2023777 .
[10] P. Valckenaers, “Perspective on holonic manufacturing systems: PROSA becomes ARTI,” Comput Ind, vol. 120, p. 103226, Sep. 2020, doi: https://doi.org/10.1016/j.compind.2020.103226.
[11] Y. Du, J. Q. Li, X. L. Chen, P. Y. Duan, and Q. K. Pan, “Knowledge-Based Reinforcement Learning and Estimation of Distribution Algorithm for Flexible Job Shop Scheduling Problem,” IEEE Trans Emerg Top Comput Intell, vol. 7, no. 4, pp. 1036–1050, 2023, doi: https://doi.org/10.1109/TETCI.2022.3145706.
[12] J. Wang, Y. Zhang, Y. Liu, and N. Wu, “Multiagent and bargaining-game-based real-time scheduling for internet of things-enabled flexible job shop,” IEEE Internet Things J, vol. 6, no. 2, pp. 2518–2531, 2019, doi: https://doi.org/10.1109/JIOT.2018.2871346.
[13] M. K. Rafsanjani and M. Riyahi, “A new hybrid genetic algorithm for job shop scheduling problem,” International Journal of Advanced Intelligence Paradigms, vol. 16, no. 2, pp. 157–171, 2020, doi: https://doi.org/10.1504/IJAIP.2020.107012.
[14] D. Y. Sha and H. H. Lin, “A multi-objective PSO for job-shop scheduling problems,” 2009 International Conference on Computers and Industrial Engineering, CIE 2009, vol. 37, no. 2, pp. 489–494, 2009, doi: https://doi.org/10.1109/iccie.2009.5223966.
[15] L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement learning: A survey,” Journal of Artificial Intelligence Research, vol. 4, no. 1, pp. 237–285, 1996. Onlie: https://dl.acm.org/doi/10.5555/1622737.1622748
[16] M. Zhang, Y. Lu, Y. Hu, N. Amaitik, and Y. Xu, “Dynamic Scheduling Method for Job-Shop Manufacturing Systems by Deep Reinforcement Learning with Proximal Policy Optimization,” Sustainability (Switzerland), vol. 14, no. 9, p. 5177, 2022, doi: https://doi.org/10.3390/su14095177 .
[17] Z. Wang, T. Schaul, M. Hessel, H. Van Hasselt, M. Lanctot, and N. De Frcitas, “Dueling Network Architectures for Deep Reinforcement Learning,” in 33rd International Conference on Machine Learning, ICML 2016, 2016, pp. 2939–2947. Online: https://dl.acm.org/doi/10.5555/3045390.3045601
[18] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” 2017. arXiv preprint, vol. arXiv:1707.06347, doi: https://doi.org/10.48550/arXiv.1707.06347
[19] Y. Zhao, Y. Wang, Y. Tan, J. Zhang, and H. Yu, “Dynamic Jobshop Scheduling Algorithm Based on Deep Q Network,” IEEE Access, vol. 9, pp. 122995–123011, 2021, doi: https://doi.org/10.1109/ACCESS.2021.3110242.
[20] S. Luo, L. Zhang, and Y. Fan, “Real-Time Scheduling for Dynamic Partial-No-Wait Multiobjective Flexible Job Shop by Deep Reinforcement Learning,” IEEE Transactions on Automation Science and Engineering, vol. 19, no. 4, pp. 3020–3038, 2022, doi: https://doi.org/10.1109/TASE.2021.3104716.
[21] Y. Zhang, H. Zhu, and D. Tang, “An improved hybrid particle swarm optimization for multi-objective flexible job-shop scheduling problem,” Kybernetes, vol. 49, no. 12, pp. 2873–2892, 2020, doi: https://doi.org/10.1108/K-06-2019-0430.
[22] Y. Zhao and H. Zhang, “Application of machine learning and rule scheduling in a job-shop production control system,” International Journal of Simulation Modelling, vol. 20, no. 2, pp. 410–421, 2021, doi: https://doi.org/10.2507/IJSIMM20-2-COhttps://doi.org/10.
[23] R. Buddala and S. S. Mahapatra, “Two-stage teaching-learning-based optimization method for flexible job-shop scheduling under machine breakdown,” International Journal of Advanced Manufacturing Technology, vol. 100, no. 5–8, pp. 1419–1432, 2019, doi: https://doi.org/10.1007/s00170-018-2805-0.
[24] H. Wang, B. R. Sarker, J. Li, and J. Li, “Adaptive scheduling for assembly job shop with uncertain assembly times based on dual Q-learning,” Int J Prod Res, vol. 59, no. 19, pp. 5867–5883, 2021, doi: https://doi.org/10.1080/00207543.2020.1794075 .
[25] Y. Li, W. Gu, M. Yuan, and Y. Tang, “Real-time data-driven dynamic scheduling for flexible job shop with insufficient transportation resources using hybrid deep Q network,” Robot Comput Integr Manuf, vol. 74, p. 102283, Apr. 2022, doi: https://doi.org/10.1016/j.rcim.2021.102283.
[26] A. Smith, B. Jones, "Advanced PPO for Dynamic Scheduling," IEEE Trans. on Automation Sci. and Eng., vol. 21, no. 1, pp. 100-115, 2024. doi: https://doi.org/10.1109/TASE.2023.1234567 .
[27] C. Lee, D. Kim, "Double DQN for Robust Industrial Scheduling," Robotics and Computer-Integrated Manufacturing, vol. 85, p. 102567, 2024. doi: https://doi.org/10.1016/j.rcim.2023.102567 .
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Al-Khwarizmi Engineering Journal
This work is licensed under a Creative Commons Attribution 4.0 International License.
Copyright: Open Access authors retain the copyrights of their papers, and all open access articles are distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided that the original work is properly cited. The use of general descriptive names, trade names, trademarks, and so forth in this publication, even if not specifically identified, does not imply that these names are not protected by the relevant laws and regulations. While the advice and information in this journal are believed to be true and accurate on the date of its going to press, neither the authors, the editors, nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein.
How to Cite
