Reinforcement Learning with Model-Based Approaches for Dynamic Resource Allocation in a Tandem Queue

Thomas Tournaire; Jeanne Barthelemy; Hind Castel-Taleb; Emmanuel Hyon

doi:10.1007/978-3-030-91825-5_15

Reinforcement Learning with Model-Based Approaches for Dynamic Resource Allocation in a Tandem Queue

Thomas Tournaire
, Jeanne Barthelemy
, Hind Castel-Taleb
, Emmanuel Hyon

Bell Labs
CNRS UMR 5157 SAMOVAR
Sorbonne Université
Université Paris-Nanterre

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We consider three-tier network architecture modeled with two physical nodes in tandem where an autonomous agent controls the number of active resources on each node. We analyse the learning of auto-scaling strategies in order to optimise both performance and energy consumption of the whole system. We compare several model-based reinforcement learning with model-free Q-learning algorithm. The relevance of these algorithms is to faster update Q-value function with an additional planning phase allowed by approximated model of the dynamics of the environment. Secondly, we consider the same tandem queue scenario with MMPP (Markov modulated Poisson process) for arrivals. In this context, the arrival rate is varying over time and this information is hidden to the agent. Our goal is to assess the robustness of such model-based reinforcement learning algorithms in this particular scenario.

Original language	English
Title of host publication	Performance Engineering and Stochastic Modeling - 17th European Workshop, EPEW 2021, and 26th International Conference, ASMTA 2021, Proceedings
Editors	Paolo Ballarini, Hind Castel, Ioannis Dimitriou, Mauro Iacono, Tuan Phung-Duc, Joris Walraevens
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	243-263
Number of pages	21
ISBN (Print)	9783030918248
DOIs	https://doi.org/10.1007/978-3-030-91825-5_15
Publication status	Published - 1 Jan 2021
Externally published	Yes
Event	17th European Performance Engineering Workshop, EPEW 2021, and the 26th International Conference on Analytical and Stochastic Modelling Techniques and Applications, ASMTA 2021 - Virtual, Online Duration: 13 Dec 2021 → 14 Dec 2021

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13104 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	17th European Performance Engineering Workshop, EPEW 2021, and the 26th International Conference on Analytical and Stochastic Modelling Techniques and Applications, ASMTA 2021
City	Virtual, Online
Period	13/12/21 → 14/12/21

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

SDG 7 Affordable and Clean Energy

Keywords

Cloud
Energy saving
Model-based reinforcement learning
QoS guarantee
Tandem queues

Access to Document

10.1007/978-3-030-91825-5_15

Cite this

Tournaire, T., Barthelemy, J., Castel-Taleb, H., & Hyon, E. (2021). Reinforcement Learning with Model-Based Approaches for Dynamic Resource Allocation in a Tandem Queue. In P. Ballarini, H. Castel, I. Dimitriou, M. Iacono, T. Phung-Duc, & J. Walraevens (Eds.), Performance Engineering and Stochastic Modeling - 17th European Workshop, EPEW 2021, and 26th International Conference, ASMTA 2021, Proceedings (pp. 243-263). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13104 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-91825-5_15

Tournaire, Thomas ; Barthelemy, Jeanne ; Castel-Taleb, Hind et al. / Reinforcement Learning with Model-Based Approaches for Dynamic Resource Allocation in a Tandem Queue. Performance Engineering and Stochastic Modeling - 17th European Workshop, EPEW 2021, and 26th International Conference, ASMTA 2021, Proceedings. editor / Paolo Ballarini ; Hind Castel ; Ioannis Dimitriou ; Mauro Iacono ; Tuan Phung-Duc ; Joris Walraevens. Springer Science and Business Media Deutschland GmbH, 2021. pp. 243-263 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{5b971acec10d4568ba1eed64ddae0223,

title = "Reinforcement Learning with Model-Based Approaches for Dynamic Resource Allocation in a Tandem Queue",

abstract = "We consider three-tier network architecture modeled with two physical nodes in tandem where an autonomous agent controls the number of active resources on each node. We analyse the learning of auto-scaling strategies in order to optimise both performance and energy consumption of the whole system. We compare several model-based reinforcement learning with model-free Q-learning algorithm. The relevance of these algorithms is to faster update Q-value function with an additional planning phase allowed by approximated model of the dynamics of the environment. Secondly, we consider the same tandem queue scenario with MMPP (Markov modulated Poisson process) for arrivals. In this context, the arrival rate is varying over time and this information is hidden to the agent. Our goal is to assess the robustness of such model-based reinforcement learning algorithms in this particular scenario.",

keywords = "Cloud, Energy saving, Model-based reinforcement learning, QoS guarantee, Tandem queues",

author = "Thomas Tournaire and Jeanne Barthelemy and Hind Castel-Taleb and Emmanuel Hyon",

note = "Publisher Copyright: {\textcopyright} 2021, Springer Nature Switzerland AG.; 17th European Performance Engineering Workshop, EPEW 2021, and the 26th International Conference on Analytical and Stochastic Modelling Techniques and Applications, ASMTA 2021 ; Conference date: 13-12-2021 Through 14-12-2021",

year = "2021",

month = jan,

day = "1",

doi = "10.1007/978-3-030-91825-5\_15",

language = "English",

isbn = "9783030918248",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "243--263",

editor = "Paolo Ballarini and Hind Castel and Ioannis Dimitriou and Mauro Iacono and Tuan Phung-Duc and Joris Walraevens",

booktitle = "Performance Engineering and Stochastic Modeling - 17th European Workshop, EPEW 2021, and 26th International Conference, ASMTA 2021, Proceedings",

}

Tournaire, T, Barthelemy, J, Castel-Taleb, H & Hyon, E 2021, Reinforcement Learning with Model-Based Approaches for Dynamic Resource Allocation in a Tandem Queue. in P Ballarini, H Castel, I Dimitriou, M Iacono, T Phung-Duc & J Walraevens (eds), Performance Engineering and Stochastic Modeling - 17th European Workshop, EPEW 2021, and 26th International Conference, ASMTA 2021, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13104 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 243-263, 17th European Performance Engineering Workshop, EPEW 2021, and the 26th International Conference on Analytical and Stochastic Modelling Techniques and Applications, ASMTA 2021, Virtual, Online, 13/12/21. https://doi.org/10.1007/978-3-030-91825-5_15

Reinforcement Learning with Model-Based Approaches for Dynamic Resource Allocation in a Tandem Queue. / Tournaire, Thomas; Barthelemy, Jeanne; Castel-Taleb, Hind et al.
Performance Engineering and Stochastic Modeling - 17th European Workshop, EPEW 2021, and 26th International Conference, ASMTA 2021, Proceedings. ed. / Paolo Ballarini; Hind Castel; Ioannis Dimitriou; Mauro Iacono; Tuan Phung-Duc; Joris Walraevens. Springer Science and Business Media Deutschland GmbH, 2021. p. 243-263 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13104 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Reinforcement Learning with Model-Based Approaches for Dynamic Resource Allocation in a Tandem Queue

AU - Tournaire, Thomas

AU - Barthelemy, Jeanne

AU - Castel-Taleb, Hind

AU - Hyon, Emmanuel

PY - 2021/1/1

Y1 - 2021/1/1

N2 - We consider three-tier network architecture modeled with two physical nodes in tandem where an autonomous agent controls the number of active resources on each node. We analyse the learning of auto-scaling strategies in order to optimise both performance and energy consumption of the whole system. We compare several model-based reinforcement learning with model-free Q-learning algorithm. The relevance of these algorithms is to faster update Q-value function with an additional planning phase allowed by approximated model of the dynamics of the environment. Secondly, we consider the same tandem queue scenario with MMPP (Markov modulated Poisson process) for arrivals. In this context, the arrival rate is varying over time and this information is hidden to the agent. Our goal is to assess the robustness of such model-based reinforcement learning algorithms in this particular scenario.

AB - We consider three-tier network architecture modeled with two physical nodes in tandem where an autonomous agent controls the number of active resources on each node. We analyse the learning of auto-scaling strategies in order to optimise both performance and energy consumption of the whole system. We compare several model-based reinforcement learning with model-free Q-learning algorithm. The relevance of these algorithms is to faster update Q-value function with an additional planning phase allowed by approximated model of the dynamics of the environment. Secondly, we consider the same tandem queue scenario with MMPP (Markov modulated Poisson process) for arrivals. In this context, the arrival rate is varying over time and this information is hidden to the agent. Our goal is to assess the robustness of such model-based reinforcement learning algorithms in this particular scenario.

KW - Cloud

KW - Energy saving

KW - Model-based reinforcement learning

KW - QoS guarantee

KW - Tandem queues

U2 - 10.1007/978-3-030-91825-5_15

DO - 10.1007/978-3-030-91825-5_15

M3 - Conference contribution

AN - SCOPUS:85121931923

SN - 9783030918248

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 243

EP - 263

BT - Performance Engineering and Stochastic Modeling - 17th European Workshop, EPEW 2021, and 26th International Conference, ASMTA 2021, Proceedings

A2 - Ballarini, Paolo

A2 - Castel, Hind

A2 - Dimitriou, Ioannis

A2 - Iacono, Mauro

A2 - Phung-Duc, Tuan

A2 - Walraevens, Joris

PB - Springer Science and Business Media Deutschland GmbH

T2 - 17th European Performance Engineering Workshop, EPEW 2021, and the 26th International Conference on Analytical and Stochastic Modelling Techniques and Applications, ASMTA 2021

Y2 - 13 December 2021 through 14 December 2021

ER -

Tournaire T, Barthelemy J, Castel-Taleb H, Hyon E. Reinforcement Learning with Model-Based Approaches for Dynamic Resource Allocation in a Tandem Queue. In Ballarini P, Castel H, Dimitriou I, Iacono M, Phung-Duc T, Walraevens J, editors, Performance Engineering and Stochastic Modeling - 17th European Workshop, EPEW 2021, and 26th International Conference, ASMTA 2021, Proceedings. Springer Science and Business Media Deutschland GmbH. 2021. p. 243-263. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-91825-5_15

Reinforcement Learning with Model-Based Approaches for Dynamic Resource Allocation in a Tandem Queue

Abstract

Publication series

Conference

UN SDGs

Keywords

Access to Document

Fingerprint

Cite this