TY - GEN
T1 - A Context-Aware Benchmarking Approach for Scenario-Based Evaluation of Smart Devices
AU - Papadakis, Nikolaos
AU - Tsirikas, Athanasios
AU - Fouskas, Fotios
AU - Chatterjie, Krish
AU - Badolato, Christian
AU - Blanc, Gregory
AU - Yus, Roberto
AU - Magoutis, Kostas
AU - Bouloukakis, Georgios
N1 - Publisher Copyright:
© 2025 Copyright held by the owner/author(s).
PY - 2025/12/12
Y1 - 2025/12/12
N2 - Understanding how smart devices behave under real-world conditions remains challenging, as existing benchmarking tools often isolate metrics like energy consumption or protocol compliance without considering realistic interactions or coordinated use. This paper presents a scenario-driven benchmarking framework tailored for smart devices which integrates control, orchestration, instrumentation, and monitoring into a unified, reproducible workflow. Our framework enables structured specification of interaction scenarios, precise scheduling of device actions, and synchronized measurement of energy usage and network activity. We validate our approach through a smart home testbed comprised of off-the-shelf speakers, lights, and fans across scenarios that reflect typical user interactions and adversarial network conditions. Results reveal behavioral differences not evident from device specifications alone, highlighting the importance of context-rich, scenario-based evaluation for understanding smart device performance in practice.
AB - Understanding how smart devices behave under real-world conditions remains challenging, as existing benchmarking tools often isolate metrics like energy consumption or protocol compliance without considering realistic interactions or coordinated use. This paper presents a scenario-driven benchmarking framework tailored for smart devices which integrates control, orchestration, instrumentation, and monitoring into a unified, reproducible workflow. Our framework enables structured specification of interaction scenarios, precise scheduling of device actions, and synchronized measurement of energy usage and network activity. We validate our approach through a smart home testbed comprised of off-the-shelf speakers, lights, and fans across scenarios that reflect typical user interactions and adversarial network conditions. Results reveal behavioral differences not evident from device specifications alone, highlighting the importance of context-rich, scenario-based evaluation for understanding smart device performance in practice.
KW - Benchmarking
KW - Context-aware Systems
KW - Energy Profiling
KW - Home Automation
KW - Network Traffic Analysis
KW - Smart Devices
UR - https://www.scopus.com/pages/publications/105025594282
U2 - 10.1145/3770501.3770516
DO - 10.1145/3770501.3770516
M3 - Conference contribution
AN - SCOPUS:105025594282
T3 - IOT 2025 - Proceedings of the 15th International Conference on the Internet of Things 2025
SP - 122
EP - 130
BT - IOT 2025 - Proceedings of the 15th International Conference on the Internet of Things 2025
A2 - Nastic, Stefan
A2 - Michahelles, Florian
A2 - Ristov, Sashko
A2 - Dazzi, Patrizio
A2 - Wolling, Florian
PB - Association for Computing Machinery, Inc
T2 - 15th International Conference on the Internet of Things, IoT 2025
Y2 - 18 November 2025 through 21 November 2025
ER -
