ATOFIS, an AR Training System for Manual Assembly: A Full Comparative Evaluation against Guides

This paper reports on a user study to comparatively evaluate two AR training systems designed for step-by-step manual operations: ATOFIS-recently proposed in the literature, and Microsoft Dynamics 365 Guides (hereinafter Guides)-one of the most relevant state-of-the-art commercial solutions. The user study (N=16) was conducted in two stages-i.e., training and authoring, on a partial replica of a real-world assembly workstation. During training, the participant learns a sequence of manual operations by performing two assembly cycles, guided by each of the two AR training systems. During authoring, the participant creates the two sets of AR work instructions used in the next training session, one set with each of the two authoring systems. We bound the authoring and training procedures during the experiment to comparatively assess the AR systems overall, and address at the same time an evaluation gap observed in the literature. The experimental results demonstrated advantages of the authoring approach proposed by ATOFIS (i.e., low-cost, formalized, in-situ, immersive and on-the-fly), proved the usability and effectiveness of the AR instructions authored with ATOFIS and validated a set of hypotheses formulated by the authors of the system. ATOFIS authoring was 1.72 × faster and unanimously preferred by the participants; ATOFIS training reported zero assembly errors and was 13% faster than Guides. ATOFIS reported excellent system usability (i.e., SUS) and mental workload (i.e., NASA-TLX) scores for both authoring and training, outperforming Guides on all dimensions.