User Experience (UX) evaluation is fundamental for digital product improvement, yet traditional written questionnaires face limitations in engagement, accessibility, and response consistency. To address this, we present the design, development, and validation of voice-based adaptations of the User Experience Questionnaire, or UEQ, using natural conversational interfaces. This research introduces two distinct implementations: direct scale mapping approach called Voice1-7, and a novel branched-dialog approach called Voice2Q, which uses sequential questions to capture attitude direction and intensity. Further,we propose Voice2Q+VC, a voice-first variant with minimal visual cues to enhance clarity while preserving voice interaction primacy. Multi-stage validation was conducted through multiple case studies involving 345 participants in the design and evaluation phases. These studies compared voice implementations against the standard written version of UEQ. Statistical analyses across diverse participant groups confirmed that both Voice2Q and Voice1-7 produced similar results to the written UEQ in core UX dimensions —Attractiveness, Perspicuity, Efficiency, Dependability, Stimulation, Novelty—establishing their measurement validity. A Usability, UX and cognitive workload comparison was conducted. Results revealed thatVoice2Q significantly reduced response inconsistencies and excelled in UX hedonic stimulation, while the written UEQ retained advantages in UX pragmatic efficiency. Voice2Q+VC mitigated workload challenges inherent in voice-only interactions, outperforming the written UEQ in Usability and UX hedonic dimensions while approaching its UX pragmatic performance levels. These results suggest that supplemental visuals can optimize voice-driven evaluation without sacrificing conversational engagement. This work confirms voice-based UEQ as a statistically valid, accessible alternative to written formats, with benefits in response reliability and user engagement. The Voice2Q+VC implementation presents a promising paradigm for balancing natural interaction with cognitive efficiency in UX evaluation.

1. J. A. Krosnick and M. K. Berent, “Comparisons of party identification and policy preferences: The impact of survey question format,” American Journal of Political Science, pp. 941–964, 1993, doi: 10.2307/2111580.
2. E. Cho, M. D. Molina, and J. Wang, “The effects of modality, device, and task differences on perceived human likeness of voice-activated virtual assistants,” Cyberpsychology, Behavior, and Social Networking, vol. 22, no. 8, pp. 515–520, 2019, doi: 10.1089/cyber.2018.0571.
3. J. K. Höhne, K. Gavras, and J. Claassen, “Typing or speaking? Comparing text and voice answers to open questions on sensitive topics in smartphone surveys,” Social Science Computer Review, vol. 42, no. 4, pp. 1066–1085, 2024, doi: 10.1177/08944393231160961.
4. B. Laugwitz, T. Held, and M. Schrepp, “Construction and evaluation of a user experience questionnaire,” in Symposium of the Austrian HCI and Usability Engineering Group, pp. 63–76, Springer, 2008, doi: 10.1007/978-3-540-89350-9_6.
5. C. Lallemand and G. Gronier, Méthodes de Design UX: 30 Méthodes Fondamentales pour Concevoir et Évaluer les Systèmes Interactifs, Éditions Eyrolles, 2015.
6. M. Hassenzahl, M. Burmester, and F. Koller, “AttrakDiff: Ein Fragebogen zur Messung Wahrgenommener Hedonischer und Pragmatischer Qualität,” in Mensch & Computer 2003: Interaktion in Bewegung, pp. 187–196, Springer, 2003, doi: 10.1007/978-3-322-80058-9_19.
7. I. Díaz-Oreiro, G. López, L. Quesada, and L. A. Guerrero, “UX evaluation with standardized questionnaires in ubiquitous computing and ambient intelligence: A systematic literature review,” Advances in Human-Computer Interaction, vol. 2021, no. 1, p. 5518722, 2021, doi: 10.1155/2021/5518722.
8. D. Norman, J. Miller, and A. Henderson, “What you see, some of what’s in the future, and how we go about doing it: HI at Apple Computer,” in Conference Companion on Human Factors in Computing Systems, p. 155, 1995, doi: 10.1145/223355.223477.
9. E. Law, V. Roto, A. P. Vermeeren, J. Kort, and M. Hassenzahl, “Towards a shared definition of user experience,” in CHI ’08 Extended Abstracts on Human Factors in Computing Systems, CHI EA ’08, pp. 2395–2398, Association for Computing Machinery, New York, NY, USA, 2008, doi: 10.1145/1358628.1358693.
10. V. Roto, E.-C. Law, A. Vermeeren, and J. Hoonhout, User Experience White Paper: Bringing Clarity to the Concept of User Experience, 2011.
11. C. Lallemand, G. Gronier, and V. Koenig, “User experience: A concept without consensus? Exploring practitioners’ perspectives through an international survey,” Computers in Human Behavior, vol. 43, pp. 35–48, 2015, doi: 10.1016/j.chb.2014.10.048.
12. ISO, “9241-210:2010. Ergonomics of Human-System Interaction—Part 210: Human-Centred Design for Interactive Systems,” International Organization for Standardization, Switzerland, vol. 2, 2009.
13. A. P. Vermeeren, E. L.-C. Law, V. Roto, M. Obrist, J. Hoonhout, and K. Väänänen-Vainio-Mattila, “User experience evaluation methods: Current state and development needs,” in Proceedings of the 6th Nordic Conference on Human-Computer Interaction: Extending Boundaries, pp. 521–530, 2010, doi: 10.1145/1868914.1868973.
14. A. Hinderks, “Design and evaluation of a short version of the user experience questionnaire (UEQ-S),” International Journal of Interactive Multimedia and Artificial Intelligence, 2017, doi: 10.9781/ijimai.2017.09.001.
15. D. Wallach, J. Conrad, and T. Steimle, “The UX Metrics Table: A Missing Artifact,” in International Conference of Design, User Experience, and Usability, pp. 507–517, Springer, 2017, doi: 10.1007/978-3-319-58634-2_37.
16. G. Gronier, C. Lallemand, and A. Chauvet, “Mesurer la Formation de la Première Impression d’une Interface à l’Aide du Test des 5 Secondes,” in Huitième Colloque de Psychologie Ergonomique (EPIQUE), 2015.
17. V. Roto, M. Obrist, and K. Väänänen-Vainio-Mattila, “User experience evaluation methods in academic and industrial contexts,” in Proceedings of the Workshop UXEM, vol. 9, pp. 1–5, 2009.
18. C. Lallemand and V. Koenig, “How could an intranet be like a friend to me? Why standardized UX scales don’t always fit,” in Proceedings of the European Conference on Cognitive Ergonomics, pp. 9–16, 2017, doi: 10.1145/3121283.3121288.
19. C. L. B. Maia and E. S. Furtado, “A systematic review about user experience evaluation,” in International Conference of Design, User Experience, and Usability, pp. 445–455, Springer, 2016, doi: 10.1007/978-3-319-40409-7_42.
20. Y. Forster, S. Hergeth, F. Naujoks, and J. F. Krems, “How usability can save the day—Methodological considerations for making automated driving a success story,” in Proceedings of the 10th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, pp. 278–290, 2018, doi: 10.1145/3239060.3239076.
21. J. Klammer and F. W. van den Anker, “A platform to connect Swiss consumers of fair trade products with producers in developing countries: Needs and motivations,” in International Conference of Design, User Experience, and Usability, pp. 664–681, Springer, 2018, doi: 10.1007/978-3-319-91806-8_52.
22. J. Baumgartner, A. Sonderegger, and J. Sauer, “No need to read: Developing a pictorial single-item scale for measuring perceived usability,” International Journal of Human-Computer Studies, vol. 122, pp. 78–89, 2019, doi: 10.1016/j.ijhcs.2018.08.008.
23. D. Wigdor and D. Wixon, Brave NUI World: Designing Natural User Interfaces for Touch and Gesture, Elsevier, 2011.
24. T. Nishida, Conversational Informatics: An Engineering Approach, John Wiley & Sons, 2008.
25. D. Frohlich and P. Luff, “Applying the technology of conversation to the technology for conversation,” in Computers and Conversation, pp. 187–220, Elsevier, 1990.
26. V. K. Chaudhri, A. Cheyer, R. Guilii, W. Jarrold, K. L. Myers, and J. Niekarsz, “A case study in engineering a knowledge base for an intelligent personal assistant,” in SemDesk, pp. 25–32, 2006.
27. D. M. Kaushik and R. Jain, “Natural user interfaces: Trend in virtual interaction,” arXiv preprint arXiv:1405.0101, 2014.
28. E. A. Schegloff, Sequence Organization in Interaction: A Primer in Conversation Analysis I, vol. 1, Cambridge University Press, 2007, doi: 10.1017/CBO9780511791208.
29. D. A. Norman, Living with Complexity, MIT Press, 2016.
30. R. J. Moore and R. Arar, “Conversational UX Design: An Introduction,” in Studies in Conversational UX Design, pp. 1–16, Springer, 2018, doi: 10.1007/978-3-319-95579-7_1.
31. R. J. Moore and R. Arar, Conversational UX Design: A Practitioner’s Guide to the Natural Conversation Framework, Morgan & Claypool, 2019, doi: 10.1145/3304087.
32. H. Sacks, E. A. Schegloff, and G. Jefferson, “A simplest systematics for the organization of turn-taking for conversation,” Language, vol. 50, no. 4, pp. 696–735, 1974, doi: 10.1353/lan.1974.0010.
33. A. B. Kocaballi, L. Laranjo, and E. Coiera, “Understanding and measuring user experience in conversational interfaces,” Interacting with Computers, vol. 31, no. 2, pp. 192–207, 2019, doi: 10.1093/iwc/iwz022.
34. A. L. Iniguez-Carrillo, L. S. Gaytan-Lugo, M. A. Garcia-Ruiz, and R. Maciel-Arellano, “Usability questionnaires to evaluate voice user interfaces,” IEEE Latin America Transactions, vol. 19, no. 9, pp. 1468–1477, 2021, doi: 10.1109/TLA.2021.9477283.
35. J. R. Lewis, “Standardized questionnaires for voice interaction design,” Voice Interaction Design, vol. 1, no. 1, pp. 1–16, 2016.
36. F. Iniesto, T. Coughlan, and K. Lister, “Implementing an accessible conversational user interface: Applying feedback from university students and disability support advisors,” in Proceedings of the 18th International Web for All Conference, pp. 1–5, 2021, doi: 10.1145/3430263.3452436.
37. J. Wei, W. Jiang, C. Wang, D. Yu, J. Goncalves, T. Dingler, and V. Kostakos, “Understanding how to administer voice surveys through smart speakers,” Proceedings of the ACM on Human-Computer Interaction, vol. 6, no. CSCW2, pp. 1–32, 2022, doi: 10.1145/3555767.
38. A. Barbaric, C. Munteanu, H. Ross, and J. A. Cafazzo, “A voice app design for heart failure self-management: Proof-of-concept implementation study,” JMIR Formative Research, vol. 6, no. 12, p. e40021, 2022, doi: 10.2196/40021.
39. J.-G. Shin, G.-Y. Choi, H.-J. Hwang, and S.-H. Kim, “Evaluation of emotional satisfaction using questionnaires in voice-based human–AI interaction,” Applied Sciences, vol. 11, no. 4, p. 1920, 2021, doi: 10.3390/app11041920.
40. E. Kuang, E. Jahangirzadeh Soure, M. Fan, J. Zhao, and K. Shinohara, “Collaboration with conversational AI assistants for UX evaluation: Questions and how to ask them (voice vs. text),” in Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–15, 2023, doi: 10.1145/3544548.3581247.
41. B. Zarouali, T. Araujo, J. Ohme, and C. De Vreese, “Comparing chatbots and online surveys for (longitudinal) data collection: An investigation of response characteristics, data quality, and user evaluation,” Communication Methods and Measures, vol. 18, no. 1, pp. 72–91, 2024, doi: 10.1080/19312458.2023.2210576.
42. E. A. Beam, “Social media as a recruitment and data collection tool: Experimental evidence on the relative effectiveness of web surveys and chatbots,” Journal of Development Economics, vol. 162, p. 103069, 2023, doi: 10.1016/j.jdeveco.2023.103069.
43. H. Soni et al., “Virtual conversational agents versus online forms: Patient experience and preferences for health data collection,” Frontiers in Digital Health, vol. 4, p. 954069, 2022, doi: 10.3389/fdgth.2022.954069.
44. P. Sprengholz and C. Betsch, “Ok Google: Using virtual assistants for data collection in psychological and behavioral research,” Behavior Research Methods, vol. 54, no. 3, pp. 1227–1239, 2022, doi: 10.3758/s13428-021-01629-y.
45. I. Celino and G. R. Calegari, “Submitting surveys via a conversational interface: An evaluation of user acceptance and approach effectiveness,” International Journal of Human-Computer Studies, vol. 139, p. 102410, 2020, doi: 10.1016/j.ijhcs.2020.102410.
46. R. Maharjan, D. A. Rohani, P. Bækgaard, J. Bardram, and K. Doherty, “Can we talk? Design implications for the questionnaire-driven self-report of health and wellbeing via conversational agent,” in Proceedings of the 3rd Conference on Conversational User Interfaces, pp. 1–11, 2021, doi: 10.1145/3469595.3469600.
47. P. V. Miller, “Alternative question forms for attitude scale questions in telephone interviews,” Public Opinion Quarterly, vol. 48, no. 4, pp. 766–778, 1984, doi: 10.1086/268879.
48. L. R. Fabrigar and J. A. Krosnick, “Attitude measurement and questionnaire design,” in Blackwell Encyclopedia of Social Psychology, pp. 42–47, 1995.
49. J. H. Yu, G. Albaum, and M. Swenson, “Is a central tendency error inherent in the use of semantic differential scales in different cultures?” International Journal of Market Research, vol. 45, no. 2, pp. 1–16, 2003, doi: 10.1177/147078530304500201.
50. V. D. de Rada Igúzquiza, “¿Influye el diseño de las preguntas en las respuestas de los entrevistados?” Revista Española de Sociología, vol. 31, no. 1, p. a83, 2022, doi: 10.22325/fes/res.2022.83.
51. J. C. Mata-Serrano, I. Díaz-Oreiro, G. López, and L. A. Guerrero, “Comparing written and voice captured responses of the User Experience Questionnaire (UEQ),” in International Conference on Information Technology & Systems, pp. 519–529, Springer, 2022, doi: 10.1007/978-3-030-96293-7_43.
52. M. Rauschenberger, M. Schrepp, M. Pérez Cota, S. Olschner, and J. Thomaschewski, “Efficient measurement of the user experience of interactive products: How to use the User Experience Questionnaire (UEQ). Example: Spanish language version,” International Journal of Interactive Multimedia and Artificial Intelligence, vol. 2, no. 1, pp. 39–45, 2013, doi: 10.9781/ijimai.2013.215.
53. I. Díaz-Oreiro, G. López, L. Quesada, and L. A. Guerrero, “Conversational design patterns for a UX evaluation instrument implemented by voice,” in International Conference on Information Technology & Systems, pp. 530–540, Springer, 2022, doi: 10.1007/978-3-030-96293-7_44.
54. J. Brooke et al., “SUS—A Quick and Dirty Usability Scale,” in Usability Evaluation in Industry, vol. 189, no. 194, pp. 4–7, 1996, doi: 10.1201/9781498710411-35.
55. S. G. Hart and L. E. Staveland, “Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research,” in Advances in Psychology, vol. 52, pp. 139–183, Elsevier, 1988, doi: 10.1016/S0166-4115(08)62386-9.
56. M. Hassenzahl, “The effect of perceived hedonic quality on product appealingness,” International Journal of Human-Computer Interaction, vol. 13, no. 4, pp. 481–499, 2001, doi: 10.1207/S15327590IJHC1304_07.
57. S. G. Hart, “NASA-Task Load Index (NASA-TLX): 20 years later,” Proceedings of the Human Factors and Ergonomics Society Annual Meeting, vol. 50, pp. 904–908, 2006, doi: 10.1177/154193120605000909.
58. R. Wang and J. A. Krosnick, “Middle alternatives and measurement validity: A recommendation for survey researchers,” International Journal of Social Research Methodology, vol. 23, no. 2, pp. 169–184, 2020, doi: 10.1080/13645579.2019.1645384.
59. A. Schankin, M. Budde, T. Riedel, and M. Beigl, “Psychometric properties of the User Experience Questionnaire (UEQ),” in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1–11, 2022, doi: 10.1145/3491102.3502098.

Citations by Dimensions

Citations by PlumX

Crossref Citations

1. Ignacio Diaz-Oreiro, Gustavo Lopez, “Implementing SAP Fiori in S/4HANA Transitions: Key Guidelines, Challenges, Strategic Implications, AI Integration Recommendations”, Journal of Engineering Research and Sciences, vol. 4, no. 11, pp. 1–9, 2025. doi: 10.55708/js0411001
2. Ignacio Diaz-Oreiro, Gustavo Lopez, “Content Recommendation E-learning System for Personalized Learners to Enhance User Experience using SCORM”, Journal of Engineering Research and Sciences, vol. 4, no. 9, pp. 30–46, 2025. doi: 10.55708/js0409004
3. Ignacio Diaz-Oreiro, Gustavo Lopez, “Human-Computer Interaction for Older Adults – a Literature Review on Technology Acceptance of eHealth Systems”, Journal of Engineering Research and Sciences, vol. 1, no. 4, pp. 119–126, 2022. doi: 10.55708/js0104014