The evaluation of usability, user experience, and accessibility (UAUX) has been extensively studied. However, it has often been limited to specific domains with small samples. This paper presents an extensive systematic literature review of 582 case studies published between 2014 and 2023. Our findings confirm the continued dominance of usability with the System Usability Scale (SUS) remaining the most prevalent evaluation instrument, UX evaluation is consolidating as a distinct field, frequently employing standardized tools such as the User Experience Questionnaire, and accessibility remains significantly underrepresented. The review also highlights a persistent reliance on single-method approaches, a concerning use of ad-hoc, unvalidated questionnaires, and notable geographic concentration of research. By comparing our results with a prior tertiary review, we identify methodological preferences and persistent gaps, such as the lack of integrated UAUX evaluation and the need for more longitudinal studies. Another valuable conclusion of this research is clear evidence on the number of participants commonly involved in UAUX evaluation studies.

1. Introduction

Given that digital technologies have become ubiquitous in current society, Human Computer Interaction in general and usability, accessibility and user experience (UAUX) are a critical factor of success for any digital system. Each day technologies become more complex and integral part of our life. Therefore, ensuring that such systems are not only useful, but also intuitive, efficient and satisfying to use is a general concern among designers. This has led to the growing importance of the evaluation of UAUX.

In this paper we refer to usability as the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use [1]. Accessibility is conceived as ensuring that digital products are usable by people with the widest range of abilities, including those with disabilities [2]. Finally, user experience encompasses a broader spectrum, including all aspects of a user’s perceptions and responses resulting from the use or anticipated use of a product, system, or service [3].

Even though it is not ideal, this research uses academic research to create a foundational idea on the UAUX evaluation practices, with a dataset of 582 research pa-pers published from 2014 until 2023, this paper offers a comprehensive snapshot of UAUX evaluation and its evolution. Our study captures empirical evidence of diverse domains such as eHealth, eLearning, eGovernment, virtual/augmented reality, and others.

The included studies employ a diverse set of evaluation methods including standardized questionnaires (e.g., System Usability Scale (SUS), User Experience Questionnaire (UEQ)), heuristic evaluations, think-aloud protocols, cognitive walkthroughs, and automated accessibility testing tools.

The main goals of this paper with this diverse and large number of included studies are:

1. Analyze the distribution and frequency of evaluation methods and metrics used across studies.
2. Examine the interrelationship and co-evaluation of
3. Highlight emerging trends, methodological preferences, and apparent research gaps.

All of these with a clear identification of domains and technologies that allow a comprehensive analysis of the results and compare them with a past study in which a tertiary review was conducted [4]. Unlike previous reviews limited to dozens of studies within specific domains, this work synthesizes 582 primary studies spanning a decade, enabling a large scale empirical characterization of UAUX evaluation practices.

The following section (2) summarizes the tertiary study that motivated this research, after that, section 3 describes the methodology used for the Systematic Literature Review. Then, section 4 shows our results including time and geographic, the number of participants included in the primary studies, the aspects that were evaluated and the evaluation techniques. Finally, section 5 concludes the paper and provides insights for future work.

2. Tertiary review findings

This section summarizes the results presented in [4] as a base to compare the results of this literature review. The tertiary study included 45 secondary studies on the evaluation of UAUX.

The tertiary study found that usability was the most evaluated aspect (27 SLRs), followed by user experience (16 SLRs) and accessibility (8 SLRs). Furthermore, the most common domains evaluated were web and education (13 SLRs) followed by mobile apps (8 SLRs).

The in depth analysis reveals that regarding usability, the most common evaluation practices are questionnaires especially the System Usability Scale (SUS). Other frequently employed methods include Heuristic Evaluation (often using Nielsen’s heuristics), Usability Testing, and the Think-Aloud protocol.

Regarding accessibility, automated evaluation tools dominate the methods, particularly WAVE, AChecker, and TAW were the most used. The Web Content Accessibility Guidelines (WCAG) served as the primary standard referenced in these evaluations.

Finally, concerning user experience, questionnaires (both standardized and custom) were identified as the most common evaluation method. User Experience Questionnaire (UEQ) and AttrakDiff were the most used standardized methods. However, it was common to find researchers using ad hoc questionnaires that were often unvalidated.

Using HCI quality attributes, the tertiary study identified satisfaction (13 SLRs), efficiency (13 SLRs), effective-ness (12 SLRs), attractiveness (12 SLRs), and learnability (11 SLRs) as the most consistently measured attributes.

One of the main findings of the tertiary study, which in fact supports this research, is that the median number of primary studies analyzed within the secondary SLRs was 31, a number that allows for a review in a particular domain but is quite small for a general analysis such as the one presented in this paper.

Finally, the tertiary study concluded with several opportunities that inform the rationale for our extended work:

• It was found that there was an overreliance on single evaluation methods, particularly with a heavy dependence on questionnaires and automated
• Many studies reported using ad hoc
• There is a general lack of longitudinal assessment because evaluations were predominantly conducted after short-term use.
• There is a need for comparative studies that include less common evaluation tools and expand the analysis.

With this summary, our research builds its premises using a much larger sample to create a more holistic vision of UAUX evaluation practices, methods, and tools.

3. SLR Methodology

The methodology followed in this research was con-ducted following two main methods [5] and [6]. Figure 1 shows the process followed. The main goal of the review was to find out:

• Which tools and methodologies are most frequently applied in case studies evaluating UAUX?
• What are the primary dimensions or characteristics measured in these case studies?
• How are these case studies methodologically conducted and reported?

3.1. Search Process

To identify relevant case studies, a systematic search was conducted across three major digital libraries: Springer Link, IEEE Digital Library, and ACM Digital Library The search query was structured using a combination of keywords to capture studies focused on evaluation of UAUX and it included three main parts: the evaluation (“test” OR “evaluation” OR “assess” OR “measure” OR “case study”), the concepts evaluated (“UX” OR “user experience” OR “usability” OR “accessibility”), and the context (“application” OR “software” OR “system” OR “website” OR “app”) Searches were limited to titles, abstracts, and keywords to increase relevance. No initial filters were applied for publication year.

3.2. Inclusion and Exclusion Criteria

To ensure the selected literature aligned with the study’s focus on case studies, the following criteria were applied: Papers must be written in English, they must be a case study and be considered primary research, and papers must be published between January 2014 and December 2023. Although this last criterion was defined a priori, the initial database query was intentionally run without a date restriction, the year filter was subsequently applied as a post-retrieval exclusion step during screening.

The exclusion criteria consisted of: papers that proposed frameworks, papers that presented designs or developments without empirical evidence of evaluation, non full papers, duplicate studies or studies not accessible in full text.

3.3. Screening

The screening process followed several stages. First, the initial retrieval and deduplication. After this, the title and abstract screening in which two reviewers independently screened titles and abstracts against the inclusion/exclusion criteria. Conflicts were resolved through discussion or consultation with a third reviewer.

Third, relevant papers were obtained in full text and assessed for eligibility. Papers were excluded if they did not meet the case study definition or lacked sufficient de-tail on the evaluation process. The final list of included case studies was validated by the research team to ensure consistency with the research objectives.

A total of 1742 papers from various sources (231 from ACM, 528 from IEEE, and 983 from Springer) were later identified gathered, and 958 papers were initially identified as case studies. After applying the publication year filter (2014-2023), 582 papers remained eligible for inclusion in this review and were included for data extraction.

3.4. Data Extraction

A structured data extraction form was developed to systematically capture relevant information from each included case study. The following data points were collected:

• Bibliographic Information: Authors, publication year, venue, and country of origin.
• Sample size
• Evaluated platform: web, mobile,
• Evaluated aspects: whether the study focused on usability, user experience, accessibility, or a combina-tion.
• Methods and Tools: Specific evaluation instruments used (e.g., SUS, UEQ, heuristic evaluation, automated tools).

Data extraction was performed independently by two researchers, with discrepancies resolved through consensus. Descriptive statistics were used to synthesize findings.

4. Results

The following subsections show the results divided into various dimensions, such as time, geography, aspects evaluated, number of participants, evaluation techniques, and title patterns.

4.1. Year Distribution

As first dimension of analysis, the distribution per year showed a clear upward trend between 2014 and 2019. A small drop in 2020 (likely due to the global disruption caused by the COVID-19 pandemic) and an upward trend again between 2020 and 2022. The growth reflects the increasing importance of UAUX. Table 1 shows the concrete data.

Table 1: Number of studies published per year.

4.2. Geographic Distribution

The patterns detected are not limited to temporal trends. The geographical distribution (see Figure 2) of the 582 case studies shows concentration in some regions of the world. Indonesia stands out as the country in which the most case studies were conducted (62 studies), followed by the United States (42 studies), Brazil (38 studies), the United Kingdom (23 studies), and China and Spain (21 studies).

However, a considerable number (68 studies) did not specify the country of origin, which limits the findings of this analysis. Furthermore, 26 studies involved international collaborations across multiple countries. A total of 17 countries (Afghanistan, Belgium, Cyprus, the Czech Republic, Egypt, Ethiopia, Ghana, Japan, Jordan, Kosovo, Luxembourg, Namibia, New Zealand, Oman, Palestine, Singapore, and Uganda) appear in only one case study each.

Interestingly, Asia has 43% of the published research, followed by Europe with 17.5%. America (both north and south) have 22.8%. Africa and Oceania together have less than 3%.

4.3. Aspects evaluated

After describing when and where these studies were conducted, it is necessary to examine what was evaluated. Table 2 shows all the possible combinations of aspects evaluated considering UAUX domain, with the corresponding frequency. This way, we can disaggregate studies into eight categories, including one for those that cannot be classified due to ambiguities or unclear reporting of the evaluation.

Table 2: Aspects evaluated.

Because many studies evaluate more than one UAUX aspect simultaneously, the frequencies reported below reflect each aspect’s presence across all relevant combinations rather than mutually exclusive counts.

Usability is the most frequently evaluated attribute, appearing in 383 studies (65.8%) across all combinations. Studies evaluating usability only represent the largest category with 208 studies (35.7%) [7–214]. UX shows a substantial presence, with 134 studies (23.0%) dedicated exclusively to its evaluation [215–348], in addition to 94 studies (16.2%) that combine it with usability [349–442]. In total, UX is present in 260 studies (44.7%).

Finally, Accessibility is the aspect with the lowest representation, present in 134 studies (23.0%) overall. Accessibility-only studies account for 42 studies (7.2%) [443–484]. Interestingly, there are 60 studies that include accessibility in combination with usability (10.3%) [485–544], while the UX + Accessibility combination is the least frequent of all, with merely 11 studies (1.9%)[545–555].

It is important to note that the predominance of usability was expected as it is consistent with the historical tradition of the field, where effectiveness, efficiency, and satisfaction have been the fundamental metrics. Regarding UX, the proportion of UX-only studies suggests that user experience is consolidating as a field. It is concerning that accessibility has the lowest representation, especially in the current context of population aging, legal frameworks for inclusion (such as WCAG and ISO standards), and growing awareness of the need for universal design.

Only 21 studies (3.6%) [556–576] simultaneously evaluate all three UAUX aspects, but modern digital systems require a balance between efficient functioning (usability), rewarding experiences (UX), and universal access (accessibility); therefore, this lack of integration represents a significant limitation in the current literature.

Besides, 12 studies (2.1%) could not be clearly classified into the above categories [577–588]. This may be because they evaluated related but different attributes (such as engagement, trust, or technology acceptance), or due to ambiguities in the reporting of the methods used. This category highlights the need to standardize terminology and reporting in software quality evaluations.

4.4. Number of participants

An analysis of sample sizes across different types of UAUX evaluations reveals notable differences in the distribution of participants, as shown in Figure 3.

Usability only studies represent the largest group, with 194 studies reporting sample sizes and 14 unspecified. The median was 20 participants, ranging from 1 to 1,074. The first quartile was 10, the third quartile 44.25, yielding an interquartile range of 34.25.

UX-only studies had 119 reporting sample sizes and 15 unspecified. The median number was 36 participants, with a minimum of 3 and a maximum of 21,314. The first quartile was 15, the third quartile 74, and the interquartile range 59.

For accessibility only studies, 31 studies reported their sample sizes, while 11 did not. These studies had a median of 20 participants, with a minimum of 1 and a maximum of 400.

Thefirst quartile was 7, the third quartile 56, resulting in an interquartile range of 49.

Studies evaluating usability and UX together had 89 reporting sample sizes and only 5 unspecified. The median was 32 participants, with a minimum of 5 and a maximum of 24,000—the largest range observed. The first quartile was 12.5, the third quartile 88, and the interquartile range 75.5.

For studies combining usability and accessibility, 41 reported sample sizes, with 19 unspecified. The median was 19 participants, with a minimum of 2 and a maximum of 1,272. The first quartile was 10.5, the third quartile 45.5, and the interquartile range 35.

Studies combining UX and accessibility were the least numerous, with only 7 reporting sample sizes and 4 un-specified. The median was 6 participants, ranging from 5 to 47. The first quartile was 5, the third quartile 12, and the interquartile range 7—the narrowest of all categories.

Finally, studies covering usability, UX, and accessibility had 14 reporting sample sizes and 7 unspecified. The median was 20 participants, ranging from 4 to 84. The first quartile was 5.75, the third quartile 37, and the interquartile range 31.25.

4.5. Evaluation techniques

Beyond what was evaluated and with how many participants, a key question is how these evaluations were conducted. Usability only studies represent the largest category, employing a wide range of techniques. The System Usability Scale (SUS) is clearly the predominant standardized questionnaire. Furthermore, heuristic evaluation, particularly Nielsen’s, is extensively used. Several studies also complement with think-aloud protocols and task analysis (measuring time on task, completion rates, and error counts). Most of these results were expected, however, a significant number of studies rely on custom, unvalidated questionnaires, a practice that raises concerns about reliability and comparability.

On the other hand, UX-only studies demonstrate a strong preference for standardized instruments, with the User Experience Questionnaire (UEQ) and AttrakDiff as the most common standardized questionnaires. Other common questionnaires used in specific domains were Self Assessment Manikin (SAM) and game-specific instruments. Interestingly, physiological and biometric methods were commonly used when evaluating UX (e.g., eye tracking, EEG measurements). Another set of techniques applied in this group were more qualitative, including interviews and focus groups. Again, ad-hoc questionnaires were used in various studies.

Finally, in accessibility-only studies, automatic tools are predominantly used to check compliance with WCAG guidelines (e.g., WAVE, AChecker, TAW). Other types of tools such as screen reader and voice over were applied in some studies. Some studies combined automated and manual methods especially when the study included users with disabilities in their testing sample.

Regarding studies that combined several aspects, these have diverse methodological approaches, commonly using mixed methods (e.g., usability testing and automatic accessibility evaluation or UX self-reported and usability metrics). These studies commonly include experiments and more statistics analysis in their results. Finally, fully integrated studies evaluating all three dimensions employ diverse techniques commonly using user studies, questionnaires, observation and automatic evaluation of accessibility.

To systematically characterize the evaluation techniques employed across the 582 case studies, all methods and tools reported by each study were extracted and catalogued during the data extraction phase. Given that many studies applied more than one technique, the total number of method mentions (n = 1,444) exceeds the number of included studies, reflecting the multimethod nature of a substantial portion of the corpus. These mentions were normalized and grouped into canonical categories to ac-count for terminological variation across studies. Table 3 presents the 20 most frequently reported methods across all study types, together with their absolute frequency.

Table 3: Top 20 Evaluation Techniques Reported.

4.6. Title analysis

As complementary perspective, the titles of the 582 papers reveal important patterns in how researchers frame and communicate their UAUX evaluation studies. Of course, as a result of our query, the term evaluation is the most used (approximately 65% of the papers). Furthermore, a signif-icant majority of titles explicitly mention the application domain (e.g., education, Healthcare, E-Government).

Many titles also specify the evaluation methods employed, and a notable subset of titles specifies the user population under study specially for accessibility (e.g., deaf users, older adults, children with autism). Another interesting finding is that titles increasingly reference emerging technologies (e.g., virtual reality, artificial intelligence, chatbots or voice interfaces). This applies both to the tech-nologies evaluated and, in some cases, to the evaluation techniques applied.

5. Conclusions

Taking into account the findings, this section analyzes their implications and, at the same time, identifies possible directions for future research.

This systematic literature review of 582 case studies pro-vides a comprehensive overview of the practices, trends, and gaps in the evaluation of usability, user experience, and accessibility over the past decade. By analyzing a large and diverse dataset, this study offers a granular, empirical complement to previous secondary studies.

Our analysis confirms that usability remains the most common aspect of UAUX evaluation, with the System Usability Scale (SUS) solidifying its position as the de-facto standard instrument. The prevalence of usability-only studies suggests that for many practitioners, it remains the primary, and sometimes sole, focus of evaluation.

User experience is maturing as a distinct field, evidenced by a substantial number of UX-only studies and the common use of standardized questionnaires. However, the most significant finding is the persistent and pronounced underrepresentation of accessibility, despite growing legal and ethical imperatives for digital inclusion, accessibility is often treated as an afterthought or a separate, siloed concern rather than an integral component of design.

The study also reveals several persistent methodological challenges, for instance, the continued reliance on single method evaluations, in particular questionnaires and automated tools. Furthermore, the frequent use of adhoc, unvalidated questionnaires undermines the reliability and comparability of research findings.

The geographic concentration of research, with a significant portion emanating from a few countries and entire continents like Africa and Oceania being severely under-represented, limits the generalizability of findings and suggests a lack of diverse user perspectives in the literature. In response to the research questions, this review has shown that: (1) The most frequently applied methodologies are dominated by standardized usability and UX questionnaires and automated accessibility tools. (2) The primary dimensions measured continue to be the classic usability attributes of effectiveness, efficiency, and satisfaction, while UX studies capture a broader, more hedonic set of qualities. Methodologically, case studies are predominantly shortterm, small-to-medium scale evaluations that often fail to integrate all three UAUX aspects.

This research has several implications for practice and future research. For practitioners, it provides a guide to the most common and accepted evaluation methods, while also cautioning against the over-reliance on single methods and the use of unvalidated instruments, it advocates for a more integrated approach to evaluation. For researchers, the findings highlight a clear need for: (1) more comparative studies that triangulate findings from multiple evaluation methods; (2) the development and validation of new instruments that can holistically assess the interplay of UAUX; (3) a greater focus on longitudinal research designs; and (4) increased efforts to conduct and publish case studies from underrepresented regions and with diverse user populations.

Finally, a very interesting finding of this study is that only a small fraction of studies mentions artificial intelligence (AI), machine learning, or related techniques. This indicates that AI integration in UAUX evaluation is still in its early stages, as in most cases, AI appears as the object rather than the instrument of evaluation (i.e., AI-powered systems are being evaluated using traditional UAUX methods, but AI itself is rarely used as a tool for conducting UAUX evaluations).

The findings and implications of this research have limitations, that should be considered when interpreting the findings. First, the search was restricted (english language and three databases). Additionally, the exclusion of grey literature, means that real-world evaluation practices conducted outside academic settings are likely underrepresented in our findings.

The review is also subject to publication bias, as studies reporting positive or statistically significant usability findings are more likely to appear in peer-reviewed venues than those with null or inconclusive results. Furthermore, while dual-reviewer screening and consensus-based conflict resolution were applied throughout the selection process, a formal quality appraisal instrument was not systematically used to score the methodological rigor of each included study. Consequently, while dual-reviewer agreement was applied during the inclusion/exclusion screening phase, inter-rater reliability in terms of quality appraisal scoring was not formally measured and cannot be reported. Finally, the dataset covers publications from 2014 through 2023 and therefore represents a historical snapshot of the field.

Based on this analysis, several promising directions for future research emerge, including the development of more automated or personalized methods and AI-assisted techniques. Furthermore, longitudinal studies examining how user experience evolve are critically needed, as current snapshot evaluations fail to capture the dynamic nature of human computer interaction.

Citations by Dimensions

Citations by PlumX

Crossref Citations

1. Ignacio Diaz-Oreiro, Gustavo Lopez, “Beyond Written Surveys: Validating Voice-Based Implementations of the User Experience Questionnaire”, Journal of Engineering Research and Sciences, vol. 5, no. 6, pp. 15–26, 2026. doi: 10.55708/js0506002
2. Trupti Raikar, Vinil Apelagunta, “Implementing SAP Fiori in S/4HANA Transitions: Key Guidelines, Challenges, Strategic Implications, AI Integration Recommendations”, Journal of Engineering Research and Sciences, vol. 4, no. 11, pp. 1–9, 2025. doi: 10.55708/js0411001
3. Pasindu Udugahapattuwa, Shantha Fernando, “Content Recommendation E-learning System for Personalized Learners to Enhance User Experience using SCORM”, Journal of Engineering Research and Sciences, vol. 4, no. 9, pp. 30–46, 2025. doi: 10.55708/js0409004
4. Awais Ahmad, Peter Mozelius, “Human-Computer Interaction for Older Adults – a Literature Review on Technology Acceptance of eHealth Systems”, Journal of Engineering Research and Sciences, vol. 1, no. 4, pp. 119–126, 2022. doi: 10.55708/js0104014