Friess, Erin. “Discourse Variations between Usability Evaluation and Usability Reports.” Journal of Usability Studies 6, no. 3 (2011): 102-16.
Erin Friess evaluates the differences between usability evaluations and usability reports. She investigated the usability evaluation process and compared the results against the language of usability reports and established that many usability evaluators modified the language and outcomes of the usability studies in their reports. Friess suggests several reasons for this discrepancy and calls for further research into the topic.
Friess used a comparison of language used in end-user usability participants with that of the language in the usability reports submitted by the usability evaluators. After examining her results, Friess established that:
- 84% of findings had some basis in the usability evaluation
- 16% of findings had no basis from the usability evaluation
Of the 84% of findings that had some basis:
- 55% were accurate findings
- 29% were potential inaccurate.
In each category, soundbites and interpretation were the key attributors to the evaluator’s findings. From her results, Friess suggests a number of reasons for the discrepancies found between usability evaluations and usability reports.
- Confirmation bias in oral reports
- Bias in what’s omitted in the usability reports
- Biases in Client desires
- Poor interpretative skills
In each of these categories, biases are natural and are to be expected. Adequate training should be able to correct the impact biases have. Poor interpretative skills, however, are a challenge to overcome. Poor interpretative skills lead to biases, as those conducting usability tests will guide end-users towards predetermined conclusions if they anticipate potential issues or chose to interpret end-user behavior in line with what they expect to see. Educating usability testers on how to interpret end-user comments, behavior, and questions will go a long way to reducing the discrepancy between end-user usability evaluations and usability evaluator reports.