Ensuring Web Interface Quality through Usability-based Split Testing

Authors: Maximilian Speicher, Andreas Both (R&D, Unister GmbH), Martin Gaedke

Introduction

Usability is a crucial quality aspect of web applications, as it guarantees customer satisfaction and loyalty. Yet, effective approaches to usability evaluation are only applied at very slow iteration cycles in today's industry. In contrast, conversion-based split testing seems more attractive to e-commerce companies due to its more efficient and easy-todeploy nature. We introduce Usability-based Split Testing as an alternative to the above approaches for ensuring web interface quality, along with a corresponding tool called WaPPU. By design, our novel method yields better effectiveness than using conversions at higher efficiency than traditional evaluation methods. To achieve this, we build upon the concept of split testing but leverage user interactions for deriving quantitative metrics of usability. From these interactions, we can also learn models for predicting usability in the absence of explicit user feedback. We have applied our approach in a split test of a real-world search engine interface. Results show that we are able to effectively detect even subtle differences in usability. Moreover, WaPPU can learn usability models of reasonable prediction quality, from which we also derived interaction-based heuristics that can be instantly applied to search engine results pages.

Resources

• Complete list of interaction features and complete user study results: PDF
• Complete correlations for the original search engine results page (context HD / not familiar): CSV
• Individual usability models per usability item (context HD / not familiar):
  • to be used with WEKA
  • Informativeness: training data, model, test set.
  • Understandability: training data, model, test set.
  • Confusion: training data, model, test set.
  • Distraction: training data, model, test set.
  • Readability: training data, model, test set.
  • Information density: training data, model, test set.
  • Accessibility: training data, model, test set.

Acknowledgments

This work has been carried out in cooperation with Unister GmbH, Leipzig and was supported by the ESF and the Free State of Saxony.