Masterarbeit
Visual Analysis of User Interfaces: Integrating Analyzers for Improved UI Object Detection
Completion
2019/12
Research Area
Students
Juan Sebastian Quintero
Victor Flamenco
Advisers
Dr.-Ing. Sebastian Heil
Description
User Interfaces (UIs) change rapidly due to frequent changes in user requirements and evolving technologies. This can affect user-adoption negatively, creating the need for visual UI analysis. Manual analysis is time-consuming. Many automatic approaches for the analysis of Web User Interfaces (WUIs) are based on DOM parsing, rendering them dependent on the WUI implementation details and limiting feasibility to other types of UIs. Thus, a complementary vision-based approach using object detection techniques is required and should be combined with existing analyzers. The solution must accept UI screenshots as input and compute Regions of Interest, UI object types and confidence scores as output.
Current implementations of visual analysis of UIs do not reach sufficient object detection rates. Thus, the quality of UI object detection must be improved. To achieve this goal, this thesis investigates alternatives in the fields of Computer Vision, Deep Learning and Multi-Agent systems. The resulting system needs to combine improved vision-based analysis with existing a platform for integrating several DOM-based and vision-based analyzers to improve the overall quality of UI object detection for atomic and composite UI objects, measured in terms of precision and recall.
The objective of this thesis is to find an approach or a combination of approaches to solve the above problem in the context of UI object detection based on Computer Vision, Deep Learning and Multi-Agent systems. This includes the state of the art analysis regarding these three fields. The demonstration of feasibility with an implementation prototype of the concept is part of this thesis as well as a suitable evaluation on a representative test dataset including quality measurements.
