{"id":9776,"date":"2021-01-01T00:00:00","date_gmt":"2021-01-01T00:00:00","guid":{"rendered":"https:\/\/vsr.informatik.tu-chemnitz.de\/student-projects\/2021\/9776\/"},"modified":"2026-09-16T14:42:15","modified_gmt":"2026-09-16T14:42:15","slug":"ui-object-detection","status":"publish","type":"student_project","link":"https:\/\/vsr.informatik.tu-chemnitz.de\/edu\/studentprojects\/2021\/008\/","title":{"rendered":"UI Object Detection"},"content":{"rendered":"\n<div class=\"wp-block-vsr-blocks-header\">\n\t<h3 class=\"grey linie\">\n\t\tDescription\t<\/h3>\n\t<\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Detection of objects in User Interfaces (UIs) is an important pre-requisite for several analysis and frontend development tasks such as object-dependent UI metrics, UI Tests, Similarity Analysis and Layout transfer, UI generation, UI search etc. Approaches working on visual input, i.e. screenshots of user interfaces, are independent of implementation details such as specific frontend frameworks, platforms (web, mobile, desktop applications), do not require the availability of\/access to source code or dedicated accessibility APIs and can potentially better resemble human visual perception and understanding of composite UI elements.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The latest solutions based on Deep Neural Networks and Computer Vision achieve a prediction quality of about F1=.6 at an IoU threshold of .9. This increase of more than 40% over previous models is a significant improvement. However, to put the UI object detector into use, two challenges remain open: 1) the reported results need to be indepently verified through a replication study to test the claimed prediction quality and 2) the quality has to be further improved for the domain of web user interface analysis to allow integration into our visual analysis infrastructure. This thesis addresses the two challenges by replicating the experiments based on the available materials (code, pre-trained models for baselines and the highest-performing approach, datasets) and existing datasets of our professorship. Based on the results verification, the existing model should be further optimized for the specific domain of web user interfaces to increase the quality and therefore make the model more suitable as input for the computation of object-dependent metrics.<\/p>\n\n\n\n<div class=\"wp-block-vsr-blocks-header\">\n\t<h3 class=\"grey linie\">\n\t\tDescription (German)\t<\/h3>\n\t<\/div>\n\n\n\n<p class=\"wp-block-paragraph\">Die Erkennung von Objekten in Nutzerschnittstellen (UIs) ist eine wichtige Voraussetzung f\u00fcr verschiedene Analyse- und Frontend-Entwicklungsaufgaben wie beispielsweise objekt-abh\u00e4ngige UI Metriken, UI Tests, \u00c4hnlichkeitsanalyse und Layout-Transfer, UI Generierung, UI Suche usw. Ans\u00e4tze, welche visuelle Eingabedaten, d.h. Screenshots, verarbeiten, sind unabh\u00e4ngig von Implementierungsdetails wie spezifischen Frontend Frameworks, Plattformen (Web-, Mobile, oder Desktop-Anwendungen), setzen nicht die Verf\u00fcgbarkeit von bzw. den Zugriff auf Quelltext oder dedizierte Accessbility APIs voraus und k\u00f6nnen potentiell die menschliche visuelle Wahrnehmung und das Verst\u00e4ndnis von kompositen UI Elementen besser nachvollziehen.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Neuste L\u00f6sungen basierend auf Deep Neural Networks und Computer Vision erreichen eine Vorhersagequalit\u00e4t von ca. F1=.6 bei einem IoU Schwellenwert von .9. Diese Steigerung von mehr als 40% gegen\u00fcber vorhergehenden Modellen ist eine signifikante Verbesserung. Um jedoch den UI Ojekt Detector wirklich einsetzen zu k\u00f6nnen, bleiben zwei Herausforderungen offen: 1) die ver\u00f6ffentlichten Ergebnisse m\u00fcssen unabh\u00e4ngig durch eine Replikationsstudie verifiziert werden, um die behauptete Vorhersagequalit\u00e4t zu testen und 2) die Qualit\u00e4t muss weiter verbessert werden f\u00fcr die Dom\u00e4ne der web user interface analysis um die Integration in unsere bestehende visuelle Analyseinfrastruktur zu erm\u00f6glichen. Diese Arbeit geht die beiden Herausforderungen an durch die Replikation der Experimente basierend auf den vorhandenen Materialien (Quelltext, Vor-trainierte Modelle f\u00fcr die Baselines und den besten bekannten Ansatz, Datens\u00e4tze) und existierenden Datens\u00e4tzen unserer Professur. Basierend auf den Verifikationsresultaten soll das Modell weiter f\u00fcr die spezifische Dom\u00e4ne von Web User Interfaces optimiert werden um die Qualit\u00e4t zu erh\u00f6hen und das Modell dadurch geeigneter als Datenquelle f\u00fcr die Berechnung von objekt-abh\u00e4ngigen Metriken zu machen.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Detection of objects in User Interfaces (UIs) is an important pre-requisite for several analysis and frontend development tasks such as object-dependent UI metrics, UI Tests, Similarity Analysis and Layout transfer, UI generation, UI search etc. Approaches working on visual input, i.e. screenshots of user interfaces, are independent of implementation details such as specific frontend frameworks, [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":11377,"comment_status":"open","ping_status":"closed","template":"","meta":{"student_project_code_name":"","student_project_completion":"2021-12-17","student_project_deadline":"2021-09-29","student_project_id_legacy":"2021-008","student_project_archive_id_legacy":"","student_project_thesis_link_legacy":"","student_project_description_link_legacy":"","student_project_medium_legacy":"","student_project_pubdate_legacy":"2021-03-01","student_project_schemadata_legacy":"","student_project_second_version_title_legacy":"","migration_allowed":true,"breadcrumb_parent":413,"footnotes":"","vsr_identifier_year":2021,"vsr_sequence_id":8,"vsr_primary_id":557},"student_project_type":[],"student_project_status":[],"student_project_group":[],"student_project_research_area":[],"student_project_tag":[318,278,320,317,321,319],"class_list":["post-9776","student_project","type-student_project","status-publish","has-post-thumbnail","hentry","student_project_tag-computer-vision","student_project_tag-deep-learning","student_project_tag-hci","student_project_tag-object-detection","student_project_tag-ui-uix","student_project_tag-visual-analysis"],"_links":{"self":[{"href":"https:\/\/vsr.informatik.tu-chemnitz.de\/index.php?rest_route=\/wp\/v2\/student_project\/9776","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/vsr.informatik.tu-chemnitz.de\/index.php?rest_route=\/wp\/v2\/student_project"}],"about":[{"href":"https:\/\/vsr.informatik.tu-chemnitz.de\/index.php?rest_route=\/wp\/v2\/types\/student_project"}],"author":[{"embeddable":true,"href":"https:\/\/vsr.informatik.tu-chemnitz.de\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/vsr.informatik.tu-chemnitz.de\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=9776"}],"version-history":[{"count":17,"href":"https:\/\/vsr.informatik.tu-chemnitz.de\/index.php?rest_route=\/wp\/v2\/student_project\/9776\/revisions"}],"predecessor-version":[{"id":29921,"href":"https:\/\/vsr.informatik.tu-chemnitz.de\/index.php?rest_route=\/wp\/v2\/student_project\/9776\/revisions\/29921"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/vsr.informatik.tu-chemnitz.de\/index.php?rest_route=\/wp\/v2\/media\/11377"}],"wp:attachment":[{"href":"https:\/\/vsr.informatik.tu-chemnitz.de\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=9776"}],"wp:term":[{"taxonomy":"student_project_type","embeddable":true,"href":"https:\/\/vsr.informatik.tu-chemnitz.de\/index.php?rest_route=%2Fwp%2Fv2%2Fstudent_project_type&post=9776"},{"taxonomy":"student_project_status","embeddable":true,"href":"https:\/\/vsr.informatik.tu-chemnitz.de\/index.php?rest_route=%2Fwp%2Fv2%2Fstudent_project_status&post=9776"},{"taxonomy":"student_project_group","embeddable":true,"href":"https:\/\/vsr.informatik.tu-chemnitz.de\/index.php?rest_route=%2Fwp%2Fv2%2Fstudent_project_group&post=9776"},{"taxonomy":"student_project_research_area","embeddable":true,"href":"https:\/\/vsr.informatik.tu-chemnitz.de\/index.php?rest_route=%2Fwp%2Fv2%2Fstudent_project_research_area&post=9776"},{"taxonomy":"student_project_tag","embeddable":true,"href":"https:\/\/vsr.informatik.tu-chemnitz.de\/index.php?rest_route=%2Fwp%2Fv2%2Fstudent_project_tag&post=9776"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}