Screen Tracking

Screen tracking allows researchers to see what happens on their respondents’ screens and learn more about individual experiences on social media platforms. Regardless of whether you are researching the content of news feeds or the way people interact with different types of visual information, screen tracking provides an opportunity to investigate how platforms structure content and deliver it according to personal preferences.

It is a powerful method that allows for working with individual participants. It captures the specific content on platforms, including but not limited to TikTok, Facebook, Instagram, and YouTube.

How does screen tracking work?

The method is structured around using an app for participants’ screen tracking and further processing participants’ screen recordings using computational techniques and a machine learning algorithm.

  1. At the initial stage, a researcher designs a study and obtains ethics approval. After that, they are registered with the AIO system and issued credentials that will be used to register participants and collect the data.

  2. Participant App Installation: Using the credentials provided by the researcher, respondents download, install and configure their app (Android only). That includes granting necessary permissions such as screen recording and tracking app usage.

  3. App Activation: The app runs quietly in the background. When a participant opens a pre-defined target app (e.g., Facebook), the tool activates automatically and asks for permission to begin capturing the screen.

  4. Screen Capture: The participant continues using their phone as they normally would, while a pre-trained vision model (YOLOv11) runs within the app and captures the specific content relevant to the study and specified by the researcher. The model ignores or discards any other content, such as personal posts and messages.

  5. Data Transfer and Processing: Once data collection is complete, the app stops or is closed by the user, while the collected videos are further transferred to the cloud for processing. The data is then extracted from the capture, automatically categorised, sorted, and delivered in the form of a standardised dataset.

  6. Consent and Researcher Access: The participant is then able to see the dataset and agree to share it with the researcher, who then can examine it or export it into a file for further analysis.

What are the advantages of screen tracking?

As a tool, screen tracking allows to capture personalised curated social media feeds of different demographic groups. These feeds are based on individual preferences and the black box algorithms of social media platforms that are not publicly available. Looking at these algorithms using the eyes of the respondent is a reliable way to understand how platforms function.

Screen tracking also presents the opportunity to study the real user experience within platforms’ native environments, accompanied by all interactive elements as they appear on the screen. Furthermore, the method facilitates access to closed and visually-driven platforms, where traditional data collection methods fail. The data collected is not just text; it is pictures, reels, and ads that provide context for qualitative and quantitative analyses or mixed methods.

What research can screen tracking support?

Screen tracking has been used by ADM+S and affiliated projects, some of which are described as case studies on the Ad Observatory page. The method has been used to explore the kinds of gambling and alcohol advertisements young people are exposed to and assess the content of ads during elections. The app can also be leveraged to understand how misinformation can be visually spread through recommendation algorithms, what trends are popular within online communities, and monitor platforms’ policy compliance in terms of content moderation.

Screen tracking can be used to identify content relevant to a specific study. It maintains cross-platform capability and provides ethical access to data. If you want to learn more about this method or have a project that can potentially benefit from this form of data donation, please contact us.