Robotic-supported Data Loss Detection in Android Applications

The gist: The work proposes R-DLD, a robot-assisted infrastructure for data loss detection in Android applications, offering less invasive and more realistic tests by interacting directly with smartphone sensors.

Introduction and Motivation

Smartphones present challenges in testing due to diverse hardware and operating system versions, and the need for non-invasive testing that mirrors end-user interaction In this context, robotic automation has emerged as an effective solution for addressing the diversity of smartphone systems and versions, and in performing some testing tasks non-invasively. Riganelli and colleagues introduced Data Loss Detector (DLD), a technique for revealing data loss defects in Android Inspired by DLD, and motivated by the need to test mobile devices in an environment that is as close as possible to the production setup, this paper introduces a robot-assisted infrastructure for data loss detection in Android applications called R-DLD (Roboticsupported Data Loss Detection). Unlike DLD, which simulates sensor inputs to induce device screen orientation changes, R-DLD takes a less invasive and more realistic approach by exercising the actual mobile phone sensors for the orientation changes.

R-DLD Infrastructure

The R-DLD infrastructure is a robotic device built on the Arduino platform [21], aiming to support the four types of smartphone orientations depicted in Figure 2 (Portrait, Inverted Portrait, Left Landscape, and Right Landscape). The robot was constructed using cost-effective materials, including an Arduino board, a stepper motor, a power source, and a chassis. A C-based program was implemented for a computer to communicate with R-DLD via a serial interface in order to position the smartphone in the desired orientation during the data loss test. The employed motor requires 200 steps for a full revolution, translating to 1.8 degrees per step, and specific numbers of steps were needed for each orientation (0, 50, 100, and 150 steps).

Methodology and Evaluation

The empirical evaluation involved a conceptual replication of the original DLD study [26] by adapting the research questions to our new context with the robotic infrastructure. The data collection process was performed using web scraping techniques to obtain application details, and an analysis of the Manifest was used to check for screen orientation restrictions. The experiment employed 77 randomly selected apps from an Android store, and 2,250 events were carried out per app with real rotations. The processing time was observed to be 50% longer (4.5 hours) than the simulator (AVD) study time of 3 hours.

Results and Analysis

The evaluation identified 341 data loss issues in 77 randomly selected apps from an Android store. All reported bugs received responses from the developer, with 89.55% confirming the data loss problems, while 35.82% have been subsequently fixed. The analysis of results showed that 2,160 (60.2%) alerts were classified as True Bugs and 1,429 (39.8%) were classified as False Alarms. When considering all applications, the R-DLD achieved 1.38 True Bugs for each False Alarm.

Conclusion and Relevance

The work demonstrates that the data loss problem is relevant to developers, as 47 issues had data loss bugs confirmed among those addressed, representing 89.55% of the addressed bugs. The proposal is more realistic because the orientation changes are triggered by the physical rotation of the smartphone, thus interacting with the sensors as in a real usage scenario. Future work aims to investigate replacing the screenshot oracle with computer vision techniques and using binary classification algorithms to categorize alerts into True Bugs or False Alarms. The artifacts produced during R-DLD execution, such as screenshots before and after the double rotation, were included to facilitate developer reproduction of the issue.

Feasibility

The handmade robotic arm built as part of this work costs approximately 60 dollars, which is much lower than acquiring an industrial robot. Integrating the automated testing tool with the robot is a relatively simple task that involves intercepting ADB commands and replacing them with robotic instructions. The speed of the robotic arm was verified to be less than 2 seconds for each orientation change, which is sufficient to avoid interfering with the oracle verification. The study highlights that an AVD has limitations, such as not being able to run tests involving phone calls or Bluetooth connections.

References

[1] Domenico Amalfitano, Vincenzo Riccio, Ana C. R. Paiva, and Anna Rita Fasolino. 2018. Why does the orientation change mess up my Android application? From GUI failures to code faults. Software Testing, Verification and Reliability 28, 1 (2018), e1654

[2] Android. 2022. The Activity Lifecycle. https://developer.android.com/guide/components/activities/activity-lifecycle [Online; accessed 12-February-2022].

[3] Android. 2023. Handle configuration changes. https://developer.android.com/guide/topics/resources/runtime-changes [Online; accessed 12-February-2022].

[4] Arduino. 2024. AccelStepper - Arduino Reference. https://www.arduino.cc/reference/en/libraries/accelstepper/

[5] Debdeep Banerjee and Kevin Yu. 2018. Robotic Arm-Based Face Recognition Software Test Automation. IEEE Access 6 (2018), 37858–37868 https://doi.org/10.1109/ACCESS.2018.2854754

[6] Debdeep Banerjee and Kevin Yu. 2020. 3D Face Authentication Software Test Automation. IEEE Access 8 (2020), 46546–46558 https://doi.org/10.1109/ACCESS.2020.2978899

[7] Debdeep Banerjee, Kevin Yu, and Garima Aggarwal. 2018. Hand Jitter Reduction Algorithm Software Test Automation Using Robotic Arm. IEEE Access 6 (2018), 23582–23590 https://doi.org/10.1109/ACCESS.2018.2846761

[8] Debdeep Banerjee, Kevin Yu, and Garima Aggarwal. 2018. Image Rectification Software Test Automation Using a Robotic ARM. IEEE Access 6 (2018), 34075–34085 https://doi.org/10.1109/ACCESS.2018.2873284

[9] Lucas Cabral, Breno Miranda, Igor Lima, and Marcelo d’Amorim. 2022. RVprio: A tool for prioritizing runtime verification violations. Software Testing, Verification and Reliability 32, 5 (2022), e1813 https://doi.org/10.1002/stvr.1813

[10] Mihai Craciunescu, Stefan Mocanu, Cristian Dobre, and Radu Dobrescu. 2018. Robot Based Automated Testing Procedure Dedicated to Mobile Devices. In 2018 25th International Conference on Systems, Signals and Image Processing (IWSSIP). 1–4 https://doi.org/10.1109/IWSSIP.2018.8439614

[12] Android Developer. 2023. Android Debug Bridge (ADB). https://developer.android.com/tools/adb

[13] Davi Freitas, Breno Miranda, and Juliano Iyoda. 2024. RFNIT: Robotic Framework for Non-invasive Testing. In Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering https://doi.org/10.1145/3663529.3663871

[14] Wunan Guo, Zhen Dong, Liwei Shen, Wei Tian, Ting Su, and Xin Peng. 2022. Detecting and Fixing Data Loss Issues in Android Apps. In ISSTA 2022 https://doi.org/10.1145/3533767.3534402

[15] Wunan Guo, Zhen Dong, Liwei Shen, Wei Tian, Ting Su, and Xin Peng. 2022.

1. A robotic-supported infrastructure for data loss detection (R-DLD) can be implemented to perform less invasive and more realistic tests by interacting directly with smartphone sensors, enabling testing in an environment as close as possible to the production setup.

2. The improved system can identify data loss defects by exercising the actual mobile phone sensors for the orientation changes, moving beyond software simulation to uncover issues missed by traditional methods.

3. The system can be structured to evaluate a broader range of applications, specifically by incorporating filtering mechanisms based on the AndroidManifest.xml analysis, such as discarding apps with complete blocking of the application orientation or those with specific configuration instructions like configChanges.

4. The AI system can generate a comprehensive replication package that includes artifacts like all artifacts produced during our work, facilitating independent verification and replication of results, thereby increasing the transparency and reproducibility of the findings.

5. The system can utilize binary classification algorithms to categorize alerts into True Bugs or False Alarms, potentially improving the manual analysis phase by applying filters to efficiently group captures for more targeted investigation.

Smartphones have become integral to modern life due to their diverse applications. However, the development of applications for these devices faces significant challenges, primarily due to two factors: (i) the diverse hardware and operating system versions that require testing across multiple configurations; and (ii) the imperative activity of testing the smartphone in a non-invasive way (similar to an end-user interaction), which is expensive, tedious and error-prone as it is usually carried out manually. In this context, robotic automation has emerged as an effective solution for addressing the diversity of smartphone systems and versions, and in performing some testing tasks non-invasively. In addition, automated robots facilitate quick, accurate, and repetitive testing, thus enabling developers to validate their applications across various configurations effectively. This results in a less invasive verification and in a significant reduction in time spent on manual testing, thus accelerating the development cycle. Our work proposes R-DLD (Robotic-supported Data Loss Detection), a robot-assisted infrastructure for data loss detection in Android applications, offeringless invasive and more realistic tests by interacting directly with smartphone sensors. The robot is constructed using cost-effective materials, facilitating its adoption in testing environments. In our empirical evaluation R-DLD successfully identified 341 data loss issues in 77 randomly selected apps from an Android store. All reported bugs received responses from the developer, with 89.55% confirming the data loss problems, while 35.82% have being subsequently fixed.