Comparative Reliability Analysis of Selenium and Playwright: Evaluating Automated Software Testing Tools

Abstract

Aims: This study aims to evaluate and compare the reliability of Selenium and Playwright, two leading frameworks for automated web testing. The assessment focuses on key reliability metrics, including uptime and the Rate of Occurrence of Failures (ROCOF). Study Design: A comparative experimental study conducted under controlled testing conditions. Place and Duration of Study: The study was conducted over a 24-hour continuous testing period using two laptops with distinct hardware configurations: an HDD-equipped HP laptop and an SSD-equipped Dell laptop. Methodology: Reliability was measured using two core metrics: uptime and ROCOF. Tests were conducted on an HP laptop (HDD) and a Dell laptop (SSD). Two Python scripts — one for Selenium and one for Playwright — were developed to execute identical actions. For the 24-hour uptime test, Selenium ran on HP and Playwright on Dell. ROCOF was assessed at six-time intervals — 8:00 AM, 8:15 AM, 4:00 PM, 4:15 PM, 12:00 AM, and 12:15 AM — by alternating tool execution between HP and Dell, allowing for analysis of the effects of hardware and time of day on failure rates. Results: Selenium achieved 100% uptime with no failures, while Playwright recorded 99.72% uptime with four downtimes. For ROCOF, both tools had one failure per 10-test sequence, but Selenium’s higher failure rate per second (0.1208 on HP, 0.1336 on Dell) was due to faster execution times (7.93s on HP, 7.87s on Dell) compared to Playwright (36.74s on HP, 35.84s on Dell). The SSD-equipped Dell laptop outperformed the HDD-based HP, with faster completion times (43.71s vs. 44.67s). Conclusion: Selenium is ideal for scenarios requiring uninterrupted uptime, while Playwright's consistent response times suit dynamic web application testing. The study highlights hardware's role in performance, with SSDs offering superior speed and stability. These findings guide practitioners in choosing tools based on hardware, stability, and execution needs.