Context

In software development projects, there are often hidden effectiveness leaks that can significantly impact productivity and product quality. One such leak is the lack of automated testing, particularly for integration and end-to-end scenarios. This issue is compounded by the difficulty in convincing project owners of the business value of implementing comprehensive automated tests.

‍

Problem Identification and Consequences

The main challenge is the prevalent difficulty of pinpointing the the business value in adding automated tests, especially for time-consuming test types like integration or end-to-end tests. This results in:

1. Reliance on manual "Sanity Check" tests, consuming 4-5 hours per release candidate.
2. Increased risk of regressions and bugs in important features.
3. Limited coverage of complex scenarios involving multiple external services.
4. Difficulty in justifying the time and resources needed for implementing robust automated tests.

The consequences of not addressing this issue included potential quality issues, slower release cycles, and inefficient use of developer and QA time.

‍

Solution Implementation

To address this challenge, in one of our projects, we implemented a multi-faceted approach:

1. Started with simple, high-value test scenarios (e.g., account creation) to demonstrate quick wins.
2. Developed a comprehensive reporting system that provided human-readable test results, including:
   • Release notes grouped by fixes and features
   • Sanity check results for 13 key test scenarios
   • Detailed UI test reports with descriptions, steps, and assertions
   • Screen snapshots for manual design review
3. Integrated the reporting system with the CI/CD pipeline, automatically generating and sharing reports via Confluence and Slack.
4. Gradually expanded test coverage to include more complex scenarios.
5. Invested approximately 500 hours in implementing these automated tests over time.

‍

Business and Product Gains

The implementation of this solution led to several significant improvements:

1. Time savings: Eliminated the need for 70 hours of manual testing per month across iOS and Android platforms.
2. Improved product stability: Comprehensive automated testing reduced the risk of regressions and bugs.
3. Faster release cycles: Automated "Sanity Checks" allowed for quicker validation of release candidates.
4. Enhanced developer knowledge: The process of creating tests improved the team's understanding of the system and its edge cases.
5. Better handling of complex scenarios: Automated tests could cover intricate workflows involving multiple external systems.
6. Improved communication: The new reporting system provided clear, understandable test results to both technical and non-technical stakeholders.
7. Long-term cost savings: While the initial investment was significant, it began to yield returns after about 10 months.

By focusing on proving the value of automated tests through tangible results and clear communication, we were able to overcome the initial resistance and create a more efficient, higher-quality development process.

‍