How to build APP testing system from zero? The following is a sorting out of my experience. I will share my thoughts, the challenges and solutions I have encountered when I build application testing system in my company.

1. Automation testing

Automation testing mainly includes UI automated functional testing, interfaces automated testing and other special automated testing.

● UI functional automation testing

UI functional automation testing is mainly about the automation testing on the UI interface. The clicks on the UI interface are realized through scripts instead of manual work.

Automated testing on UI functions can effectively reduce the manpower investment of highly repetitive functional testing in interface features. The fast and efficient regression for UI function can be realized by using the script.

However, the shortcomings of this type of testing are obvious, including high maintenance costs, tendency to misjudging, and compatibility challenges.

The test is based on interface operation, therefore, the interface stability becomes the biggest constraint on maintaining the scripts. Frequent changes in interface interaction mean that test case scripts need to be updated constantly, which takes up a lot of testing resources.

The vulnerability to misjudge is mainly due to the fact that recognition based on UI controls is prone to slow loading or abnormalities due to network conditions, device configuration, test environment, etc., which leads to inaccurate judgments during the execution of test cases and thus affects the accuracy of the tests.

The compatibility problem refers to the unpredictability of test scripts executed on different devices, different operating systems and different hardware, resulting in inaccurate test results.

According to the above comparison of advantages and disadvantages, we mainly implement the testing of the core path of the APP in UI functional automation testing, and carry out UI functional automation testing for functional modules that require a lot of repetitive execution, repetitive verification and low frequency of UI interface changes.

The repetitive execution and repetitive verification means high utilisation of the test scripts, and the low frequency of UI interface changes means low maintenance costs. These types of use cases are with high ROI and should be given the highest priority to the practice of UI functional automation testing.

In the process of UI functional automation testing, you can sort out and manage the related controls, test cases and test sets, and consolidate the repeatable work in time to reduce the waste of resources. When UI functionality changes, the maintenance costs can also be reduced.

● Interface automation testing

In the section on UI functional automation testing, we mentioned the constraint of automation: stability. Because of the instability of the UI interface, the cost of UI functional automation testing is relatively high. Therefore, the interface, which is more stable, is more suitable for automation testing.

The interface of an app may change due to the different requirements of the product manager at different stages, but the interface is usually more stable, which is a good guarantee for us to carry out automation testing.

We need to sort out and prioritise the interface called by the application according to their functional modules, understand what each interface represents, the range of values of different parameters, the different inputs and outputs, and summarise the return of errors or exceptions, so as to ensure the effectiveness and completeness of interface testing.

After the start of the interface automation test, you need to work and share with the development engineers to maintain an interface document so that the test engineers can get access to any subsequent additions or reductions in interfaces or changes to existing interfaces and make adjustments to the interface automation test cases accordingly.

● Other special automation testing

In addition to the above automation testing, we can also use automation to do some special testing to help improve our testing quality and testing efficiency. We need to think about which tasks can be achieved through automation, which tests can be automated to improve testing efficiency, which function points can be automated to achieve long-term test monitoring, etc. In our daily testing work.

In short, we can use a variety of automated testing tools and test strategy to assist us in testing.

2. Performance Testing

In the testing system of the project I am responsible for, performance testing mainly includes three dimensions, performance testing in the time dimension, performance testing in the resource dimension and fluency testing.

● Time dimension

Performance testing in the time dimension refers to the response time of functional features after a click. We are familiar with the first screen loading time and the response time after a click to jump

There are various ways to perform performance testing in the time dimension, such as using screenshots, time stamps, third-party scripts, or image recognition technology to calculate time.

In the testing process, we have to pre-develop the tool in the context of the project, whether it is a one-off test, or whether it needs to be tested subsequently, whether it needs to be developed into a tool for subsequent long-term use, whether it is to be used on a single device, or whether it needs to be compatible on different devices, whether the tool needs to be open source or provides a data interface for subsequent integration with the team's testing platform, and so on.

● Resource dimension

The performance testing in the resource dimension mainly refers to the consumption of various system resources during the use of APP, including CPU, memory, power, traffic and so on.

The choice of testing tools is based on the different testing terminals, and the dimensions that need to be monitored are also determined according to the project.

The performance testing in the resource dimension can be done in two parts, one is the performance testing during the testing process and the other is the collection of online performance data.

Performance testing in the testing process can be assessed according to business testing requirements, which scenarios need to be tested, whether the current version is tested once, or each subsequent version has to be tested for comparison, whether only the performance data of the local devices needs to be tested, or whether the performance data needs to be collected on multiple devices, whether only this APP needs to be tested, or whether it needs to be tested against competing products, etc.

Based on this, you need to assess whether test cases need to be implemented through automation scripts for subsequent re-use. If the longitudinal and historical version comparison testing is required, you need to ensure that the test environment and test devices are as consistent as possible, so that the test results are more realistic and reliable.

In addition, the processing and calculation of test data can be performed by automation scripts which help to save the cost of manpower. If necessary, a platform can be built to store test data for subsequent analysis and access.

The collection of online performance data requires the development engineer to report the relevant data during the implementation of the function, and after the function is released, the online data will be fished, processed and calculated to find out the potential problems.

● Fluency testing

Fluency testing is the most intuitive expression of the user experience and is a must for performance tests. There are a few points to note about the approach to fluency testing.

Firstly, how do we plan the use cases for the fluency test. Secondly, how to use the test results to analyse and improve. Thirdly, how do we monitor the fluency according to the online data after the app is released.

Regarding the design of fluency test cases, it is necessary to combine the core functions of the APP and the common paths of users, which need to be supported by online data. The majority of users' jumping paths in the app are what we need to focus on. In addition, the paths that are prone to lagging also need to be paid attention to during the testing process.

The analysis and use of data of fluency testing and the monitoring of online data require the test engineer and development engineer to work together to plan and troubleshoot.

3. Stability testing

This part can be carried out in two phases, the testing phase before the release of the app and the online operation phase after the release.

In the testing phase, we can carry out stability testing around Monkey testing and code walks. The static code scanning tools can also be used in this phase.

During the Monkey testing process, we should focus on the devices, environment and frequency of test execution, and perform analysis of the problems found in the process.

For code walks, you can combine with the modules that are prone to crashes during functional testing, work with the development engineers to pair programming and check for possible problems.

As for static code scanning, it is necessary for development engineers to address the problems scanned and keep good habits to avoid relevant problems.

In the operations phase, we can carry out stability testing based on the analysis of the reported crash data from the external network. This relies more on the development engineers, but in this process, the test engineers can analyse the reported data and locate the basic crashed data, such as common systems and models, so as to improve and optimise the daily stability testing.

4. Summary

For the common APP testing system, the main testing is automation testing, performance testing and stability testing, in addition, there are other aspects, including security testing. When starting to build an APP testing system, based on the project, we need to combine the current state and phase of the project, the problems encountered, the team members and other aspects to confirm the priority of the relevant work, so as to gradually promote the building of the testing system.