The mobile application test has its own characteristics, which are different from other traditional tests, and there are some unique test methods and ideas that we want to share today.

Mobile applications can be divided into three categories.

- Web APP

- Native APP

- Hybrid APP

Web App refers to the web browser on the mobile side, which is actually no different from the web browser on the computer side, except that the operating system that the web browser use is no longer Windows and Linux, but iOS and Android.

The technologies used by Web App are mainly traditional Web technology stacks such as HTML, JavaScript, CSS, etc. Of course, now HTML5 is also widely used. In addition, the content of the pages accessed by WebApp are all placed on the server-side, which is essentially a Web page, so it is inherently cross-platform.

Native App refers to the native app on the mobile terminal, which is apk for Android and ipa for iOS. NativeApp is a third-party application based on mobile phone operating systems (iOS and Android), written and run using native programs.

For Native App development, the language used by Android is usually Java, and the language used by iOS is Objective-C. Generally speaking, Native App can provide a better user experience and performance, and can conveniently operate the phone’s resources.

Hybrid App is an App form between Web App and Native App. Hybrid App uses the advantages of the Web App and Native App to display HTML5 pages through a natively implemented NativeContainer. In a more common way, it can be boiled down to embedding a Webview in a native mobile application and then accessing web pages through the Webview.

Hybrid App has the features of simple maintenance and update, excellent user experience and better cross-platform. It is the current mainstream mobile application development model.

According to the characteristics of different types of applications, we summarize their test methods as follows.

● Web App Testing

Obviously, its essence is the testing of Web browsers. All GUI automated testing methods and technologies, such as data-driven, page object model(POM), business process encapsulation, etc., are all suitable for Web App testing.

If the web page is based on adaptive web design (it conforms to the Responsive Web design specification), and the test framework supports Responsive Page, then in principle, the GUI automated test cases that were developed before and run on the PC Web can be directly used on the browser of the mobile terminals without any modification with the premise that your mobile browser must support WebDriver. Responsive Web Design refers to a web design technology that can automatically recognize the screen resolution of the same web page and make corresponding adjustments.

● Native App Testing

Native App testing, although different platforms will use different automated testing solutions, iOS generally uses XCUITest Driver, while Android generally uses UiAutomator2 or Espresso, etc., but the data-driven, page object model, and business process encapsulation still apply.

● Hybrid App Testing

Hybrid App testing will be a little more complicated. For Native Container testing, native testing frameworks such as XCUITest or UiAutomator2 may need to be used. HTML5 testing in Container is basically the same as traditional web page testing. GUI-based testing ideas and methods can continue to apply.

The only thing to note is that Native Container and Webview belong to two different contexts respectively. The default Context of the Native Container is “NATIVE APP”, and the default Context of Webview is “WEBVIEW_+ Process name under test”.

Therefore, when you need to manipulate the web page elements in the Webview, you need to switch to the Context of the Webview first.

The difference between Web testing and APP testing

● Similarities

There is no difference between WEB testing and App testing in terms of process. All include test planning, use case design, test execution, defect management, and test reports. Technically speaking, the test types of WEB test and APP test are basically similar, and both test types such as functional test, performance test, security test, and GUI test are required.

● Differences

Their main difference lies in the test details and methods.

1.Performance testing

In WEB testing, only the response time is required. While in App testing, traffic testing and power consumption testing are also considered.

2.Compatibility testing

On the WEB side, it is compatible with different browsers while on the App side, it is compatible with mobile devices. Therefore, the corresponding compatibility test tools are different. For WEB testing, you need to use different browsers for compatibility testing (commonly compatible with chrome, firefox, Edge, etc.). If it is a mobile phone, then It needs to be compatible with different mobile terminals brands, different resolutions, different Android versions and even different operating systems.

3.Installation test

For Web test, basically, there is no installation test required, but for APP test, it is necessary.

As App testing is based on mobile devices, there are some special tests for mobile devices. Such as conflict test, operational test, network test (weak network test, network switching).

Conflict test: when operating a certain software, there are external events such as incoming calls, text messages, low battery reminders, etc.

Operational test: such as horizontal scroll test, gesture test.

Network test: including weak network and network switching test. Need to test the user experience caused by the weak network, the key is to consider whether the rollback and refresh will cause a second submission.

Upgrade test: the reminder mechanism of the upgrade test, whether the cancellation of the upgrade will affect the use of the original functions, and whether the user data is cleared after the upgrade.

In terms of system architecture, for Web test, as long as update the server-side, the client-side will be updated synchronously. And the client-side of each user is completely consistent. However, for the APP test, we can not make sure of the complete consistency unless the user updates their own. If the server-side of an APP is modified, then a regression test of the core version used by the users is a must.

From this point of view, in addition to the different test frameworks used in mobile testing, the test design itself and GUI testing have the same effect. There should be other different test ideas and methods for the mobile terminal.

Special testing for mobile application

For mobile applications, it is often not enough to successfully complete all business function tests. When mobile applications are installed and used by a large number of users, many problems that were not anticipated at all will be exposed.

For example:

Excessive traffic usage

Excessive power consumption

Crash or flashback on some equipment terminals

Abnormal behavior after switching between multiple mobile applications

Unable to install or uninstall smoothly on some equipment terminals

Unable to be used normally in a weak network environment

The frequent occurrence of ANR(Application Not Responding) in the Android environment

...

There are many similar problems. In order to avoid or reduce the occurrence of such situations, many special tests specific to mobile applications are conducted in addition to regular functional tests.

1. Conflit test

Conflict test is also called interrupt test, which refers to the test that other events or applications interrupt the current operation while using the app.

For example, when the App is running, a phone call comes in suddenly, or a text message is received, or it is a system alarm. Therefore, during App testing, you need to take these common interruptions into account and conduct related tests.

This type of testing is basically manual testing, and they are all carried out on a real machine, without using an emulator.

The reason for using manual testing, first of all, is that such tests often have many scenarios, and many events are difficult to simulate through automated methods, such as incoming calls and receiving text messages. These factors will cause the cost of automated testing to be too high and the gains outweigh the losses. Therefore, cross-event testing is often based on manual testing.

Secondly, the reason for using the real machine is that many problems can only be reproduced on the real machine, and it is meaningless to use the simulator to test.

For cross-event testing, the scenarios that need to be covered mainly include

When multiple apps are running backstage at the same time, will it affect normal functions if switching to the foreground alternately?

Does the alternate switching of multiple apps that require the same system resources affect normal functions, for example, there are two apps that need to play music, then during the alternate switching process, whether the music playing function is normal?

Answer calls while the app is running

Receive information when the App is running

System upgrade prompt when App is running

A system alarm event occurs when the App is running

Switch to low battery mode when the App is running

The third-party security software pops up an alarm when the App is running

A network switch occurs while the app is running, for example, switching from Wifi to 4G network, or from 4G network to 3G network, etc.

In fact, you can find that the scenarios that need to be covered are also the set of test cases that we will test in the future. Each scenario is a set of test cases.

2. Compatibility test

Compatibility test, as the name implies, is to ensure that the App is functionally correct under various terminal devices, operating systems, screen resolutions, and various network environments. Common App compatibility tests often need to cover the following test scenarios:

Compatibility of different operating systems, including Andoird and iOS;

Compatibility under mainstream device resolution;

Compatibility of mainstream mobile terminal models;

Compatibility with different language settings in the same operating system;

Compatibility under different network connections, such as Wifi, GPRS, EDGE, CDMA200, etc.;

On a single device, compatibility with mainstream popular apps, such as Facebook，Instagram, Youtube, Tiktok, etc.

...

Usually, the same or similar test cases are conducted on various real machines for compatibility testing, so automated testing is a common method. Due to the requirement to cover a large number of real machine, except large companies that build their own private mobile device could based on Appium + SeleniumGrid + OpenST, other companies generally use third-party mobile device cloud to complete compatibility tests, such as 51Testing.

3. Flow Test

Since apps often need to run in a mobile Internet environment, and mobile Internet is usually billed according to actual usage, if your app consumes too much traffic, it will increase user traffic charges and decrease function loading.

Traffic test usually includes the following aspects.

The traffic of the business operations performed by the App

The consumption of traffic when the App is running backstage

The traffic of the first startup after the App is installed

The size of the App installation package itself

The traffic required for in-app purchases or upgrades

Traffic testing often uses tools that come with Android and iOS and you can also use network analysis tools.

For the Android system, network traffic information is usually stored in the /proc/net/dev directory, or you can directly use ADB tools to obtain real-time traffic information. Emmagee, a lightweight performance monitoring tool for Android, is similar to the Windows system performance monitor, which can display information such as CPU, memory, and traffic in the App running process in real-time.

For the iOS system, you can use the Network Activity in the performance analysis toolset that comes with Xcode to analyze specific traffic usage.

However, the ultimate goal of the traffic test is not to obtain the traffic data of the App, but to find ways to reduce the traffic generated by the App. It is not the job of the tester to reduce the traffic, but understanding some common methods will also help your daily testing work.

Use data compression, especially for pictures.

Use optimized data format, for example, a JSON file with the same amount of information is smaller than an XML file.

When requiring both encryption and compression, it must be compressed first before encrypted.

Reduce the number of backstage calls triggered by a single GUI operation.

Only the necessary data is returned as much as possible.

Start using the client’s caching mechanism.

4. Power consumption test

Power consumption is one of the key factors for the success of a mobile application. Nowadays, there are many apps that can provide similar services or functions. If the app consumes much more power and makes the device heat up, then your users will definitely uninstall your app and change it into another one. The most typical are navigation applications such as maps, which are particularly sensitive to power consumption.

The power consumption test usually covers the following three aspects.

when the App is running but not performing business operations;

when the App is running and intensively performing business operations;

when the App is running backstage.

There are hardware-based methods and software-based methods for power consumption tests.

Based on the software method, Android and iOS have their own methods. Android uses the adb command adb shell dumpsys battery to obtain the power consumption information of the application. In the power consumption test, the History Batterian tool launched by Google is good for analyzing the power consumption.

IOS collects power consumption information through Apple’s official tool Sysdiagnose, and then can further analyze power consumption through Energy Diagnostics in the Instrument toolchain.

5. Weak network test

Different from traditional desktop applications, the network environment of mobile applications is more diverse, the scenarios of switching between different networks occur frequently. Even in the same network environment, the network connection status will sometimes be good and bad. Under the scenarios of riding the subway, passing through tunnels, and staying underground garages, fluctuating latency, frequent packet loss, and frequent disconnection often occur.

Therefore, the testing of mobile applications needs to ensure the quality in a complex network environment. Try to simulate the different network environments, find out and fix as many problems as possible before the APP is released.

The best use of ATC is that it can switch different network environments at any time through the web interface on mobile terminal devices. At the same time, multiple mobile terminals can connect to the same WiFi, and each simulates different network environments without any impact on the other. In other words, a set of ATC can meet all your network simulation requirements.

6. Boundary test

Boundary testing refers to the verification test of the behavior function of mobile apps in some critical states. The basic idea is to find out various potential critical scenarios, and verify and test each category of critical scenarios.

The main scenarios are as follows.

Scenarios where the system memory occupies more than 90%.

Scenarios where the system storage occupies more than 95%.

Scenarios where the flight mode is switched back and forth.

Scenarios where the App does not have certain system access control, for example, the App cannot access the album or address book due to privacy settings.

If you use the App for a long time, are there any abnormalities in system resources, such as memory leaks, the excessive number of links, etc?

Scenarios of ANR(application not responding).

Scenarios where the operating system time is earlier or later than the standard time.

Scenarios of time zone switching.

For power consumption test, traffic test, and app performance test, how to determine whether the data is normal? Which value of traffic consumption is considered to need to be optimized, which value of memory and CPU is considered as abnormal and needs to be optimized? In fact, there is no clear standard. The main method is to compare with the historical data of existing versions and similar apps.

In the following part, we will combine some actual testing points with the scenarios below.

1. Crash

Device fragmentation: Due to the extreme diversity of devices, apps may behave differently on different devices.

Bandwidth limit: A network with poor bandwidth may not have enough fast response time for App.

Network switch: Switching between different networks may affect the stability of the App.

Memory management: The available memory is too low, or the use of unauthorized memory locations may cause the App to crash.

Overmuch users: Too many connections may cause the App crash.

Code error: New function that has not been tested may cause the App to crash in the production environment.

Third-party services: Ads or pop-up may cause App crash.

2. App installation and uninstallation

This is a scenario that is not available on the Web. Basically, you need to consider different operating systems, different operating system versions, the different modifications that mobile phone manufacturers do in different operating systems versions, etc.

Installation procedure

1) Whether each option meets the outline design specification.

2) UI test of the installation guide.

3) Whether to support cancellation and the operation process after a cancellation (whether there is any residual data).

4) Handling of unexpected situations (crash, restart, power outage, network outage).

5) Insufficient installation space.

After installation

1) Whether it is operating normally.

2) Whether the folders and files are in the specified directory after installation.

3) Whether redundant directory structure and files are generated.

Upgrade

1) Is the function the same as the requirement after the upgrade?

2) Does the function of the module related to the upgrade module work normally?

3) UI test of the upgrade interface (mandatory/non-mandatory).

4) Test of the unexpected situation during upgrade installation (crash, restart, power failure).

5) Version verification: Version 1.0–2.0 or 1.0–3.0

6) Retention of user data, settings, and the status during the upgrade, pay attention to the status or settings that have been removed from the new version.

7) Whether it is possible to overwrite installation by separate versions.

8) Whether it is possible to overwrite and install the lower version.

9) If ignores the current version upgrade, whether there is a prompt when there is a new upgrade version?

Uninstallation

1) UI test of direct system uninstallation and uninstalling procedure.

2) Delete the folder directly before uninstall.

3) Whether to support cancellation during uninstallation and the software status after cancellation.

4) Unexpected situations during uninstallation (crash, network disconnection, power outage, restart).

5) Clean up the installation directory, and the SD card storage data will not be cleaned up after installation.

6) When there is no update or network, it is necessary to give users the correct prompt.

3. Starting and stopping the app

1) Whether the welcome screen appears at the first startup, whether you can enter the App, and whether the stay time is reasonable.

2) Whether the information pulled after the first startup is correct.

3) Whether the restart time meets expectations.

4) Is the app function normal when restarting the app?

5) Status check after restarting: such as initialization information, initial status, influence to the network after starting.

6) Process and service check after restart: process name, process number, service name, service number, SDK called by a third party such as GPS.

7) Whether the login status is normal after the logged application restart?

8) Whether it can restart when the crash occurs.

9) Manually terminate the process and whether the service can be started here.

10) whether the app can start normally when other software stop the process or clear software data.

4. Interrupt test

1) Lock screen interruption: lock the screen when staying in the operation interface, and check whether the operation is normal after recovery.

2) Front-end and back-end switching: check if the function is normal after the switching of front-end and back-end.

3) Load interruption: conduct frequent switching when loading the page request and interface frame.

4) System abnormality interruption: such as shutdown, power outage, incoming call.

5. Software compatibility

compatibility with common software

compatibility with the input method

compatibility with security software

compatibility with communication software

compatibility with competitive software

Summary

According to different technical architectures, mobile applications are mainly divided into three categories: Web App, Native App, and Hybrid App. The testing methods for these three types of applications are essentially GUI testing.

From the perspective of business function testing, the test case design of mobile applications is similar to the traditional computer GUI automated testing, except that the test framework is different. Data-driven, page object model and business process packaging are still applicable;

Various special tests are the focus of mobile application tests, which are also different from traditional GUI tests. Special tests include conflit test, compatibility test, traffic test, power consumption test, weak network test, and boundary test.

The above are some methods and ideas of mobile application testing, we hope it will be helpful to you.