Mobile game testing differs from the regular mobile app testing. Effective mobile game testing derives from a well-structured and systematic approach, use of test automation framework and seamless integration with your agile process. Naturally, the simplistic view of testing is to identify bugs and help developers to remove them. However, to aim for efficient, result-driven testing it is mandatory to have test automation as a part of the overall development process.
This is the first actual blog post on the series of the Best Practices in Mobile Game Testing. Each Wednesday for the next 5 weeks we will deep-dive into a new topic, focused on something important for mobile game developers.
Testing in general, but applying also to game testing, there are two different forms of testing that can be categorized as a Black-Box and White-Box testing. These definitions are well-known in software testing world but process-wise aren’t different with game testing. Only their goals are slightly different in game testing:
Black-box testing – focuses on the functional and overall playability aspects of the game. In this type of testing, for example, testing the graphical user interface, user experience or any visual appearance is in the key role. Menus, graphical elements, special effects, animations, and the actual gameplay are those under test with Black-box approach.
White-box testing – focuses on the architectural, integration and systemic aspects of mobile game: how third-party components, databases, social media/external entities, as well as graphics/game engines, audio play and so on are integrated in to your game.
In the test automation context, black-box testing focuses on controls, capabilities, features, gameplay, and automates the flow of testing. For this, it is typically recommended to build test scripts that know how game is progressing, what kinds of faulty states could there be, and also focus on image recognition. As many graphics cannot be recognized as control mechanism – buttons, menus, sliders etc. – image recognition will help to identify those that are meant for controlling (and not just as a graphical element).
Test automation with white-box approach focuses on testing the integration aspects of the game e.g. how game utilizes social media and how integration works.
METHODS USED IN MOBILE GAME TESTING
The testing methodology and used forms of testing aren’t too different from application and other software testing. The most important ones when it comes mobile game testing are as follows:
Functional testing is probably the most common method in mobile game testing. Typically functional testing is associated with manual testing and playing ‘game through’. However, to be agile, all functional testing should be automated. Functional testing – with help of test automation frameworks – requires some basic understanding of programming and setting up environment for testing. Automated functional testing can reveal issues related to user interface (and graphics), stability, game flow/mechanism, and integration of graphics assets.
Compatibility testing is all about making that game compatible across different devices. The typical misconception with compatibility testing is that it will be only done with the final version of the game. If compatibility testing is done during all development phases – known also as smoke testing – it can help to reveal incompatibility issues with any parts of the game, its third-party components or integrations with those actual devices that end-users use. This is very important to get the game working robust and well across all possible device variants – hardware and software.
Performance testing is important for game testing. Sluggish and badly performing games can ensure that game won’t be successful, gets bad ratings and won’t make its creator name. Related to performance testing it is important to understand how used device ecosystem varies and what are actual requirements of the game for its users. Many games are by default developed (and tested) on high-end devices, without any chance to get decent performance level on low-end devices.
Localization testing becomes highly important when your game is targeted for the global markets. The word ‘global’ means your game needs to be ‘local’ for everyone. When your game titles, texts and content needs to be translated and tested with devices in multiple languages, these types of tests can be easily performed automatically (with help of cloud-based device access and test automation) to change the language of devices, and even do it again with the same test runs. The problems layouts can be easily picked from screenshots.
Regression testing needs to happen when anything changes in your software. Most of the mobile games today have a server-client interaction, requiring a login, uploading of data (e.g. results) and downloading of data (e.g. data, images). If you’re developing these kinds of services around your game, users need to ensure all their changes are done in code, to either server side or client side do not break the functionality of the service. Automated testing is really the only option in this case, manual doesn’t cut it.
Load testing tests the limits of a system, such as the number of players on a server, the graphic content on the screen (e.g. frames per second, FPS), or memory consumption (allocation and deallocation of it). Load testing can be built seamlessly in test automation context and heavy loads of data, usage or interactions can be simulated with test scripts and real content form backend servers. In this category you can count in multi-player testing as it can involve several different scenarios to be tested.
6 KEY AREAS IN MOBILE GAME TESTING
#1: User Interface and Functionality
The User Interface and overall functionality of it has a direct impact on how successful your game will be. These two things, including visual appeal and the gameplay are the most important thing to get right – and ensure device fragmentation doesn’t break any of these. In the UI various things needs to be tested:
– UI layouts and elements: All games are typically targeted for maximal set of different screen resolutions and different types of screens. Regression testing should be done each and every time when UI layout changes to ensure that game works.
– Menu structure and functions: Testing menu structured, functionality and correct behaviour can be automated with instrumentation and help of different test automation frameworks.
– Screen orientation: Surprisingly this gets so wrong with many games out there. For example, if screen orientation changes during the game what happens, and what actually supposed to happen? Does the game work well in both landscape and portrait modes?
– Screen resolution: There are lots of screen resolutions especially on Android and typically autoscale can help game developers. However, it is highly recommended to test your game across these different resolutions to ensure that the graphics do not get stretched.
#2: Graphics Performance
Performance needs to be consistent across all different device variants that your game’s users have. Because of this it is highly recommended that you use as many real devices for testing as possible. In order to determine how well your game responds to various levels of usage, you should consider creating tests which last for hours to observe how well the game as well as how much of battery is used during long usage. To determine if you game will be able to run successfully for a sustained duration under heavy load, Load/Stress tests can be used. The performance tests will measure how responsive your game can be on a real device.
For example, take a look at Gamebench. It can help game developers to understand those potential bottlenecks in performance, reveal unused potential with CPU and GPU resources, and help to push the boundaries of Android gaming and visuals, without forgetting the most important thing – a need to test the game on real devices.
#3: Usability and User Experience (=Good Entertainment)
Testing usability, navigation flow, and eventually what user experience your game delivers to gamers simply cannot be done on a desktop with a mouse and keyboard. So forget emulators and use only real devices for testing. And to get the best possible understanding of how great usability and entertainment your game is providing, here are two important things you should consider when testing ‘user experience’:
User Interactions and Excellent Responsiveness – It is important to test the game for its performance as this will make either the positive or negative user experience. For example, performance lags are easy to expose using actual devices.
Real occurring events on background – Interrupts, battery consumption, how charger effects to overall performance and usage. These all have significant impact on user experience – and good entertainment level.
#4: Multi-player/User Features
Nowadays, multi-player/user feature is very common in mobile games. A need to test multi-player abilities is there and is naturally more challenging and requires the other end to perform as a real counterpart. For example, typical case is that the game communicates with backend server. In these cases, connectivity is essential and the synchronization between backend and devices that need to get information about the gameplay. There are tons of different scenarios you should be testing and many of those will severely impact the game’s experience, resulting in negative feedback and gets uninstalled by users.
#5: Social Integrations
Social network integration is also very important part of your game. In many games, it is essential to be able to share results across
your ecosystem, friends or just for yourself. This should be thoroughly tested in the context of real Android and iOS devices, with different OS versions and different device configurations, to understand the functionality and easiness-of-use.
#6: Security and Liabilities
Nearly all game developers use some open source components as part of their game. This is highly recommended and well-accepted development practice as it offloads the task of developing code for non-core functions in your mobile game. However, identifying third-party code, its vulnerabilities, and license restrictions are very often neglected by game devs.
This was the first blog post on the series of the Best Practices in Mobile Game Testing. Next time we will focus more on graphics, image recognition and other things related to improving testing for your mobile games.