If it comes down to a decision between what’s realistic and what’s fun, we’re going to choose the fun path and then rationalise it based on some obscure historical incident…
– Sid Meier
As we said in our previous article, our number one tool for auto testing is bots. For our developers bots are wonderful and make life much easier. However, bots are unable to check whether the graphics and interface are working correctly. They just have no graphics, no interface. Therefore, to check these important elements of World of Warships we also work with a special test version of the client.
Our First Attempts
At first, we tried to approach the problem of mass testing using head-on, by taking a test tool that allows you to search for desired images on the screen and, with its help, perform a simple test. No sooner said than done, the tool worked and even coped with completing complex tests, such as checking that a ship burns in the right way. However, this tool was always breaking down. Any action performed by the artist – adjusting light, changing image sizes, adding new visual effects etc. – interfered with the tool’s performance.
Still, this tool was useful, especially in simple routine tasks. On command, the tool took the latest game version, unfolded it on a test stand and quickly tested it, reporting: “Task complete! Game works!” Otherwise, it was collecting a list of errors and sending notifications about them. At night, the tool was doing its work and in the morning we received an e-mail saying either that “Everything is fine” or giving a list of errors like – “I cannot find the ship at the dock!”
However, many regular failings of this tool interfere with our work and so we began to look for other options…
A solution from another field
In web application development, the kind of problems of checking that we face are also common, like how checking a website through a simple search is not the most reliable method of doing so. A slight change in design can break such a test. So the solution for the problem is to use a special test framework, i.e. a set of functions for creating tests, which considers the entire page as a single object with a defined purpose. Accordingly, the test tool does not function as a test by functioning like “find the button with a picture ‘Send’ and press it” but instead it tries to perform the operation “press ‘Send’ button” itself. The idea was taken on board.
Adaptation
The direct transfer of this approach to game development was impossible for the obvious reason that test objects are completely different to those that are displayed in a browser. So Automators, together with developers, had to work hard to find how to adapt it into a solution for our game. As a result, automatic scripts were able to access controls on testbeds.
And now, when we are launching an automatic smoke-test just to check if it works, the tool does not look for an image; it now works directly with interface elements. Our colleagues liked that the developers could now run a quick check before it is given to players for testing.
How does it work?
During the test, the automated test client simulates a player staring at the screen and searching for the necessary elements to achieve something like “Go to the dock” or “check presence of all menu items and ships.” These things are the typical operations and responsibilities for the test version of the client and, in the future, it will allow for a full check up on tech trees to ensure that all the connections are there or not.
This automated test client, of course, has its drawbacks too and cannot check the aesthetic qualities of maps, for example. However, it has helped to greatly reduce the amount of manual checkups required.
For example, recently during a check of a new U.S. battleship, it was discovered that it would not sink after a torpedo attack. As it turned out, the ship had quite good anti-torpedo protection. The real ship too had one of the best anti-torpedo protection systems and weak destroyer-torpedoes couldn’t do any damage. In reality, Japanese torpedoes of that time could penetrate even a submarines defences, so something had to be altered.
We have a game that should be fun to play. Guided by the widely known motto – “In the case of having to choose between gameplay and realism you should choose gameplay” – we started our reconfiguration. Such torpedoes should be able to destroy the first section of anti-torpedo protection and the second wave would destroy the bulkhead between the engine room and anti-torpedo protection.
Teamwork with auto-bots.
Currently we are trying to build something that would allow our tests to incorporate the work of bots and the test client simultaneously. While one “part” of such test, i.e. the test bots, is busy with mass checkups, the client is free to check if everything works correctly during the battle, let’s say, if the sinking of a ship is correctly causing the score to change without error.
We also use functions like replays and battle recordings to help with the process of testing but that’s a story for different time…
