What is bot? It is a script
A script which looks like a player
As in other games, we are using bots to help test World of Warships. Bots are a useful and constantly developing tool, they simulate a player and can perform actions programmed by scripts. Our bots are special versions of the game client without any interface or graphics, so they are lightweight and many of them can be launched simultaneously on the same machine, each one controlled by a script.
Bots are good at resolving problems where massiveness is needed or tedious testing of situations that have clearly described boundaries or parameters. If we need to test the statement that “a full torpedo salvo from the destroyer Shimakaze should sink any ship,” they will be able to handle it. An additional benefit is that bots can work at night, when the developers are tired and sleeping, and then in the morning they will provide you with results. Bots are primarily used to test servers, their capacity and how are they going to work under pressure, but sometimes they are used for development purposes.
And the last important detail: since we work in a subdivision of automated QA, we call our bots “Autobots.”
Crazy firing bots!
One of our bots’ goals is collecting statistics of guns firing from different ships. As we have historically true (correct) guns, armoring, trajectory and shells, game designers decided that it would be nice to have a tool that allows them to see how well ships will survive under different gun salvos. Bots were the perfect solution for these tests – ships are subjected to fire from different distances and guns, and then the collected statistics can be grouped in convenient tables and sent to the developers.
The process of data collection can be seen of first picture. Here it is in simple steps: bots line up on a special, completely blank, map where there is only water. Opposite from them, somewhere around 1 km ahead, their targets are placed in a line. Next, the bots fire one after another at their targets and once they’ve finished firing, the statistics are gathered and stored in tables.
It is quite interesting that the results from the bot tests were very similar to real armor penetration tables. This allowed us to go further and configure “paper-ships” and “paper-guns” – objects for which we did not have any info about penetration and armament.
Bots invade the servers
Our colleagues from the stress testing department also love using bots. They can easily generate the required server pressure by using a number of bots whose actions are predictable, and when you have a lot of work to do you can easily write a script in advance, simply by programming “in a half an hour send all the bots to the game.”
Our first and longtime task was to check server queues. This scenario is very simple: use as many bots as we can to try to log into server and check what is happening to server. During the first test, the server could not handle more than 500 bots, so we reconfigured it and now server works as needed
Preparing for alpha
Such testing was useful in preparation for the alpha test when we were testing our future server to see how many random battles it could handle at once. 1500 bots swarmed onto the server, selected ships and went into battles. In these battles, the bots were actively trying to discover each other and then immediately destroy the target. Our first prototype of smart bots learned not just how to shoot, but also how to maneuver, as well as to search for the enemy in such places where other bots have not passed through.
The first test was quite artificial: it tested only an ideal scenario of players’ behavior, but as we know, live testers play in unpredictable ways. But this version also gave us information to work with: bots were logging in and logging out from game, entering battles and fighting almost like humans. The server passed its tests with flying colors when real Alpha Testers arrived. And as we have said before, such a mass sprint helped us to discover several issues, which we immediately began working on. After all, some issues can only be found in long-term operation.
Automation of routine
Another interesting and necessary task for bots is the automated testing of in-game logic. We definitely need to make sure that our ships do move, anti-aircraft guns do fire, and torpedoes do drown other ships. Testing all these features would be quite hard to organize, especially in the long term. Arranging each ship for manual testing would mean choosing which ship to go to the battle, checking what you need, and then repeat the same process on next ship. It would be unrealistic to attempt to organize manual testing when we come close to the release date, as we will have lots of different ships and tech trees.
For tasks such as these, bots come to the rescue. During nights, while the development team is asleep, different parts of game logic is being tested by bots: usage of weapons, ability of all available ships to sink in the game, correct movements, equipment and etc. Most of the work is done by bots, but selective control is done by humans. And only the combination of automatic and manual control gives the best results.
They are also used to run quick smoke tests – it’s like buying a new gadget, turn it on and checking if there is any smoke coming from it. In this case, the bots quickly start a prepared battle scenario with checks like “well, looks good, nothing is broken” for a few minutes. Such tests can be used for any new functionality before it is send to manual checkup. Right now we are trying to extend the usage of bots and find new ways that they can help us test the game.
Other types of bots:
Of course, despite all the benefits bots provide, they are far from perfect and have their own limitations. They cannot see the game to give feedback regarding wrong textures, bad colors used, etc. They also cannot behave as humans, especially in the area of interface tests such as pressing buttons or locating information in the GUI. And that is why we have second type of bot with a special “heavy” client with the full interface and graphics. We call this bot “client bot”, and we use it for a variety of different types of tests.
But that’s a story to tell on another day.