Playtesting is a Mess
Game design as a profession is a grey, murky zone sitting between science and art. What sets it apart from most other design fields is that user testing is integral to the design process. If you took architecture profession for example, there is a lack of practice in testing real life mockups and test-builds throughout the design development. This way, the design decisions made on orthographic and perspective drawings, many of which are based on assumptions, cannot be verified until the day of conception. However, the virtuality of video games lends itself to a much tighter feedback loop from the time an idea is conceived to the time it is verified.
Playtesting is a design mechanic that is unique to game designers.
So it leads many to view playtesting as a special privilege to those working in the entertainment technology industry. However, like everything in life, this unique opportunity brings with it its own set of unique challenges. It is a double-edged sword – while it can serve as a valuable tool in navigating the swamp of player psychology, it may itself be a can of worms that must be treated with extra care.
Depending on how one looks at it, it can be viewed either as science or art. It demands a high degree of scientific rigor but also raises a set of questions that are subjected to artistic interpretation. It is a blend of scientific experiments and artistic expressions. In other words, playtesting is a mess.
This duality is the root cause of many confusions that game designers face today. With the high degree of complexity, scale, and polish that are expected of video games today, it has become all the more difficult to objectively isolate and measure an aspect of a game without being lost in the swamp of data that are indistinguishable from one another.
So how does one exactly navigate through such hodgepodge?
Playtesting is an experiment that first simulates an ideal context and situation of the players to verify a particular design hypothesis. In order to focus on addressing a specific problem, many of the variables are often simplified to the bare minimum to eliminate any possibility for external influencers. It is a highly controlled simulation that must be carefully considered, designed, executed, and analyzed.
The game designer in this case is kind of like a mad scientist. One must possess a rigorous understanding of human perceptual and behavioural psychology, and a level of sophistication and discipline to parse and organize the datasets into tangible conclusions regarding the problem statement. Human mind is one of the most complex systems in the world, and the game designer’s job is to insightfully sift through the mountains of factors to discover an actionable solution.
For example, during the development of its highly acclaimed AAA first-person shooter title Halo 3, Bungie Studios have extensively used Microsoft Labs’ state-of-the-art testing facilities to mine the vast quantities of playtest data necessary for testing a game of such magnitude and scale.
Of course, there are potential pitfalls to this laboratorial, data-driven approach. First, the designers must take heed in ensuring that the playtesting experience is fairly replicating and approximating the actual gameplay experience. Players are often used to playing in their personal comfort zones – in living rooms or bedrooms – surrounded by familiar environments and people. The physical setup itself has a tremendous influence on how players behave. Even the slightest minor detail may elicit the players to display aberrant behaviours and mislead the designers to draw incorrect conclusions.
For example, although not directly relevant to the topic at hand, Gerard Jones addresses the problem of incorrect contextualization in research experiments in his book Killing Monsters. In describing the origin of such problematic tendency, he says that the researchers are “…creating abnormal situations and thus inducing abnormal reactions…” This runs the risk of the test subjects producing outlying results that may confound the observers and hence mislead them into a black hole.
So preconception is a powerful force, and many problems can arise not just from how the datasets are collected, but from how they are interpreted.
As culturally sensitive and personal as game development goes, it is the game designer’s responsibility to be as objective as possible when analyzing the players’ behaviours and withdrawing conclusions from them. As the common saying goes, “We can only see the answer to the questions we ask.” It is therefore imperative that the designer’s line of sight steer clear of the personal, social, and cultural biases that may pollute one’s sense of judgement.
Again taking the Gerard Jones case, the researches surrounding the relationship between violent video games and their players’ aggression have been heavily criticized for their lop-sided assumption that the aggressive behaviours of players are reflecting the violence that these games portray. On the surface, this may seem true. However, when one looks beyond the temptation in generally attributing aggressive behaviours to violence, it becomes clear that there are many factors at play. What is preconceived as violence may as well easily be just a simple act of play.
However, when one is dealing with something as deep and complex as human emotions, objective, scientific analysis may simply not provide sufficient basis to determine the right course of actions to take.
In general, entertainment media relies on keeping the audience engaged through the presented content. Video games in particular, however, introduce another layer of complexity through its interactivity. Hence the ability to predict how the players will think, feel, and act is especially important for a game designer. Empathy is an extremely valuable asset.
So when it comes to playtesting, game designers must be extremely keen listeners and observers. This is not to say that they should jot down every piece of feedback they receive from the players. What happens quite often is that the players will give direct advices on what they want the game to be, but not how they want it to feel. It is important that the designers be wary of the players’ preconceptions and predilections, and be able to identify the inner craving that the players themselves cannot articulate.
Christopher Alexander, a visionary of his time, takes a jab at narrowing down the qualities that the human subconscious universally appreciates. In his book Timeless Way of Building, he draws a common trait across all appreciated pieces of architecture, a trait that he calls the Quality Without A Name. Although the definition is not necessarily singular, one can take with some certitude that innate human desire is real and measurable.
On that note, it is equally important to acknowledge that one cannot satisfy everyone. Why would then there be so many different genres in entertainment media? Hence the definition of a “fun” game is largely dependent on the context and the individual, and the designer must accept these conditions of terms when designing for the target audience.
Game designers must therefore know not only which are the right questions to ask, but also which are the right answers to take.
With so many facets to consider, playtesting is a nightmare for game designers. As Jesse Schell himself have openly admitted, “The most absolute thing I hate about game design is the playtest. It is absolutely sickening. You have to give up your precious baby that you have been nurturing this whole time…”
To a large extent, I sympathize with the above statement. Playtesting is a heartbreaking process for many game designers who invest a tremendous amount of effort and passion into their work. In contrary to other design professions, I believe that video games command the deepest level of human emotion. It is not surprising that playtesting can be emotionally devastating.
At the end of the day, a game designer must believe in what he/she is making. While fresh eyes can provide new perspectives, one is still the world’s most expert on the subject at hand. So when you feel yourself lost amidst the cloud of information, go with your gut feeling.
Playtesting is hard. Really really hard. It is a weird dynamic of push and pull that game designers must play with their audiences, all the while keeping the vision undiluted by the muddy puddles of user feedback.
However, if one has the imagination and the will to persevere through the relentless waves of criticism and revision, he/she may perhaps be on the right path to the top.