This article talks about how to prepare for testing for your Virtual Reality applications. The process has some important differences and considerations when compared to desktop or mobile testing. It is not an article that deep dives into what user testing is, or why you should do it. It does not cover technical setup of tools, and assumes you have appropriate access to the respective technology.
Testing products in progress with users is extremely beneficial. It helps us to validate ideas in products, find issues in flows, and prevent potential bugs from making it to production. As Virtual Reality (VR) becomes more popular, it’s imperative that we test these applications with users as early as possible for, at a minimum, safety reasons. If you avoid user testing in VR, there is a risk your product could have negative long term consequences on users. VR applications can cause nausea and motion sickness, causing vomiting in use or post-use. Permanent hearing loss due to exposure from loud audio. Long term usage of nearsighted work can cause myopia – damaging of the eyes. Tripping or falling while immersed in VR can cause bodily harm. These are concerns you should be aware of when designing and testing your application, trying to decrease the chances of these issues affecting your users.
Virtual Reality puts users into a simulated environment with the use of technology. It creates a 3D world for the user to interact with, and immerses them into a simulation that uses vision, hearing and touch to interact with this artificial world. It takes people out of the real world, and into one where they can experiment, play, and learn in a virtual world. Applicable industries are vast, including healthcare, automotive, law enforcement, military, construction and fabrication. As VR opportunities increase, common development workflows will be created. User testing is one of those crucial workflows in the creation of VR applications.
If you’ve ever done user testing with a desktop or mobile app, you’ll find some overlap of process. First, you’ll define what you’re testing for. Is it the setup of the app? Is it the user interface controls? Is it a specific workflow in the app? This will depend on where you are in the build process of your application, and you can create additional future tests to understand various cases. For this test, your best course of action is testing for one item, as this will help keep your tests focused and short. When testing VR, you need to keep your tests under 20 minutes. Any longer than 20 minutes, and you run the risk of having participants become fatigued and/or nauseated. Compared to traditional application testing, which can run over an hour, 20 minutes might feel short, but have no fear. Even in 20 minutes, you can gather a lot of meaningful data.
You’ll want to write a testing plan – a series of questions and steps you’ll ask each user. This creates a baseline to help keep tests comparable between users. The questions and steps are influenced by the what you’re testing. This plan helps keep the entire testing team on task, and ensures meaningful content is collected.
Finding participants to test your application can be one of the hardest parts of VR testing. Traditional testing may allow participants to be remote, but VR testing is going to (usually) mandate participants to be on site due to hardware setup. This will limit the pool of potential participants to location. Your goal for recruiting is between 5 and 8 participants. If you have less than 5, you can still test – but your results may not be as trustworthy. If you have more than 8, reserve some participants for the next user test. Higher valued participants will be ones that are potential users of your product – regardless of their VR experience, even rookies. Don’t take participants just because they have VR experience, they may skew the data if they’re not going to be product users. Be prepared to introduce your participants to the VR equipment and bake-in time for walkthroughs and questions they may have. VR doesn’t have heavily-used common patterns yet, and a lot of people don’t have extensive familiarity with it (or any familiarity with it). This is a stark contrast to testing desktop or mobile apps, where users are typically quite comfortable with the interfaces and constraints. Ensure the participant is comfortable, and knows that they can take breaks or end the test at anytime. If recording any data that may be considered sensitive, share what data is collected from the user and why.
Unlike traditional testing, there is a bit more physical prep work involved with VR testing. Schedule your test at least 1 week out, so you can prepare a dedicated testing area. You’ll need the machine and equipment to run your app, cleaning supplies for said equipment (or replacement equipment if swapping for each participant), room for the user to move around (if applicable), room for cameras and their stands, and a room that can comfortably fit 4 to 6 people. Keep this space free of clutter and remove anything non-essential to the test. Participants won’t be able to see the real world when they are in your VR app, so remove potential dangers for them – cords, boxes, machines, chairs, desks, etc. Participants can easily trip or bump into these items if they are left out, potentially hurting your participant and your test data. It is advised to task someone with keeping the participant safe and ready for potential nausea.
You’ll need a minimum of 2 people available to administer the test. The Protector, as mentioned above, will keep the participant physically safe during the test. The Facilitator, will be the voice that guides the user through the tasks in the test and communicates with the participant. When appropriate and applicable, allowing the Facilitator to enter the VR environment with the user to conduct the test, creates a better cohesive experience and is less jarring. This can be done as audio-only, talking through the VR headset or as an actual avatar or game piece in the VR app itself. Along with these 2 roles, the Recorder is a helpful third and the Tech is a beneficial fourth. The Recorder is in charge of making sure video cameras are working, cameras are on the participant, and recording of what the user is seeing (when possible). The Recorder is also in charge of saving all the video, and organizing it into logical order for future review. The Tech provides support for the VR app when there are technical issues and restarting or resetting the app between participants.
After the test (remember, 20 minutes maximum), you want to ensure the participant is okay and not going to be nauseated or sick. Have them fill out your post-interview questionnaire, answer any remaining questions and do a Simulator Sickness Questionnaire (SSQ). The SSQ is helpful in making sure that participants are leaving in a healthy state. Originally used for airline pilots, it’s been helpful for VR applications due to the similarity of motion sickness awareness. Schedule breaks between tests, with a minimum of 30 mins. Use this time to reset the VR application, clean the VR gear or swap it with already clean equipment, save and compile notes, and mental breaks. Do a maximum of 4 tests per day, to avoid team fatigue.
Post conclusions of the tests, gather the team to review notes and footage. Discuss what went well, what didn’t go well, and what surprised people. Take these edits to your product roadmap, and adjust the application accordingly – knowing that you’re moving in the best direction.
At Cantina, we are dedicated to learning and using the latest Virtual and Augmented Reality technology. Our focused AR/VR working group spends time working with this emerging technology, learning best practices and exploring the latest tools. At our downtown Boston location, we have dedicated space for both developing and testing Virtual Reality and Augmented Reality experiences. If you’re interested in learning how we can help your AR/VR application, please get in touch with us today.
