Integrating User Testing With Agile Product Development

For years if not decades top management executives from large corporations have talked about how the user experience (UX) of their customers is a top priority.

However, a quick & easy user test of even some of the most important and well known companies shows that this has, until recently, been more lip service than concrete actions.

As an example, here are a few highlight reels of UX issues we identified in two recent demonstration studies:

 

The reason for this mismatch between statements and results is not hard to find. Until mid 2015 there was a severe lack of resources underpinning the supposed priority on the digital user experience.

As a result, there were very limited resources assigned to the discipline of user experience design, user testing and user research, whether in terms of the number of personnel or the budget that these UX professionals have at their disposal.

The situation was not much better within the agencies serving large corporates. In a typical digital agency environment, there’d often be no more than 4-5 user experience designers handling UX needs for all the clients in an office of over 100 people.

The tide is finally turning. As consumers are becoming more design-conscious and the economy is becoming more experience-driven, the demand for UX designers has skyrocketed.

In addition, most large corporates are quickly staffing up their UX departments, increasing the number of user experience researchers by a factor of 400% or more in the space of 12 months.

This trend reflects the rising attention paid to UX as a discipline, which is also made possible by better and more effective user testing tools and methods.

Agile and User Testing

In parallel to the increased resourcing in terms of UX professionals, user research and user testing budgets, an increasing number of companies have been adopting an Agile methodology for project management and product development.

The Agile methodology focuses on building workable prototypes rapidly and then testing them to better inform the next iteration. 

It’s a departure from the legacy waterfall software development method, a sequential, non-iterative design process. The waterfall method is often described as flowing steadily downwards (like a waterfall) through the phases of conception, initiation, analysis, design, construction, testing, production/implementation, and maintenance.

Unlike the conventional waterfall workflow, the Agile approach and sprint-based development allow more opportunities for teams to test and improve the user experience design of a product, thereby bringing the importance of good UX to the forefront of the software development process.

Since 2015, Agile UX processes and systems have been adopted by many companies, large and small, and have implied the adoption of fast paced user research, user testing, and usability studies to support the iterative nature of design > test > redesign in an Agile UX Testing process.

The Agile methodology is becoming increasingly popular because it offers tremendous cost-saving advantages in eliminating development inefficiencies when compared to the waterfall workflow

In the waterfall methodology, user research and user testing are often treated as items to check off the list at the end of a product development process to “validate” previous decisions, rather than as a “learning” process; By the time a team received the user testing results, it was often too expensive to make the required changes if they significantly altered the features, functionality or UI.

In an Agile methodology, the process of user research and user testing has become an integral part of the design and development flow. Agile UX testing is often performed in parallel to design and development sprints.

The results of such integrated “Agile UX Testing” has a significant impact on the subsequent iteration of a product, thereby making it much easier to justify the cost of investing in the research and quantify the return on investment (ROI).

Companies aren’t adopting Agile UX process just because “everyone else is doing it.” Research has shown that user acceptance and user adoption — both of which can be improved dramatically by well-designed user testing — is critical to boosting ROI and ensuring the success of a project.

Challenges In Integrating Conventional User Testing Into the Agile Design & Development Process

Even just a few years ago when many companies were still following a waterfall methodology, usability testing was often conducted in a lab and required careful planning months ahead of time. This is no longer a possible or practical set up for companies that want to apply user testing as a rapid, iterative “sanity check” to assist in the design and development process, which often span only 2-4 weeks. 

In that sense, in-lab moderated usability testing falls short when it comes to providing feedback in real-time to inform the next iteration of a product, because it is an inherently unscalable process.

For a long time, in-lab moderated usability testing and the Agile methodology have been at odds with each other because it’s hard to reconcile the time it takes to set up and conduct an in-lab moderated usability test and the fast turnaround time required by an agile sprint. 

Moreover, the cost of moderated user testing in a lab environment can be prohibitive, especially for smaller companies. Travel expenses, lab rentals, test equipment, time away from work by employees — these factors can all add up, making it impossible to incorporate frequent user testing into product development cycles.

The challenge has been to conduct usability tests in an agile and scalable fashion, so the user testing process can support the rapid product development cycles instead of slowing them down.

Remote User Testing Faster than In-Lab User Testing

In order to solve the challenges posed by conventional usability testing and make it work for the Agile process, remote usability testing was invented. 

With remote user testing, instead of orchestrating day-long usability tests done in labs, real users take part in user tests from their homes and offices, on their own schedule — a comfortable environment that allows them to react in a way that’s closer to how they would in real life.

Remote user testing allows user research to be set up and completed quickly — often with a 1-day turnaround — while providing a timely and continuous feedback loop of user experience insights to inform the Agile design and development process.

Remote user testing can be either moderated or unmoderated — each with its pros and cons:

Moderated Remote User Testing

Moderated remote user testing allows UX practitioners to design instructions and pose questions based on the actions and responses of each user testing participant. The Moderator and participant interact with each other in real-time, using screen sharing platforms like webex, gotomeeting, skype etc. 

Though this method removes the travel expenses and lab rental fees and increases schedule flexibility, companies still have to allocate specific personnel and time slots for each single participant, making this an inherently non-scalable solution.

The implementation of moderated remote testing is only possible when the UX department has enough personnel and resources to provide this type of usability testing support to the organization. In addition, the organization needs to be large enough and develop enough products to absorb the cost and justify having this level of resources on “standby” all the time.

Unmoderated Remote User Testing

During unmoderated remote user testing, UX practitioners assign pre-defined tasks and questions for participants to complete on a website, mobile app or prototype asset — via desktop or mobile devices. The UX team collect and analyze the qualitative and quantitative data via software that records every interaction, including a picture-in-picture video recording of both the participant and his/her screen (desktop or mobile).

This user testing method is inherently scalable because multiple tests can be managed simultaneously by a single user testing researcher, on the schedule most convenient to participants, with no need to tie down a user testing researcher on a 1-1 basis. It also has a relatively low overhead.

However, this method doesn’t allow real-time interactions with participants. So user testing researchers cannot probe with specific instructions or questions based on the real time actions and responses they observe from participants.

That’s where the latest innovation in remote user testing, branching logic with qualitative picture-in-picture video recordings, comes in, bridging the gap between moderated and unmoderated usability testing.

Combining the Benefits of Moderated and Unmoderated Testing

The emergence of unmoderated user testing platforms with branching logic is allowing UX designers to combine participant action and respondent dependent user test scripts with scalable user testing. 

Branching logic, with the aid of advanced software, combines the scalability of unmoderated user testing with the flexibility and customization of moderated usability testing. The moderator can customize the user test script for each participant with different instructions and questions depending on their actions and responses.

This method allows companies to conduct usability and user testing to collect both qualitative and quantitative data, including picture-in-picture video recordings of user testing participants and their screens, quickly and affordably so the results can inform the rapid and responsive product development cycles based of an Agile methodology.

This innovation in usability testing is enabling UX design & testing departments to conduct large scale, multiple and iterative user testing to support the exponential growth of digital interfaces that need rapid and effective user testing.

Userlytics Blog. May 15, 2023. Integrating User Testing With Agile Product Development. Userlytics

---

Interested in UX Testing?

Data Visualizations

---

Data Visualizations