Rigor is a key ingredient in any successful user research project. But what does rigor mean, exactly? And how can you ensure that your research is rigorous?
☕ Rigor is essential in user research
(Thanks for the topic idea and the insightful discussion, Louise!)
Doing (more) user research makes better products, as teams feel more empathy towards users, and make better decisions. But more research cannot mean we compromise on the quality. For example, the proliferation of unmoderated usability testing tools led to an increase in usability tests done but didn’t lead to much better products as usability findings got more shallow and diluted.
The big challenge is how to keep up or increase the quality, so how to balance the pace and expectations of modern product development and standards for good research. In short, how to keep user research rigorous.
It’s almost always better to talk to users, learn about them, and involve them in product development (meaning doing user research). The opposite removes the user from the user experience.
But it’s also better to do this the right way, as the opposite would be just reinforcing biases, seeking validation for opinions rather than seeking facts, and just generally leading to studies with uncertain outcomes and sentences like “We already knew this!”. Orgs get user research theater and cargo cults if rigor is not kept - instead of better insights they would need.
There needs to be clarity of why research is done, how the research insights are generated, and how the insights support design and product decisions.
Rigor in user research establishes this clarity, brings discipline to the research process, trust in the results, and builds the reputation of the researcher. It leads to informed decisions based on facts, trust in research findings, and just an overall improvement in the product’s user experience.
Rigor in short is the extent to which the research is conducted in a systematic and objective manner. How well the research follows established standards and practices. It is important to ensure rigor in user research to produce accurate, reliable, and valid findings - insights that add value.
Much of how we do user research in the industry is established in academia, and researchers coming from that sector often bring their scientific method and a mindset biased toward rigor with them. And while we usually don’t need scientific rigor in our user research practice (we are not publishing papers after all), to produce anything of value, at the minimum, we need to understand the trade-offs we are making.
A sidenote - design as an activity also needs to have rigor, especially as teams scale up. Rigor in design means having a transparent, objective, and systematic approach. Clear intentions and design rationale for decisions. This still doesn’t mean the design process needs to be a list of rigid activities, but rather the use of good practices, the right focus, and a structured exploration of the design space. Things like declaring intentions, articulating design decisions and doing great design critiques all help.
Research rigor, at least for qualitative studies, builds on four pillars - credibility, transferability, dependability, and confirmability.
Credibility means that the research findings are believable and trustworthy. In other words, if the findings are likely to be accurate and reliable. This is where following the correct process comes in, including establishing a hypothesis, clear research questions, using a variety of methods, involving stakeholders, and having an understanding of the target industry and target participants.
Transferability refers to the extent to which the research findings can be applied to other contexts or situations. In short, how likely are the findings relevant to other users or other products? Having clarity on recruitment and participant choices, looking for behaviors, and not only for demographics in screening, understanding participants, and getting a deep understanding of users’ tasks are good ways to ensure transferability.
Dependability describes whether the research findings are consistent and reliable. So how likely are the findings to be the same if the research were to be conducted again? Setting the right scope, understanding what questions can be answered through what methods, having clear research questions, and choosing the right number of participants are all connected to dependability.
Qualitative studies having a low number of participants means that there is always some variability in the exact results, but patterns found for the most part shouldn’t change, thus creating dependability.
Confirmability means that the research findings can be verified by others. Can other researchers replicate the research and reach the same conclusions given the right resources? Having a clear and transparent process helps, including clarity of test setup, defined questions, objective ways of collecting data, combining methods, and established recruitment criteria.
Good rigor might be viewed as something that keeps research projects from being agile or lean - which is not true. While user research is viewed by some people as something that takes too much time, the alternative of not doing research or doing bad research will take more time in the long term as unusable features and products no one needs are built.
Rigor might mean additional challenges (like finding the right participants might take more time than talking to anyone), but it shouldn’t take significantly more time. The devil is in the details, researchers (or People Doing Research) need training, continuous learning, and the right tools to be able to execute well. Ultimately rigor is a choice in the research process.
This is also where using good frameworks (for example RITE for testing) or tools (like automated recruitment) come in, they should enable good research, rather than make it easy to do bad research faster. Methods, tools, or frameworks promising faster research results should be understood as do we compromise on rigor or other aspects of the research?
Unfortunately, rigor does have some downsides. It’s more challenging to do (for example needs proper recruitment and clarity in session goals). It needs training (like how to ask questions the right way on interviews). Studies need to be documented and not shotgunned. And getting real facts might disturb some people, not everyone likes to hear bad news about their ideas and designs.
Rigor might end up with the need to deliver bad news, as unbiased research can result in unexpected insights that contradict previous beliefs. However, rigor builds the researcher’s credibility and trustworthiness. Delivering good quality research results builds a better reputation than delivering compromises - and excuses.
Just as with any other aspect of product development, design orgs can strive to improve the rigor of their research. Getting feedback on the study design and the findings helps. Using a variety of methods, and encouraging the use of new ones supports choosing better ones. Documenting research with clear goals and objectives increases transparency. Being skeptical about the results while focusing on continuous learning promotes a culture of rigor.
In a lot of contexts, this just means rigor is leveled up via a bunch of small, indirect activities, like asking questions about study guides and design decisions on team standups and reviews by people who care. Leaders do have a responsibility to establish care.
Training is a specific activity that helps in establishing a level of rigor and sets people up for success. This goes for both internal (for researchers or the broader design team) to establish reasonable standards and external (for People Doing Research) to spread those standards. I used to say anyone can learn to do user interviews in a day, while learning to usability test takes three days. This just means even basic methods need reviewing and time spent to keep a level of rigor.
A key is to have a clear owner for rigor - a gatekeeper for research quality. A person or group who reviews study design gives feedback, constantly pushes for improvements, and increases maturity. Design or research leadership can take this on, which might produce bottlenecks. So a better approach is to delegate it, ideally to the research team who would be trained rigorously.
It does seem like hiring for good rigor is difficult for most organizations. Part of it is the leaders who are not trained as researchers and have a hard time recognizing ones with good rigor. Part of it is orgs that have no clear idea of what value a researcher should deliver, and also not clear about the value of rigor. Part of it researchers forced to do projects without rigor that has an impact on their case studies.
For design leaders hiring researchers a few things to keep in mind when watching for rigor:
- Does the candidate have a strong understanding of research methods? - “Can you tell me about a time when you had to make a difficult decision about research methodology? How did you weigh the pros and cons of different approaches?”
- Does the candidate understand what could be sources of bias in research and how to avoid those to stay objective? - “How do you ensure that your research is objective and unbiased?”
- Does the candidate appear systematic, organized, and detail-oriented?
- Can the candidate communicate effectively both about their research practice and about their findings? - “How do you collaborate with other stakeholders to ensure that research is used to make informed decisions? How do you communicate research findings to stakeholders clearly and concisely?”
At the end of the day, the true value of user research rigor for product development is that it can help to ensure that products are designed and developed with the needs of users in mind. Which in turn leads to better products.
🥤 To recap
- Rigor is important in user research as it ensures that the findings are accurate, reliable, and valid. Meaning that the research is conducted in a systematic and objective manner.
- There are four pillars of rigor in user research: credibility, transferability, dependability, and confirmability. Credibility refers to the believability of the findings, transferability refers to the applicability of the findings to other contexts, dependability refers to the consistency of the findings, and confirmability refers to the ability to verify the findings.
- Rigor can be achieved by following a systematic research process, using a variety of research methods, and involving stakeholders in the research process. Researchers should also be aware of potential sources of bias and take steps to mitigate them.
- Rigor is not always easy to achieve, but it is essential for producing high-quality user research findings. When researchers take the time to ensure rigor, they can be confident that their findings will be accurate and reliable, and that they can be used to make informed decisions about product development.
- Several things design leaders can do to promote rigor in user research, such as providing training to researchers, setting clear expectations for research quality, and creating a culture of continuous learning.
This is a post from my newsletter, 9am26, subscribe here:
🍪 Things to snack on
Qualitative Rigor is a fairly academic and detailed description from David Dwayne Williams & Royce Kimmons. Nevertheless, it’s a good and comprehensive read. Qualitative research creates rigor through the discipline of following standards and trustworthiness through credibility, transferability, dependability, and confirmability. This also provides a simple framework to check how well studies adhere to rigor.
🁈
Janelle Ward describes the internal (how precise is the research) and external (how credible are the results) aspects of rigor in Do we need rigor in UX research?. Embracing rigor means not to view it as an obstacle, but to be clear about the research process and build a reputation for quality work in the wider org.
🁈
In What does “rigorous” research really mean?, Chris Liu makes an excellent point on rigor extending to how research insights are understood and used. A study is not finished once the report is delivered, if the points are used to underpin the opposite conclusion. Ignored study limitations (for example how many people actually answered a survey) also invalidate the rigor of the study. The article proposes three tactics: Err on the side of safety, Advocate not just for findings, but specific interpretations of findings, and Be open to being wrong.
🁈
While speed is important, it shouldn’t stop researchers from doing quality research, as Joe Munka argues in Skip User Research Unless You’re Doing It Right — Seriously. Lowering rigor will result in disposable research and questionable validity with little value, and ultimately a wasted effort. Keeping rigor might take longer, but results will be longer-lasting, have broader validity, and be usable for future iterations of the product.
🁈
Adrienne Guillory writes about how cutting corners in the research process leads to problems in The rigor of UX is in the research, not the design. The article also lists some tips on how to make the research process more robust.
🁈
One reason it’s important to talk about rigor is the state of the industry. I don’t fully agree with Cornelius Rachieru’s “Everyone Used to Be a Designer”, but the point of how lowering our standards results in more problems rings true. Making the research & design process more open, accessible is collaborative is the right way to go, as becoming more agile. The rigor of our approach however should remain solid, and new tools need to be applied to strengthen rigor, rather than weakening it.
Comments