Summary: UX Practitioners, and more importantly stakeholders must avoid making the mistake of over-generalizing or incorrectly observing user test sessions. Since the purpose of usability testing is to create informed design decisions, this is not a trivial issue. Poor (biased) observations are rampant. Let’s fix this now!
Avoiding These User Testing Observation Traps…
Observing a usability test seems straight forward: watch and form an opinion. It is that simple, but if you are not careful of the way your mind works, you can end up with the wrong set of take-aways— a potential disaster to what otherwise could be a rich learning experience. Here are the main things you need to avoid and be aware of, to get the most from your observations:
1. “Cherry-picking”: This refers to observing things that are important to your agenda, and dismissing the totality of the observed experience. The skill of avoiding cherry-picking requires lots of practice. It also requires an open mind when you are observing.
2. “Judging a book by it’s cover”: Stereotyping a user based on their segment or behavior. For example, if a user is young, assuming they know technology (and if they don’t they are obviously stupid or worth ignoring). Your best tool here is to not judge user demographics, but rather their behavior. Contrasting their behavior with all behavior, for example is very helpful.
3. “They all hated it”: Generalizing is the #1 most obvious mistake we have noticed over the years or running usability testing sessions (we’ve run over 600 tests to date). Avoid generalizing from the behaviors or thoughts of one user. Observe at least 4-6 sessions on the low side, or 8-10 on the high side (this is not a hard and fast rule, but you get the idea). It’s typical for senior managers to watch only one session and then later at the office hear them say “they hated the navigation”. Argh, just argh!
4. “Everyone is so different”: Yes and no. User segments will differ and personas or user types will differ— but usability testing is about understanding intuitiveness, which is universal, generally. On the flip side, user types can be so different, it requires you validate your observation, and thereby develop greater pattern accuracy. The key is to learn from strong contrasts in user types and across user sessions. Soon you’ll be a pro!
Where to start:
Why bother with monitoring your powers of observation? User testing is about dropping our assumptions or idea we know more than users, and learning from users and how their minds work (with our design). A good place to start with developing your observation skills is to watch all sessions (or videos if they are recorded). Live sessions are a better way to get rich learning.
What if I am using automated usability testing (eg solutions like UserTesting.com)?
Automated or “unmanned” (technically called unmoderated) testing has become very common, namely due to the cost. However, it’s an augmentation to a usability testing practice, it shouldn’t be your only source of user feedback (if you’re serious about all this). Conducting live lab usability testing can provide better insights and longer range ROI from the learning experience it offers to teams. Poor observation problems can be more rampant, because of the disconnect with the user. In addition, automated usability testing: a) Lacks live observation- you can’t ask users questions; 2) Lacks metrics capture- there’s no way to record the patterns eg 8/10 users struggled…unless you manually take notes which can be time consuming; 3) Lacks depth- most sessions are typically 20 minutes max, compared to 60-90 minutes for live lab usability tests.
Conclusion: In watching a usability test, learn to observe without bias. Avoid common perception traps, get data, evidence aka pattern intelligence before you decide something is the way you think it is! This will ensure your observation is clean and your data is clean. More importantly it will add confidence to the design decisions you make based on observation.
Learn how to get the best usability testing insights on this Full Day Course: Usability Testing Methods training
