Summary: UX efforts should improve the quality of data and informed decision-making by making sure feedback is coming from the right users. Clean user recruiting practices are critical for getting the right data from your user research efforts.
Why you need to avoid recruiting “fake users”
“Fake users” are the wrong people who will provide you with biased, tainted, inaccurate, and imprecise feedback. Precision is everything if you want to be successful with UX. When you recruit the right users, it’s priceless.
At Experience Dynamics, we learned this from doing it the other way. Like many, we used to have a Market Research firm recruit for us, but now we do it all meticulously in-house. So around ten years ago, we stopped using our Market Research firm. We didn’t like the quality of users they were giving us (as one of the best firms in the market).
Market research firms provide basic user recruiting due to their focus group services. But it’s also very common to have user no-shows, lukewarm relevant feedback, or someone who feels like a badly hired actor. We have not experienced any of that for many years– that’s when you know you are doing it right.
Authenticity is critical in getting clean data. It’s that simple. We are excited when our clients get the best data possible.
Who are fake users?
Anyone you use as a substitute for whatever reason to represent your real user.
- Your CEO.
- Your employees.
- Your agency brings you the wrong panelists.
- Consumers who are only on a panel to make some money.
- You sloppily recruited respondents from your market research firm.
- Your mother.
- The guy down the hallway, in the cube next door, on floor 3.
Why are they dangerous?
1. The wrong people give you dirty UX data. Fake users tamper with your data providing unclear or foggy findings. Note: You can bias your own findings by doing this, which is why a full dedicated UX agenda, team or expert can make a quality difference.
2. Motivation, intent and emotion add big insights to your UX research. Users recruited who are not “in the act” you are observing are never as meaningful as people motivated organically by the thing they are doing— relevant to your product. Make sure you recruit people doing what you are studying!
Experience Dynamics is often brought in to fix the damage of a brand/ interactive or marketing agency (sometimes another UX agency). We’ve seen your vendors do a focus group involving the CEO; we’ve been only allowed permission to study ‘neutral’ employees on a consumer product; we’ve been told “we cannot talk to our users” or that the advice from Steve Krug and Jakob Nielsen was to “grab someone from down the hallway”.
See Why participant recruiting is non-trivial, and why nobody is talking about this
In a rush to evangelize Usability and Usability Testing, a very bad habit emerged that is still tolerated in many organizations: using fake users to get insights that should come from bonafide users.
How to avoid recruiting fake users:
- Make sure you, or your agency are recruiting users who are doing the thing (task, problem, domain) you are studying.
- Professionally screen potential users.
- Double or triple screen users and make sure they feel right for the study.
- Re-recruit or have extra people on standby to recover from a “low data” user.
- Do decent sized samples of users, none of this 5-user stuff… at minimum test 10 users, do chair-side visits with 15 users.
A Note on Sample Sizes: At Experience Dynamics, our user tests are typically 10-15 users, and field studies typically 20-40 users. We have found this to yield the correct amount of insights, empathy, and empirical validation of business, product and UI concepts.
See The 5 User Sample Size myth: How many users should you really test your UX with?
