Flipping a coin after (disappointing) testing? Add in a qualitative component and let the users decide
In this post we use a case study from an engagement at projekt202 that dealt with features for inclusion in a mobile healthcare app.
After using the Kano method to test 21 features, we ended up with three negative features: one Reverse (best to exclude these) and two Indifferent, which should each clearly be left out. Among the remaining features there was just a single Must-Have. No features ranked as One-Dimensional (the more it exists, the more the user is satisfied). The other 17 features all ranked as Attractive.
That’s a poor strategy for allocating design and development resources. Additionally, the reason you run a Kano study in the first place is to stop using politics in feature decisions, as we discussed in part 1 of this post. Avoid a scenario where politics and coin flipping come into play by adding a qualitative discussion component to your Kano (or any!) user test.
In our one-on-one, moderated Kano sessions we organized the features into categories based on what they primarily addressed: Doctor Recommendations, Costs and Medical Bills, Advice and Education, and The Mobile Experience. Before getting into the features of a category we situated the participants in the realities of today with a discussion including things like: How do you accomplish this now? What would you do today if…?
This provided us with valuable data to return to when we needed to make actionable recommendations on attractive features. Using this data we identified several features ranked as Attractive which we determined were best excluded, for now, based on factors such as the effectiveness of participants’ current strategies. For instance, if participants had an existing method to accomplish what an attractive feature did, and they really liked and felt in control with that method, we recommended not allocating resources to that feature right now, since there isn’t enough need behind it to motivate users to change their current behavior. They were solutions without problems.
Other attractive features brought clear and immediate value or offered an exceptionally delightful experience. As mentioned, the discussion portion of the session revealed how beaten down and accepting participants were of the lack of cost transparency in health care. Features which offered assistance understanding pricing logically should have ranked as Must-Haves (features that people expect), but didn’t because people don’t expect anything better – yet. Any feature bringing price transparency would disrupt the current system, so we recommended that attractive features addressing costs be included in the initial launch.
Using qualitative data helped us better understand some of the more puzzling Kano results (transparency isn’t a basic requirement?) and allowed us to make recommendations on which features would combine into a first launch with a meaningful impact. Participants can have a hard time articulating what they really do and how they’d really feel about something in their actual environment – this is not a replacement for high-quality in-context design research. A Kano/Qualitative hybrid approach does however offer a more informed way of dealing with potential features than a simple survey, a politically charged corporate argument, or a coin toss.