Learning from Your Users: Research in Software

In this video presentation, projekt202 Principal Experience Researcher Kelly Moran takes an informative look at bringing ethnographic research into software design.

With real-world examples and personal lessons from the field, Kelly explains research practices that set up projects for success and deliver software that customers appreciate and use.

A transcript of Kelly's presentation follows.

I have a story for you. Two anthropologists go to an island. They decide to split up and each anthropologist will just study the islanders on that side of the island, and after six months they'll come back together and discuss their findings, so they do this.

After six months, they come back, and the first anthropologist says, "I learned something really exciting. I pointed at a rock and I said, 'What's that?' The islanders said, 'Unga bunga.' Then I pointed at a mountain in the distance and I said, 'What's that?' The islanders said, 'Unga bunga.' This has really amazing implications for how these people conceptualize things like space and distance. This is an amazing finding and this is really going to advance our discipline a lot." The second anthropologist said, "That is really interesting. On my side of the island, 'Unga bunga' means 'pointing finger.'"

What we have here is someone who makes a lot of assumptions, and another person who actually takes time to understand and to learn.

Let's talk about you now. You make software. Making something usable is hard enough, but making something that people appreciate? That's a whole another thing entirely.

This can be frustrating because you work really hard, and your users just don't get it. But, have you tried getting your users first?

I'm Kelly Moran, principal experience researcher and anthropologist with projekt202. I have fancy book learning and official degrees, and I also have some time spent out in the wild, as they say, in Peace Corps and in study abroad.

Now I work with projekt202, and what I do is uncover user needs. projekt202 is pretty special because researchers, designers and developers all have a chance to grow and evolve their own specific disciplines, but we come together and work really well. No one outshines the others. This can be a really difficult thing to do, particularly when people have had bad experiences in the past.

Let's get that out on the table now. I want to talk about stereotypes and preconceived notions. We'll start with me. I hear a lot, "You're an anthropologist. Are you studying me right now?" The answer is, no one wants to do their job 24 hours a day. I need a break.

The second is, "I watch the show Bones. Do you like dead people?" This is really more what I do here.

The third one I hear a lot is kind of painful. I'll refer you to the previous slide for that one.

Developers' turn. I hear, "Developers aren't very good with people. In fact, you don't really like people. You're not comfortable around them and you'd like to go hide somewhere dark." This is a really unhelpful thing to think about people that you need to work with.

The next one is, "Can you fix your computer for me?" Well, let's refer to anthropology stereotype one; no one wants to do their job all the time. We should respect the skills that we have and understand that this is an important part of our work, and not what we want to do all the time for free.

Third, "Developers aren't creative." I hear this one a lot and it really makes me sad. Everybody gets to be creative. Developers make stuff. They create things that weren't there before. This is the definition of creativity.

Designers' turn. Now designers, "They just want to make pretty stuff all the time." There seems to be a lack of understanding of the skills among all of us and we should appreciate those more.

Next, "Designers aren't capable of leadership. They're too flighty," I've heard. I've seen that one misproven a lot in my organization.

Finally, "Designers think that creative genius should be the final word." Well, when you get a group together that really works well, you see that everyone can come together and bring their own creativity, and really do the best for the user. With our powers combined, we can build something really great.

However, there's a piece missing from this puzzle. "Users don't know what they want, don't appreciate how hard our jobs are, and are just plain doing it wrong." Well, now we're back here. Right? Where everybody's thinking sort of unpleasant things about each other. How do we fix that?

Experience research is one way of doing that. That's going to direct the strategy and the design of software projects, and set them up for success.

When I do experience research, I rely a lot on my background in anthropology, as well as on the fresh ideas and perspectives of all of my colleagues. When I think about anthropology, though, I really think about this quote from Margaret Mead. Margaret Mead said that, "Anthropology demands the open-mindedness with which one must look and listen, record in astonishment, and wonder that which one would not have been able to guess." This is really important to keep in mind when you're building software for people.

I have another story about another tribe. This is a tribe that lives on an island where everyone sees things in blue. On a neighboring island, there's a tribe where everyone sees things in yellow. These two tribes, let's say they have a hard time seeing eye to eye. One day, a member of the blue tribe decides he wants to understand what it's like being with the yellow islanders and understanding how they live their lives. He puts on a pair of yellow lenses and he goes to visit. After a week, he comes back and his fellow blue tribers say, "How was it?" He says, "You know, it was really interesting, because everything was green."

When we do research on users, what we want to remember is that everybody has their own lens, their own way of interpreting things around them. We want to keep that in mind and use that approach when we're learning.

What I use is an ethnographic approach. Let's talk about how we can apply that.

Ten things about ethnography: The first is that ethnography is the descriptive study of peoples and cultures. That's the big dictionary definition. What that means is that ethnography is both a process, you go out and you do ethnography, and it's also the product that results from that. It's a descriptive written study.

Ethnographic research, by the way, is an approach; it's not a specific method. There's a lot of different ways to do ethnography. The idea is that you have a philosophy of open-mindedness and understanding, and a couple other things we're going to go through now.

Ethnographic research favors qualitative over quantitative. It's great to supplement with quantitative data when you have it, but by definition, ethnography wants to get that qualitative view. In fact, anthropologist Clifford Geertz calls it thick description.

As an example of that, let's talk about winking. Now, there's a couple of ways to interpret a wink. If you want to just describe it thinly, you could say that someone closed and opened an eye. That doesn't give a whole lot of meaning behind that. We know that a wink can be someone flirting. It can be someone letting someone in on a joke. It can be having something stuck in your eye. If I have thinly described a wink and said that, "Shelly winked at Joe," I might be implying something I don't mean to imply. Instead, I want to offer a thick description and talk about what was happening around that behavior. You want to do that when you're observing your users so that, when you come back and other people are going to have to use what you've seen, they can interpret it responsibly.

The next is that ethnographic research must be conducted in context. It's typically done over a period of time, which can be difficult to do with software. It's holistic, it seeks that wider picture. You want to see, again, the context; what's happening before and after things. We say that in order to understand the pen, you must first understand the paper. If you were designing a pen and you didn't know what kind of paper it was going to be used for, you might not design the best thing.

These observations lead to innovations.

Finally, ethnographic research is systematically conducted and responsive to emerging trends. What that means is, sometimes people think ethnographic research is a little bit fuzzy because it might look like we're going out and hanging out, but there are goals. There are big questions we're seeking to answer, but, on the other hand, we need to be responsive to what's happening in the field because people are unpredictable, and we might see something we totally didn't expect and weren't ready for in our research plan, so you need to be able to be responsive.

When you're learning from people, remember to be flexible.

The sixth point, ethnographic research utilizes key informants. This is just a fancy way of saying you've got people who know what's going on. These can be stakeholders, subject matter experts. They'll guide you and help make sure that you're interpreting “unga bunga” as “pointing finger.”

Ethnographic research also seeks out an insider perspective. In anthropology, we call this emic. That's different from the outsider perspective, which would be your etic perspective as the designer or the developer or the researcher. What it does is it takes that outsider perspective and layers it in as you're looking for interpretation, and when you're looking to innovate and design, and build solutions. It kind of combines those two, right? The yellow and the blue lenses are needed.

A good example is from the Superman movie. Lois Lane says, "What's the S stand for?" Superman says, "Well, it's not an S. On my world, it means hope." Well, Lois Lane's not really taking a very good ethnographic approach, and she's a little snarky and says, "Well, on my world, that's an S." Not really taking the time to understand what that means to Superman and what she can do with that information once she learns it.

Ethnographic research is also generative. This is important. It's not the time to evaluate your current solution. You can do that later and you should, but when you're doing that first initial research with your users, you're looking to generate new ideas. Sherlock Holmes got this right when he talked about how you shouldn't theorize before you have data, because you'll just end up twisting those theories to suit the facts.

Ethnographic research also seeks to tease out the implicit things, the things that aren't typically stated. Those things that people won't come right out and tell you. We call this making the familiar strange and making the strange familiar.

There's a really good example I have. Let's say aliens came down, and they asked you, "What do you do in the morning? How do you do your day?"

People will tell me that they get out of bed, they take a shower, they get dressed. Sometimes they switch up the order of that on accident; they're not thinking. They eat a couple times a day. They hug their children. Those types of things. Those are the things that are important for them and they want to explicitly state that to you.

What happens when that alien goes back and builds a habitat that doesn't include air, because nobody said they breathe? When we go out and talk to users, we want to find those things that are literally as natural as breathing. We're looking for air.

Ethnographic research is also inclusive. This is really important. That means that you include your users in what you're doing. You want feedback from them, they're part of it and they can give you a really good insight.

All of that was pretty complicated. How do we bring that into software? Well, there's a couple of tips that you can use to really make sure that you're using the ethnographic approach efficiently when you're doing this with software.

The first one is that really, when you're doing ethnography for software, you want to do that to reduce the possibility of failure. You want to understand what's going on so you can provide the best solution. Research shows that 75 to 90% of new products fail because no one did any research. We just kind of thought it would work.

The second one is that design ethnography avoids an over-reliance on self-reported data. What people say they do just isn't always what they actually do. An example from this comes from work I did a long time ago in grad school. We were trying to understand what green means. The client wanted to design something for customers that identified as green. This was when that was a relatively new term and they just weren't sure how to reach these green people. They didn't even know what it was. They couldn't even use the term flexibly within the organization. They said, "Find out what it means to be green."

We went out and we gave everyone disposable cameras, and we said, "Take pictures of things around your home that are green." They did that and they gave us the cameras back, and we developed those pictures. We looked at them, kind of sorted them out, talked about them amongst ourselves, developed some questions, and then took those back to those users. We said, "Tell us about these pictures."

One of the things we saw a lot of in those photos was these eco-friendly cleaners. The Seventh Generation brand in particular was very popular at this time. We said, "What do you use these cleaners for?" They said, "We use it for everything. We do all our cleaning this way." We said, "Do you ever use anything other than the Seventh Generation brand cleaners?" "Oh no, I would never do that. I have children in the home. I wouldn't put chemicals on my surfaces." It was really important to them that they keep their homes clean and friendly.

Then we said, "Well, can you show us where you keep these cleaners?" We were just following this idea that, maybe they had special sponges or something that go with them. We weren't really sure. Sometimes you just have to follow those hunches.

We were in their homes, and what they do is open up their kitchen cabinets and their Seventh Generation non-toxic, eco-friendly cleaners will be right next to a big bottle of Clorox bleach.

We'd say, "What do you use the bleach for?" They'd say, "Oh, we just use those on the really big cleaning jobs." They weren't lying, they were telling us about their aspirational selves. They really wanted to use those green cleaners exclusively, but they just weren't getting the job done.

It was important for us to learn that people wanted to be green, but lacked confidence in green products at that time. That was important for the client when they were designing and marketing.

The third thing about bringing ethnography to your projects is that it is qualitative research done in context and it looks at the user's problems from their point-of-view. What color are their lenses? That's what you want to see.

Let's talk about integrating in context research into software now. We start with the Revealing Reality phase, and in this a designer and a researcher pair up together and go out into the field. The researcher leads, and the designer is there for first-hand knowledge and experience.

After that, we come back to the office and we switch. The designer takes the lead and the researcher is there to serve the designer in whatever way they need. I might need to go back to the data and pull a quote or find some specific examples to help the designer in their work.

Then we move into building and evolving. The researcher kind of steps out at this phase. The designer and the devs will work together to build the best solutions. The researcher may come back in to do testing, and that's always ideal. In this way, we've got user input in the discovery, design and build phases, which kind of keeps the user's voice alive throughout that whole process.

I've got some lessons from the field to share with you. There's just a couple of examples from a couple of projects that I've done.

One was some enterprise software for accounting personnel. This is what professional accountants inside a corporation use to balance their accounts. This isn't personal software. This is professional, really heavy-hitting software that has to be able to handle millions and billions of dollars, and thousands and tens of thousands of transactions.

We went out and we observed, and we found a couple of things. One of the things we found is that managers were so frustrated by the way the attachments could be viewed or uploaded that they were telling their staff to only attach one file. What that meant was that lots of files would get compressed into one PDF, one big nasty spreadsheet usually, and then attached. Then the manager would go in there and review all of that big spreadsheet because they didn't want to have to click through multiple attachments.

Another thing is that the accountants were building their own cover sheets on the first page of this attachment, and it was kind of summarizing all of the account activity for the month. That's exactly what the software was supposed to be doing. Basically, people were manually doing what the software does and then attaching it as a file attachment in the software, because that's what their chief financial officer wanted to see in eventually.

Then there's my favorite observation. This was the individual who would need to look up information in the software, but would have the results return in such a clunky way that she would rather get up with a broken ankle, hobble down the hallway to a locked room, get a binder, bring it back to her desk, flip through it, find what she needed, write it down, close it up, and repeat the process because that binder had to stay in a locked room all day long. This was something she wasn't going to tell us in a survey.

These observations lead to innovations. These were all really tactical things for this particular project and sometimes that's what happened. We did things like take that cover sheet and actually rename the page in the software where that was supposed to go. Right now, the software had a separate area that was called the account home, but that didn't mean anything to these users. They had their own word for it. They called it a cover sheet. We renamed it a cover sheet, we made it look more like they were making it look, and now they could use it in the software comfortably.

We also did things like making drag-and-drop file attachments a whole lot easier and we made it so that when you opened an attachment, it stopped taking you away to a new page of the software. That way, you could look through multiple files, arrange and name those files in a more sensical way, and not have to make those big, nasty worksheets.

Here's another really fun example. This was some B2C software involving sustainability in farming. What happened was, the client wanted to help farmers find more sustainable ways to do their farming practices. We went out, we literally went into the field, and we talked with farmers and we watched them do their work. One thing that we found from the farmers was that they have an extremely practical view of sustainability.

You and I, we have really aspirational, lofty goals in terms of sustainability, and we really want to save the world. That's good, but the farmer's where the rubber meets the road. The decisions that they make today impact their fields next season. When you start talking about making the world better, it just doesn't impact them the way that they need to hear that information. We needed to be able to talk to them about immediate impacts and they understood these things, but the client wasn't reaching them, so they just didn't even want to have the conversation.

The other really important thing we learned in this project was that farmers have a very strong sense of pride and craftsmanship. Farming is a craft and farmers are master craftspeople. It takes generations to learn how to work a farm well. When the client was approaching them in a way that seemed to suggest that the farmers still had a lot to learn about doing things right, it was a really bad way to get those farmers to adopt those more sustainable practices.

Really help with the branding. Really what you need to do is get out in the field and see what your users can show you. I hope you can start doing that.

I want to just take a really quick moment to talk about testing. Testing isn't really ethnographic because you're taking it out of context, but it is an attempt, if you do it right, to understand and see through the lens of your users. I want to offer that as one way to start bringing that research into your organization if you're not already able to do that with generative research.

Testing can be really formal or it can be pretty scrappy. The idea is that you start talking with users. Start talking with them in testing and validation, in concept validation, and from that you'll start to see how going out and talking with them on the front-end is really valuable as well. Just get started.

How can you bring research into your projects? Like I said, get out in the field. Put your boots on, get out there. Find three users that you can talk to. Find two users that you can observe. Go and start learning. Once you do that, you'll find it's easier and easier to make this a regular part of your practice.

I also want to take a moment to talk about informed consent. When you're going to go out and talk to people, if it's informal and just a conversation, it's totally okay to make consent verbal. You can just say, "Hey, I want to talk to you about this product you're using. I'm from such-and-such organization, and I'll probably take what you say back to my colleagues and we'll work on this together. Is that okay?" That's fine. If, though, you're going to take photographs of those users, if you're going to do audio recordings, if you're going to take anything away from them, you need written consent. That's okay. People are really happy to help you, you just want to make sure you're really upfront and clear, and that it is informed consent. They know why you're asking and they know what you're doing with it.

How to observe: A couple of good tips for observing here. One of them is that you want to make sure that you are taking note of the physical environment. Is it open? Is it cramped? What's the lighting like? For software, those kinds of things are really important.

You also want to look at the people. Who's interacting with whom? Who's talking to who and for what reasons?

Then you want to see the artifacts as well: the equipment, the paper notes, things like that. You want to take note of those things and document it all. Notes, notes, notes. If you've got those consent forms signed, take photographs as well.

Asking questions: this is important. The first thing you need to remember is that you are not an expert in their work and play, even if you think you are, even if you're building software for designers or software for developers, you're not an expert in how they design and they develop.

The next thing is to rephrase what people say and ask them if you got it right. People love to hear their words brought back to them. In everyday conversations, people really don't listen very well. You're spending a lot of time thinking about what you want to say after your friend has stopped talking finally. Rephrase some things, and that reinforces to people that you're listening and that their words matter to you.

You also want to avoid those leading questions. Things like, "This page is really difficult, right?" You don't want to make people think that they should agree with you to make you happy. Ask more descriptive questions instead. Things like, "What about this page is working well? What about this page is not working well?" You'll get a more honest answer that way.

Finally, you want to take note of the ideas that they have, and take them back. Now, sometimes people get frustrated at this because they say, "My users have really bad ideas. My users have ideas that aren't sustainable. I know what this software needs to look like down the road, and that's not going to work."

You still want to write it down, and here's why. Everybody remembers Homer Simpson, right? Homer had the opportunity to design a car. He designed a really bad car. It did terribly. I think it sunk the market and lost him his long-lost brother. The thing was, Homer Simpson had problems. He expressed those problems in the way he tried to design the car. Now, if the designers had instead taken those ideas and followed up with what problems they were solving, they could have provided more professional, streamlined ways of solving those problems.

The last thing I want to ask you to do when you're doing research with your users is to honor the idea of reciprocity. You're taking something from them, so make sure you give something back.

I have a quick recommended reading list that's kind of fun if you're interested in learning more about ethnography.

One of them is called Guests of the Sheik. This is about a woman who wasn't originally an anthropologist but married one, and for her honeymoon ended up in Iraq. Not the best honeymoon in my opinion, but apparently she had such a great and inspiring time she came back and became an anthropologist. It's a really good read. I really recommend it as an intro to ethnography.

The next one is also another good light-hearted read [Gang Leader for a Day]. This is a sociologist. He went out to one of the roughest areas of Chicago to study gang behavior. He went with a clipboard and a survey, and he walked up to some men and said, "What is it like to be a black man in America?" This didn't really go over well. They literally held him captive in a stairwell until their legitimate gang leader arrived. Luckily, this guy was a little bit intrigued. He said, "If you want to know what it's like to be black in America, come back tomorrow." To his amazement, this guy did. He spent then several more years going back and visiting this group, and learned some really interesting thing about economics and gangs.

The final recommendation is pretty heavy. This is definitely very academic, but for those of you that want a big heavy read, Saints, Scholar, and Schizophrenics. It's about mental illness in rural Ireland. Very heavy read, very heavy topic, but also really foundational. It's there for those of you who like to dig really deep.

I want to leave you with one thought about anthropology. This comes from Bronisław Malinowski, another foundational anthropologist. This says, "Ethnography has a goal, of which an ethnographer should never lose sight. This goal is briefly to grasp the native's point of view, his relationship to life, and to realize his vision of his world."

Thank you for your time today.

projekt202 is the leader in experience-driven software strategy, design and development. We have a unique and established methodology for understanding people in context — we reveal unmet needs — which drives everything we do. This leads to a crisp, clear understanding of the customer, which shapes the design and development of new solutions and experiences. We have the expertise, teams, skills and scale to deliver sophisticated software solutions that improve any and all touchpoints across the user journey.

projekt202 has spent over 15 years bringing to life and to market compelling experiences through our Experience Strategy & InsightUser ExperienceSoftware DevelopmentMarketing & Analytics, and Program Management practices. Our talented team has delivered emotionally-rich and intuitive solutions for global brands and clients such as 7-Eleven, Allstate, Canon, Capital One, Dell, McKesson, Mercedes-Benz Financial Services, Neiman Marcus, Samsung Electronics, Subway and The Container Store, among many others.

projekt202 has offices in AustinChicagoDallas and Seattle.