How to write open-ended questions and analyze the results

There are so many good use cases for open-ended questions, and including them as part of your surveys can really increase the insight you can glean from your respondents. But:

• How do you write questions that get you the type of answers you need?
• How do you reduce all of that free-flowing text into something that’s manageable and usable?

(You can watch the video version of this post, or keep reading below.)

When to use an open-ended survey question

We all like a good hero’s journey. But, when it comes to your surveys, you are not the hero. Your respondents are. So, if you’re trying to:

• get out of your own way and figure out what the heroes care about,
• figure out how to describe something in the hero’s words,
• get an explanation for why those heroes answered the way they did on a previous survey,

you need to use an open-ended question.

What are open-ended questions?

The label open ended refers to the fact that you’re not restricting respondents to certain answers. With an open-ended question, you provide a question prompt, and then respondents can answer with whatever they want to write in text or paragraph form. That means that you might get a different answer from every single respondent.

This is in contrast to closed ended questions like checklists or Yes/No questions, where you supply a list of options to choose from, and the number of different answers you might get is limited by the range of options you’ve supplied.

Response rates for open-ended survey questions

Due to the amount of writing time involved, as well as some other factors, open-ended questions can decrease your response rates, especially if you have a lot of them. That could mean that people skip over your open-ended questions, or it could mean that they don’t submit their responses back to you at all. So, it’s best to use this question type only when you cannot capture that information another way.

How to write an open-ended survey question

To create an open-ended question that really takes advantage of this data collection opportunity, you need to let respondents know what you want them to talk about so that you get the type of responses you need without leading them to a specific answer.

Open-ended questions should start with words like: How, What, and Why:

• How would you describe
• What is
• Why do you think

So, instead of asking a Yes/No type question like:

Do you want help growing your business?

You would ask something like

What is one thing that would help your business grow?

Now, if you left it there, you’d probably get a lot of answers about money, which doesn’t really tell you anything, because that’s money to do what, exactly? Remember, this is about creating a question that gets you the type of responses you need, where that type is informational for you. That means that it’s okay to add a qualifier onto your question so that it’s not completely open ended, but actually heads in the direction you’re interested in.

So, instead of just asking, What is one thing that would help your business grow?

You could ask:

If money weren’t an object, what’s the first thing you would do to enhance or grow your business?

And if you wanted to completely eliminate money from the range of answers, you could ask something like:

Aside from purchasing from you, how would you like others to show support for your business?

Those question revisions are still How, What, and Why questions, but now they invite open-ended responses of a certain type, and still without leading to a specific answer. 

So, with your own survey questions, figure out which questions really need that open range of responses, then write them as a How, What, or Why question, and, as needed, add a qualifier to keep respondents focused on the range of things you want them to talk about in their answers.

How to analyze open-ended survey responses

Regardless of how well you’ve crafted your open-ended question, the data that you get from open ended questions will be a list of sentences or paragraphs. So how do you make all of that text usable and informational?

You’ll need to post-process this data a little bit to look for patterns and relationships.

Create a word cloud to assess and communicate open-ended survey results

If you’ve ever seen a word cloud, you’re already familiar with one way to look for patterns – word counts.

The process here is to break up all of that text into a list of individual words, and then count how many times each word appears in the open-ended responses. You’ll ignore filler words like ‘the’ and ‘and’, so you can focus on the meat of responses, that is, the nouns, adjectives, and adverbs. The final result, when displayed as a word cloud visualization, will change the size of the words based on how many times it appears in the responses. So, bigger words are more common than smaller words. There are tools that automate this process for you, so you don’t necessarily have to do it yourself.

If what you want to do is assess or communicate the frequency or popularity of words and phrases in your text data, this is probably the fastest way to do it, but it’s not necessarily the most informative or useful analysis you can perform.

Sentiment and thematic analysis of open-ended survey responses

If you want to delve a little deeper, you might want to apply sentiment or thematic analysis to your text.

With sentiment analysis, you’re basically determining whether the attitude or opinion expressed in each response was positive, negative, or neutral. You could tally these sentiments as-is, or combine them with thematic analysis.

With thematic analysis, you’re basically assigning each open-ended responses into one or more categories so that you can group responses together by those categories. Those categories might be a list of things you’re looking for in the data, or they could be phrases, concepts, and other things that emerge from the responses themselves, that is, you can create the categories as you read through the responses, based on what people said. How you approach this depends on your use case. Depending on the number of responses you have, and the nuance involved, you might want to do more than one pass through the responses to refine your category labels.

Once you’ve reduced all of that text to a small set of categories, it’s easy to do things like tally by category or find relationships between these categories and other data points in your survey data set. You can also use these categories as the basis for a word cloud.

Depending on how much data you have, how quickly you need the results, and your budget, there is natural language processing software you can use to help you with both sentiment and thematic analysis.

How will you open yourself to insight with open-ended questions?

Whatever method you use to analyze your open-ended text data, whether it’s word counts or themes or some other way to uncover patterns, the goal is to capture its essence while also reducing it down to something a little more manageable because you’ve standardized it in some way, which makes it more easily quantifiable or usable in correlations with other data.

—

If you learned something from this post, share in the comments below!

If you’d like help designing your survey and questionnaires, or analyzing and understanding the results, set up a Zoom meeting or email us to start a conversation about how we can help.