How to write multi-select checkbox questions and analyze the results

No one is just one thing.

And that matters when you’re writing things like those client personas that we’re all supposed to have.

But how do you capture that nuance in a survey?

One way is to use the multi-select checkbox question, e.g., check all that apply. But how do you write a checkbox question that gives you information instead of just a grab bag of data? By

• choosing the right “master” question
• crafting options that are both logically grouped and logically disparate
• formatting

(You can watch the video version of this post, or keep reading below.)

When to use multi-select checkbox questions

The world is a bundle of complex histories and characteristics, and sometimes, to really understand someone or something, you have to capture all of that complexity as a bundle.

Which is another way of talking about the fact that everyone and everything can be described via sets of characteristics.

So, if you’re trying to:

• discover the profile of your respondents with respect to a certain set of characteristics, or
• figure out how different sets of characteristics might correlate with different segments of respondents

you need a multi-select checkbox question.

What are multi-select checkbox questions?

If you’ve ever come across a survey question that lets you “check all that apply” or “choose your top 3,” you’re already familiar with the multi-select checkbox.

This question type basically groups together a related set of Yes/No-style questions under a single master question. Each checkbox option represents one of the Yes/No questions, that basically asks, With respect to the master question, is this characteristic true? For each option, a checked checkbox indicates a Yes, and an unchecked checkbox indicates a No.

The multi-select checkbox question is typically considered a closed-ended question. Closed-ended means that you supply a list of options to choose from, and the number of different answers you might get back is limited by the range of options you’ve supplied.

But what about options that aren’t on your radar?

And what about respondents feeling like the survey wasn’t for them, because, as far as they’re concerned, none of the options apply?

You can address both of these concerns by adding the option of “Other, please specify” to the end of your list of options. That “Other” option is open-ended, meaning respondents can fill in the blank with whatever option they think you missed, and you might get a different answer from every single respondent.

If you’re not sure that the question will apply to all of your respondents, be sure to add a closed-ended option for “Not applicable.”

Response rates and multi-select checkbox questions

What about the response rates for surveys with multi-select checkbox questions?

Easy to answer questions and the perception of there only being a small number of questions to answer often translate to higher response rates.

Checkboxes are easy to check and uncheck, so they can be easy to answer, and, by condensing multiple Yes/No questions into a single question, with only a handful of options, you create a perception of there being less to answer.

How to create multi-select checkbox questions

There are a couple of factors that will determine the informational success of your multi-select checkbox question:

• the degree to which your question options are logically grouped and logically disparate, and
• the formatting of the question and options.

The logically grouped and logically disparate criteria might sound like they’re at odds, but really, they address complementary issues with the checkbox options.

Logically grouping options for informational success

Ties back to the fact that you should only ever ask one question at a time, so that your respondents are really clear about what information you want from them.

You can foster this by making sure that all of your checkbox options fall into a single category.

Are they all going to be

• behaviors – like what respondents do in their free time
• preferences – like the topics your respondents would like you to address in a podcast or speaker series
• perceptions – like the types of marketing your respondents think they need to sustain or grow their businesses
• priorities – like where your respondents want to focus their efforts
• demographics – like ethnicities with which they self-identify
• things – like certifications they have

Or some other single category?

You also might want to add a qualifier to your question, like setting a timeframe or a limiting responses to a certain number of selections, to better define that category.

So, instead of just asking how respondents spend their free time, you could ask how respondents spend their free time on the weekends, or what their priorities are over the next 12 months. Or even, what their top 3 priorities are over the next 12 months.

You also need your checkbox options to be

Using logically disparate options for informational success

Within that logical group, you want a good range of options to help you cover your bases, but you don’t want the options to overlap, because that can make it hard for your respondents to figure out which options they’re supposed to select.

As an example, imagine you’re asking about a company’s priorities, and you’ve offered options for

• Upgrading your existing facility
• Updating your existing facility

You might know that in one case you mean buying new equipment, and in the other case cosmetic changes like reupholstering benches. But then are new lighting fixtures new equipment or cosmetic?

Most people aren’t even making this distinction, so your options don’t need to either.

Formatting multi-select checkbox questions for informational success

So, how might formatting affect the success of your question?

Vertical vs. horizontal formatting

First, if your survey application lets you display options vertically or horizontally, choose to display vertically, so you get one option per line. That makes it more likely that the answers chosen are the answers your respondents wanted to check, instead of accidentally choosing the checkbox on the wrong side of the option.

Option list length

Second, keep the list as short as possible. That usually means a maximum list length of 6 options.

You want to make sure that your respondents stay focused on the question they’re answering, and the easiest way to do that is to make sure that the question and the options can be viewed together.

If your question is about picking a top 3, meaning that you’re asking respondents to compare the options to each other, don’t make them scroll at all, so they can see all of the options at the same time.

Many survey tools will let you preview what your survey will look like on a standard computer screen as well as on a mobile device, and you want to check both.

If your question doesn’t “fit,”

• Can you reduce the number of options and still get the range you’re looking for?
• Or, can you reword the question to take up less space?

Using sentences as option text

Finally, if your options are written as sentences, word the options in a positive way, like “I endorse”, “I support”, “I agree”, because positive statements are easier to read and evaluate. And try to avoid commas in your sentences, because commas can make analyzing the data trickier.

How to analyze multi-select checkbox responses

The data that you get from a multi-select checkbox might take a few different forms, which is dependent on which survey tool you’re using to collect that data.

Work from a copy of the data

Regardless of your tool, chances are good that you’ll need to do some data transformation, meaning that you’re going to change some of the values, or add some new columns of data beyond what was collected by the survey tool, so be sure to work from a copy of the data, and not the original dataset.

How to analyze “Other” open-ended data

Every survey tool will separate the closed-ended options from the open-ended “Other” option. You’ll review and analyze the Other data on its own, and decide whether it warrants being incorporated into your analysis in its own right, because respondents basically provided a new checkbox option to consider, or whether you just want to treat it like a closed-ended check for an option called Other.

How to analyze the closed-ended checkbox data

There are 3 main variations for how the closed-ended option data might appear in spreadsheet view, and each has different ramifications for analysis and use.

Let’s say that your survey question #5 is a multi-select checkbox question with 3 closed-ended options and one open-ended option for Other. Each variation will have column headers, with each subsequent row representing a single respondent’s response to the survey. What changes is the column headers and how the response data is represented.

Variation #1

The first variation of the resulting dataset will list the Question 5 option text as the column headers. Checked checkboxes will be indicated by the digit 1 in the cell for that option. Unchecked checkboxes will either be blank, or filled with a zero.

That means that all you need to do is add the numbers to figure out how many respondents selected that option, which is how you can find out the most and least popular options selected.

Variation #2

The second variation also lists the Question 5 option text as the column headers, but now checked checkboxes will be indicated by the full option text.

To tally this, you’ll need to count filled, or non-blank, cells for each option column.

Depending on the type of analysis you want to do, you might need to replace these text strings with digits, meaning a 1 wherever there was a checked option.

Variation #3

The third variation is the most complex, and it’s the reason why it’s important to not include commas in the text of your options.

The column header for the question will now just be something like Question 5, or the text of the question itself. Responses to the question will be represented as a comma delimited list of the option text selected.

If you want to tally responses by option, or filter the rest of the dataset by a particular option, you’ll first need to post-process those comma delimited lists into another format, such as one column per option. Because the list is comma delimited, you’ll look for the commas as the indicator of where to split up the option selections, which is why you don’t want to use commas in your option text – doing so makes it harder to figure out if it’s a comma in your option text or a comma that’s a delimiter, and that can make this splitting up process harder to automate.

To find correlations between this data and other responses in the dataset, you’ll start by looking for relationships between the individual options selected for this question and other question responses.

How will you use multi-select checkbox questions in your surveys?

No one is just one thing, including your respondents.

So, how will you use multi-select checkboxes questions to learn more about the bundles of characteristics that make your respondents who they are today?

—

If you learned something from this post, share in the comments below!

If you’d like help designing your survey and questionnaires, or analyzing and understanding the results, set up a Zoom meeting or email us to start a conversation about how we can help.