Using open-ended and closed-ended questions in online surveys

Online surveys are very effective for improving the conversion rate of your website or service. You can collect quantitative data by asking closed-ended questions, and qualitative data by asking open-ended questions. Both types of questions can provide valuable insights. In this article we’re going to discuss the value and pitfalls of using open-ended and closed-ended questions.

Two types of questions

A closed-ended question (also known as a quantitative question) has a limited set of possible answers, making it easy to statistically analyze the answers.

An open-ended question (also known as a qualitative question) allows for a free-form answer. With an open-ended question you gain qualitative insights.

Answers to open-ended questions provide more depth, but may be harder to analyze in large quantities. Also, response rates may be lower, because it takes more commitment from a respondent to write a free-form answer than to select a fixed answer from a list.

You also need to consider the device where the survey is shown. Providing long text input requires much more commitment on a mobile device than on a desktop computer with a physical keyboard.

When to use closed-ended or open-ended questions?

The main advantage of quantitative questions is that they are easier to analyze, especially if you expect many respondents. Also, if data must be measured and compared over time, you need quantitative data.

One of the disadvantages of using a closed-ended question is that it can be difficult to determine the list of fixed answers from which the respondent can choose. You run the risk of missing an important option which makes your data unfit for use. Also the order of the answers can influence the respondents, because in a long list the first few answers may be favored (Informizely offers the option to randomize the list of fixed answers).

Use qualitative analysis as input for preparing quantitative questions

If you're designing a survey with a quantitative question but are unsure about the list of fixed choices to present, it's a good practice to first start with an exploratory qualitative survey to help define this list.

Start with an open-ended question and show it to a limited number of respondents:

By analyzing the free-form answers you can create a list of most popular holiday destinations for use in a quantitative survey that can be shown to more recipients.

Informizely offers a handy tool that helps analyze many free-form answers. By creating a "word cloud" of the responses, where often used words are displayed bigger than less used words, you quickly get an impression of the most used words in the responses.

In the resulting survey with the closed-ended question you still may want to include an "other" option with a free-form text field so that people can enter options that are not listed, but if this option is used too often you probably need to reconsider the list of fixed choices.

Combining closed-ended and open-ended questions

A well-known example which combines a closed-end question with an open-ended one is the Net Promotor Score (NPS). Because the data must be measured and compared over time, quantitative data needs to be gathered.

Because these quantitative data do not give you insights in why visitors gave their rating, this question is followed by an open question.

This combination of a closed-ended and an open-ended question is what makes measuring the NPS such a powerful tool: it provides you with quantitative and qualitative data at the same time.

Both quantitative as qualitative questions will give you valuable input to design your strategy to improve conversion rates. To improve your conversion rate you need to use both of them or combine them, using for example NPS or text analysis techniques as word cloud.