Question Order Effect

The Question Order Effect (order effects bias) is a cognitive bias found in surveys. It's a phenomenon that occurs when the sequence of questions in a survey or test influences respondents' answers.

In other words, answers to later questions can be significantly conditioned by earlier questions, potentially leading to false or distorted data.

One of the most famous examples of research into the question order effect was a survey prepared by Herbert Hyman and Paul Sheatsley in 1950 during the Cold War. They asked whether the USA should allow reporters from communist countries (e.g., the USSR) to come to the USA and send information back home about what they observed. The second question in the survey was the reverse, asking whether the USSR should allow reporters from the USA to visit their country and convey information to US media. In the study, the percentage of 'yes' answers to the question allowing communist reporters to visit the USA increased by 37 percentage points if the question was placed second. This is due to the so-called norm of reciprocity.

Methodological threats associated with the question order effect in surveys and tests:

1. Distortion of results: The Question Order Effect can lead to a distortion of employee satisfaction survey results, as earlier questions and the answers given to them influence the responses obtained from respondents.
2. Misinterpretation of results: If a researcher does not account for the Question Order Effect, this can lead to incorrect interpretation of results, which in turn can result in HR making poor decisions (or those conducting the survey).
3. Reduced test reliability: In the case of knowledge tests (used, for example, in recruitment tests or product knowledge tests among employees), the Question Order Effect can lead to reduced test reliability, as the order of questions can influence the answer choices made by candidates.

How to mitigate the Question Order Effect?

1. Randomisation of questions. One of the most effective ways to prevent the Question Order Effect is to use a random order of questions in a research survey or recruitment test. This way, respondents will not be able to anticipate subsequent questions, and it will not influence their answers.
2. Inclusion of control questions. Adding control questions at the beginning and end of a survey or test can help assess the impact of the Question Order Effect on respondents' answers.
3. Grouping questions by topic. This allows for a logical sequence between questions.
4. Data analysis. Researchers should also thoroughly analyse the collected data to detect any potential distortion of results caused by the Question Order Effect and adequately account for them in the interpretation of results.

Remember! In the tomHRM Surveys and Tests module, you can randomise questions. They will then be displayed to respondents in random order, helping you to reduce the Question Order Effect.

Test the module for 30 days free of charge!

In the case of HR-focused surveys, it is particularly important to pay attention to the Question Order Effect in employee satisfaction surveys. The order of questions can strongly influence the quality of the answers received.

For example, studies have shown that if a positively worded question about salary satisfaction ("How satisfied are you with your salary?") was placed in a survey, followed by a question about overall job satisfaction ("Are you satisfied with your job?"), respondents answered more positively to questions about overall satisfaction, regardless of whether they were actually satisfied with their salary (Feldman and Lynch 1988).

Summary:

The Question Order Effect can pose a serious threat to the reliability of data in surveys and recruitment tests. To avoid this phenomenon, HR professionals should apply appropriate prevention methods, such as randomising the order of questions and thoroughly analysing collected data. Only in this way can reliable and credible results be obtained to help make informed decisions in team management.

Also check out employee survey examples!