Sometimes I’m asked to review surveys or analyze data collected via surveys developed by clients and more often than not I find rating scales, (aka Likert scales) of different sizes and directions within the same survey. When I ask why, I get answers such “It” or “This is the one we have always used.”
It seems rating scales are often chosen based on preference or habit (e.g. legacy surveys), which is not surprising since there is no consensus on what rating scales work best. They all yield different results, which is disheartening in a way.
There has been a lot of research dedicated to this subject illustrating there is no simple answer to the question on which rating scales we should use.
Source: International Journal of Social Research Methodology, Vol. 13, No.1 Feb. 2010, 17-27 (Hartley and Betts)
This extensive body of research shows that different rating scales are bound to yield different results as we are mainly dealing with human perception. Rating scales mean different things to different people and the values, words, and order in which we present them have an impact on how they are interpreted. What to do?
- Whenever possible, favor question formats other than rating scales. For example, Maxdiff has been shown to discriminate better in preference and important measurements.
- If you still have to use rating scales, strive for consistency and use them with full knowledge of the bias they introduces in the data, particularly if you want to analyze data from different rating scales and data from different surveys. This is particularly relevant in tracking studies. A change in rating scale from one wave to another may show artificial significant differences mainly due to the measurement error introduced by the change in scale.
- Above all, triangulate the results with other data sources to understand how different scale points correlate with actual behavior, and ask why the person gives a particular rating. If possible use a text analytics tool to get at the heart of what the scale really means for a respondent. The example below says it all.
The post Which Rating Scales Should I Use? appeared first on Relevant Insights.