Maxdiff is a superior technique for the research of preferences or importance. In our presentation at the 2010 AMA Market Research conference in Atlanta, my colleague Kathryn Korostoff from Research Rockstar and I made the case for MaxDiff and discussed its advantages over rating, ranking and constant sum questions.
Rating questions are susceptible to:
- User scale bias: this includes acquiescence bias (tendency to agree with everything), extreme responding (using certain parts of the scale) and social desirability bias
- Scale meaning bias: a scale point can mean different things for different respondents
- Lack of discrimination: respondents often rate everything as preferred or important
Ranking questions’ limitations include:
- Order bias: we get different results depending on whether the respondent ranks the items from highest to lowest or vice versa
- It is a difficult task as respondents have to evaluate all items at the same time to determine their ranking
- Only a limited number of items can be tested without increasing the level of effort required from the respondent
- Provide ordinal data which limits the types of analysis we can do with the data
- Don’t allow for ties, which can occur in reality
Constant sum questions’ weaknesses include:
- It is a difficult task as respondents have to evaluate all items at the same time to determine the number points that they need to allocate to each item
- Like with ranking questions, only a limited number of items can be tested without increasing the level of effort required from the respondent
- Respondents engage in response strategies trying to make the task easier (e.g. allocating equal amount of points to each item; given all points to one item, etc.)
Given the problems with each of these question types, particularly with rating questions, has led to an increased interest in the use of Maximum Difference Scaling or MaxDiff as is commonly called.
Maxdiff is a trade-off analysis technique that allows us to do multiple pairwise comparisons in an effective way by asking respondents to select the most and the least preferred or important items out of a list we want to test in search for the greatest differences among items.
MAXDIFF BENEFITS
- Strong discrimination power
- It is a simple task for the respondent
- Allows to test a larger number of items
- Eliminates scaling bias
- Allows for diversity, which is necessary in international studies
- Provides ratio data and a measure of magnitude
THE PROCESS
In order to implement MaxDiff we need to:
- Identify the number of items to test.
- Create an experimental design that provides frequency balance (each items appears the same number of times), orthogonality (each item is paired with other items the same number of times), position balance (each items appears the same number of times in each position).
- Estimate utilities for each items using Hierarchical Bayes analysis or MNL and rescale them for interpretation.
The standard output of MaxDiff analysis is usually a ranking of the items tested based on rescaled utilities, but these can also be used to conduct further multivariate analysis such as correlations analysis, multiple regression, t-testing, TURF analysis, cluster analysis, latent class segmentation, etc.
MAXDIFF APPLICATIONS
MaxDiff can be used to study preferences for and importance of a number of things including:
MaxDiff is not perfect. It usually takes longer for the respondent to take, and depending on your research goals, the relative measure it provides may not be what you want. MaxDiff helps you prioritize within a given list of items, but it doesn’t tell you if all are preferred/important or not from an absolute perspective. However, the latter is less of a problem as we can include additional questions, which allow us to calibrate the MaxDiff ranking to “absolute” levels of importance or preference.
Nonetheless, next time you need to measure preferences or importance consider using Maxdiff instead of traditional approaches such as rating, ranking or constant sum questions. You will gain in data quality, greater discrimination and the ability to provide better insights to support business decisions.