Survey Methods

Posted on July 11th, 2013 by Jean Cook in What We're Learning. No Comments

Previous: Soliciting Participation

Completion Rates and Attrition

A total of 7,395 people began the survey. [Note 9] Respondents were allowed to answer for themselves as individuals or from the perspective of their band or ensemble. At the end of the first three questions—which asked for consent, [Note 10] birth year, [Note 11] and citizenship[Note 12] —there remained 6,769 eligible respondents, or 91.5% of those individuals who commenced the survey. From there, respondents continued to drop out at different stages in the survey. Table 1 categorizes the survey questions into groups of questions. This provides an overview of the structure of the survey and the content of questions. The final two columns of Table 1 reports the number of respondents completing the survey through each stage.


Most of the attrition among eligible respondents occurred early, between Questions 4 and 12. From Question 4 through Question 11, 546 respondents stopped answering questions. Another 852 respondents stopped answering at Question 12 alone. The extremely high rate of attrition at that question reflects the relative difficulty of the question, which was central to the survey’s goals and will be central to many of the results I report in this Article. Question 12 asked respondents to allocate their revenue among seven sources, as well as a miscellaneous “other” category.

If respondents were not prepared with a sufficient amount of their (or their band or ensemble’s) personal financial information, they may have dropped out of the survey. Respondents were free, however, to stop the survey and start again later, and completion times ranged into the weeks.

Respondents may also have become concerned that the survey would be too demanding (although Question 12 was probably the most quantitatively taxing question in the survey). Part-time musicians or respondents early in their careers also appeared likely to drop out at this stage. [Note 13] Question 12 was placed near the beginning of the survey based on our many rounds of beta testing. Test respondents provided the feedback that it was easier to handle that question before they became fatigued. As a result, my colleagues and I expected a certain amount of attrition to occur at this point, but were pleased to have more than 5,000 respondents make it over the Question 12 hurdle.

The final row of Table 1 shows that 4,652 respondents completed the survey through the end. Respondents may have skipped or declined to answer some questions along the way. Thus, this 4,652 figure merely denotes the number of respondents who gave a response to the final question of the survey (and most of the questions before that).

Attrition during the survey presents the issue of what counts as a sufficiently complete survey for the purpose of this data analysis. Because the survey is focused on revenue sources, Question 12 is particularly important. In this Article, I will generally treat respondents who completed the survey through Question 12 as sufficiently complete to use the information we have from them. This gives a maximum sample size of 5,371 respondents. [Note 14] Because of attrition subsequent to Question 12, however, and because some respondents skipped or declined to answer particular questions, the number of data points for individual questions will often be less than 5,371. In particular, any analysis based on estimated income from music-related sources will have a sample size of 5,013, because a few hundred respondents declined to provide information about their income.

I have analyzed the dropout rates among the three different versions of the survey. One concern was that respondents who chose the short version of the survey would be more likely to drop out during the closing questions. Moreover, those taking the short version were not offered incentives and therefore likely had less motivation to finish the survey in its entirety. As it happened, 5.5% of those respondents taking the short version of the survey stopped answering during the closing questions. [Note 15] This compares with 2.9% of those taking the medium version and 1.0% of those taking the long version. This selection effect, in which relatively impatient people opted disproportionately for the short survey, could affect analyses based on the role questions and the closing questions. Impatience might correlate with various labor-market outcomes for musicians. Selection bias of this sort should not affect analyses of the core questions.


[Note 9]: This figure of 7,395 respondents does not include (1) a few dozen obviously automated responses that were easily identifiable as coming from a handful of IP addresses in China and (2) duplicate responses from the same IP address with exactly the same information.

[Note 10]: All but five respondents consented to take part in the survey as anonymous participants after being informed about the goal, eligibility requirements, necessary preparation, estimated time to complete the survey, navigation procedures, anonymity policy, and how the results would be used.

[Note 11]: Among those respondents who did consent, 208 did not enter their year of birth. All but one of those 208 did not answer any subsequent questions either; the one respondent continued answering through Question 18 but did not complete the survey. Another 49 respondents were ineligible because they were younger than 18 years old based on the birth year they entered at Question 2.

[Note 12]: Thirty-two respondents who consented and entered a valid birth year did not answer Question 3 about citizenship. Only one of those 32 answered any subsequent questions; one respondent stopped answering at Question 5. Another 332 respondents were not U.S. citizens, making them ineligible for the study.

[Note 13]: The mean hours worked by those respondents dropping out at Question 12 was 25.8 hours per week, compared to 29.3 hours per week for those in the main sample (that is, those who completed the survey through at least Question 12). The average age of those dropping out of the survey at Question 12 was 39.7 years old, compared to a mean age of 45.2 years old for those in the main sample.

[Note 14]: Of these, 83% answered as individuals and 17% answered as members of a band or ensemble.

[Note 15]: An additional 5.2% declined to provide their ZIP code, which was the final question of the survey. Those taking both the medium and the long version declined to provide their ZIP code (conditional on reaching that final question) at a rate of 1.6%.

Next: Representativeness

Comments are closed.