TDSB Survey Results Can Be Misleading
August 23, 2020
If you plan on making data-driven decisions and that data is being collected via faulty techniques, you're going to have a bad time.
This week the Toronto District School Board announced that it was re-planning how students would return to school after survey results showed that more parents than expected were planning to send their kids back to school.
Whether classes are at 50%, 75% or 100% capacity makes a big difference in how you might plan this. In this case, only about 15-30% of parents said they were planning to use online education, with the rest preferring coming back to school. Given this data the TDSB is delaying the start of the school year.
The problem is the survey used which asked two questions:
- Would you be comfortable with online learning?
- Will you be sending kids back to school in September?
Here's the issue: this is loss aversion theory at its finest. Parents who have little intention of sending their kids to school will answer 'yes' to the second question because they want to leave their options open. They may have done so only to avoid the loss of the option of sending their kid to school, without actually having any intention of doing so.
Given the dynamic situation around Covid where people are monitoring daily cases and weekly averages, day-to-day decision-making is based on recent and relevant data because the value of data decays very quickly.
The flaw with TDSBs approach is that it failed to give weight to the importance of recency. Parents will not accurately state their intentions for a future time period without knowing the Covid trends preceding that period.
So what to do if you're TDSB and want to have an idea of how many students might show up?
You have to look at proxy metrics and not ask the direct question. The question that they can ask is this: if school started tomorrow, would you send your kids back to school?
This is a better question because as a surveyor you are gauging current attitudes based on current information. It is more accurate to extrapolate this answer and Covid statistics, and apply them to a future date, than asking people what they would do given a future, hypothetical situation.