Wednesday, November 14, 2012
1. The plural of 'anecdote' is not data.
If you shamble together a bunch of stories from your life as proof towards your point, it only proves that your path is possible, but not probable.
An example: if you went on welfare and were able to successfully raise your child, get an education, and a full-time, well-paying job (i.e., you're a welfare success story) this does not mean that everyone in the country that needs help has the exact same opportunities. It merely means that one person was able to accomplish this, not that everyone can.
Another example: just because you have talked to <10 people from the military and they all support Willard, does not necessarily mean that individuals in the military all support him too. It is possible, but without larger quantities of information you cannot support that claim.
2. The average ____ is not like you.
In learning science research we have a saying that goes, "the student is not like you" which can be applied to most other situations. In general, the quantity of variables that went into constructing the environment around you, as well as the environment within you are so numerous that you're really only representative of that exact combination of factors.
This is where statistics and data analysis comes in, not for determining how 'representative' you are, but for determining what demographics in populations actually matter when it comes to differences in health, choices, salaries, etc. To counter the effect that the >trillions of random factors that went into creating who you are, statistics requires large, random samplings of the population. Not just one story from your life.
A single story proves that a thing is possible, that there's a slight chance it might be important, but it is proof of little else. People who might appear similar to you, may make completely different choices because of those >trillion small ways they are actually different from you.
3. Be aware of selection biases
A graphic of every cognitive bias: http://www.visualcapitalist.com/every-single-cognitive-bias/
Now, even if you had multiple stories to corroborate your point, you still have to be aware of selection and cognitive biases. That is, by simply sampling from people around you, you cannot make any broad statements about the country in general. Probably not even the state in which you reside.
An example: if you believe that this is a generation of quitters because you have seen the people around you quitting their marriages, jobs, families, etc. how do we know you haven't just seen all these instances of quitting, because you've surrounded yourself with "quitters"? By only sampling those who are in your life, there is a tremendous selection bias on your observations (unless, of course, your point is to say that "the people around me are all quitters of their jobs and marriages").
[Edit 7:30pm: Also related to all this, the cherry picking fallacy, in which you only pick and choose examples that fit your story]
(BTW, did you know the divorce rate has been going down since the 1980s?)
4. Be aware of confirmation biases
So, maybe you haven't surrounded yourself with quitters, but maybe you have this running opinion that everyone is a quitter and this helps you make special note of the instances in which people quit, while biasing you against instances in which people do not quit.
Basically, a confirmation bias is a type of cognitive bias that will force you to only see events that confirm your theory, while making you blind to events that suggest a different truth. It's a little bit like a mental horse blinder.
5. Control of variables
Remember how much better things were in the good 'ole days? You know, back when things were simpler because we didn't have GMOs or women's lib or gay marriage activists or whatever? Many things have changed from the way things were back whenever. Mainly, you were younger (possibly even a child) and didn't have to make difficult decisions. It often has nothing to do with women's liberation and everything to do with the speaker being older.
When making comparative statements, make sure that everything except the one variable of interest is kept the same. This makes comparisons over time especially difficult, because time is always changing and can be a significant source of conflation especially when it comes to stages in life.
6. Science improves
You know what also has changed since the good 'ole days? Science. Just because "scientists" formerly believed the sun revolved around the earth in the 1500s, or in the 1970s scientists thought there was a global cooling, does not mean that all science is forever incorrect. It does, however, mean you need to be able to interpret data and statistics if you want to make any statements for or against a particular scientific theory.
7. Correlation versus Causation
Recently, there was a study that ran through the internetz about how increased consumption of chocolate by a population might cause increased Nobel Prizes per capita in Switzerland. This is sort of an example of where correlation might not mean causation - there could be a hidden factor that causes an increase in chocolate consumption which is the true cause of the increase in Nobel Laureates. Or maybe consuming more chocolate does make your country more likely to achieve a Nobel Prize. Without further statistical analysis, one must be very careful about claiming causality from correlational data.
I'm a big fan of this figure that illustrates the correlation vs causation point.
8. Structural Factors versus Personal Choice
The most well-known set of structural factors that affect people's options are prejudices such as sexism, racism, etc. where you are cut off from possibilities over something you cannot control. A recent study gave academics fake job applications, with the names randomly assigned to be male or female. Males were overwhelmingly chosen to be more qualified than females, despite the resumes being identical. Sociological Images summarizes the study, here. The glass ceiling fits in this category.
Another, trickier structural factor is the rewards system in our society. Such as...why do women take longer to get ready in the morning than men? Is it because they choose to put on make up and spend an hour on their hair? Why do they choose to do all this? Couldn't they just spend ~10 minutes and stroll on out the door? For most careers and women, there are repercussions for leaving the house having only spent 10 minutes getting ready.
Sociological Images really explains this better than me, here:
"Someone inclined to believe that boys and girls are different biologically may say that women are naturally more fastidious and concerned with appearance.
Someone disinclined to a biological explanation may say that this is the result of socialization. That Disney princesses and made-up moms teach little girls that they need to be “pretty” and so girls prioritize this (over sleep) as adolescents and adults. They do this automatically because they’ve internalized the idea.
A sociologist, however, would likely argue that neither of the above are true. As one of those, I would argue that the reason women, on average, spend more time on their appearance is because (1) the bare minimum for looking presentable is different for women than for men and (2) the social costs for neglecting their appearance is greater for them than it is for men. It is not biology, nor socialization, but the realities of social interaction that draw women out of bed earlier than men. We learn that our appearance matters to others and that others — strangers a little bit, friends more so, and bosses and lovers especially — offer rewards and punishments related to how well we conform to their expectations. So we make a measured choice. We primp and preen not because it’s natural, or because we’re socialized robots, but because it’s worth it or, conversely, we don’t want to pay the cost accrued when we do not."
So, these rewards & punishments function as one sort of structural factor that affect how we live out our life. In general, it always has me second guessing the meaning of "choice" and whether or not there really is such a thing.
9. How biased are your sources?
Rather than summarize the entire domain of information literacy, let's just focus on one main point for choosing where your information sources come from: authority. This goes beyond simply whether the source is Wikipedia or a .com or a .edu. Is it peer-reviewed? How many other peer-reviewed papers have cited it? Is it a primary source, or is it some mouthpiece interpreting the results for you?
If it's from the news, are other news outlets reporting it as well? We generally trust the news for reporting on events, but some news outlets are not exactly unbiased authorities on knowledge (I'm looking at you, Fox News and Huffington Post). Sociological Images summarizes a report about biased news outlet in the 2010 election so you can see that this has a pretty serious effect on people's understanding of the world around them.
10. You are only entitled to a well-informed opinion with references.
Simply put, how do we know you aren't just making up scientific theory or numbers if you do not supply your references? How do we know the sources you're basing your claims on are reliable, authoritative, and not-biased?
If you do not have the time to back up your statements with actual facts, then you really should not say anything at all, or at least don't be surprised when someone criticizes your statements for being unfounded.
[Edit 1/14/13: This link from PSYblog seems rather informative here as well.]
5 comments:
I love this post!
man, so basically unless you can stick every single person in a lab(the lab would have to be as big as the world) , and run the same experiments on them, you cant say "everybody know... (insert broad stroke brush here)". how sad. because generalizations make the world seem easier to understand.
On the bright side, you yourself don't always have to be the person to run said experiments (sometimes they don't have to be experiments, they can simply be polls/surveys of random segments of the population). Oftentimes, the data is already out there, accessible on the Internet. People just don't go through the effort to check, and because of that, they're often wrong.
Common sense is not always right. In fact, oftentimes, it's surprisingly wrong.
awesome post!!
LOL - omg Iris. My dad is totally the anecdote = data type of guy and I've inherited his style of argument. I have to be extra careful when I'm trying to make a point not to do that :)
Post a Comment