Ok, I've held off on this because I wanted to think a little more about it, see if I could come up with some answers before exposing my ignorance. But I think we're at a crossroads in our discussions here, so I'll ask the question:
Why's Nick Yee's work so fatally flawed?
Here's the context of the question as I see it. Yee acknowledges that he's working with a self-selected sample of the population, and I think I understand the inherent problem that self-selection brings. However, in statistics, isn't there a certain numerical point where a sample size, regardless of selection method, crosses a threshold and begins to become significant? Maybe I'm wrong, but at a certain point I'd assume that the sheer power of numbers begins to overwhelm the self-selection, and the sample becomes a valid representation of the population.
Further, how can we overcome the problems that you seen in Yee's methodology? It seems to me that if we're studying customers of proprietary virtual worlds, and the owners of those worlds either can't or won't make the population available to us for us to draw a statistically relevant sample, how do we proceed? If we go out with an open call for feedback, don't screen the responses, and get back 50,000 responses, isn't that an adequate sample to make some judgments on?
I guess I look at it like this - if we're studying the incidence of a type of cancer in the general population, there's really no one that "owns" the right to look at medical data or "cause of death" data, so researchers with some skill can get access to the information they need. But in the situation of Blizzard or Sony Online, they could make the argument that it would be detrimental to their relationship with their subscribers/players/customers, and detrimental to the players' in-game experience, to allow researchers to contact players for research purposes. From the game developers' perspective, facilitating research becomes not just a hassle, but actually a negative outcome. If that's true, and as a researcher you're still committed to garnering player opinions, how do you avoid something that looks a lot like Yee's solution?
Understand that I'm not arguing for sloppy research, but looking at both Yee and the PARC Play On projects, both seem hamstrung by the gatekeeping function of the publishers. Are we left to wait until we can somehow garner support from the publishers, or can we take half steps (which is how I'd see Yee's work) until such time as the playing field shifts?
As I said, I've been hesitant to ask because I have the feeling I'm asking something I should have already understood about academic research, but since I clearly don't... I'm looking forward to being enlightened.
Okay, you're right and you're wrong. Here's the straight poop. THere are two fatal flaws with Yee's work (btw, he really hides his sampling info., you have to dig to find it. I"ll post a link to what I found.)
ReplyDeleteFIrst, survey research itself is flawed. People are reflecting on or reporting their alleged behaviors; there is no unbiased, reliable observation of what they say they do and don't do. People are making sense; they read an item and they interpret it. Who knows what they meant. People often misunderstand questions or have trouble with the particular phrasing or language in the question.
Second, we have no idea how representative the sample is of hte actual population. YOu cannot generalize from sample to population without knowing how the sample came to be. First of all, how does Nick solicit respondents? How do people get there? Who stays and bothers to answer? Do noobies? Do folks like Skuggan? like Turbulence? Like Akmalla? Yee argues his gender and age breakdowns mirror those of Blizzrd. How does either know age? Blizzard only knows who pays for the account. Does Blizzard count me as one person cause I paid for Akmalla and for me? Basically Nick is handing out flyers in a parking lot.
Nick's work is ONLY problematic when he and others generalize from it. Unfortunately, as it is the only stuff out there, they tend to rely on it a lot. The PARC stuff I've only just started to see. I didn't know it was out there. However, you'll note that PARC has a clear top level link to their methodology, and they recognize the limitations of what they call the dreaded level 50 scraper cap.
Unlike Yee, PARC gathers data from inside the game world, and they have built a mod with the intention of gathering a reliable sample.
They say, in part:
We've been harvesting data on and off since late March. (We have a manual upgrade process after each patch.) To date, we have taken about 6000 snapshots, or roughly a thousand snapshots per server per faction. Afterward, we're analyzing the collected data with our own Jawa application and Excel, but that's a story for another post.
THe question for them is, do they vary servers and time and day of the week? Do they do US and EU and Asian servers, or what?
In answer to your question, at the end, I think Blizzard can't give reliable data either...at least of the sort we're talking about here. The most reliable thing to do would be to "play within ourselves," to do what we an reliably do. FOr instance, one might ask a very large guild if members would agree to respond to a survey and do a follow up interview over the net, e.g., in chat or voip/ventrilo. Then one would be very careful to explain in one's write up of the research, that this is how the sample was obtained. That's an example. THe source of the sample and the info about the sample help the reader decide to what extent these results should be generalized.
Did that make sense?