Lessons Questionable science/validity of the Dunning-Kruger Effect: Implications for competency self-assessment in diving.

[lowwall]

Let's take a step back and summarize.

The "Dunning-Kruger effect" states that the less people know about something, the greater they overestimate their knowledge of that thing. Here's their own words:

…people who lack the knowledge or wisdom to perform well are often unaware of this fact. We attribute this lack of awareness to a deficit in metacognitive skill. That is, the same incompetence that leads them to make wrong choices also deprives them of the savvy necessary to recognize competence, be it their own or anyone else’s.

This has been debunked. The method used to produce their results was flawed. This has been graphically demonstrated (both literally and figuratively) by showing that completely random data will produce the same result.

However this does not mean that people have no biases towards over or under estimation of their abilities. Various papers that have looked into this using valid methods for measuring over and under estimation. While they have found no effects of the size originally reported by Dunning and Kruger, they have found that:

• everyone overestimates their ability to a small extent on general knowledge or skills tasks
• everyone slightly underestimates their ability on complex knowledge or skills tasks
• men overestimate their knowledge or ability regardless of skill level by a moderate amount
• women very slightly underestimate their knowledge or ability regardless of skill level
• Dunning-Kruger does exist, but at a much lower level, i.e. only small overestimation by the unskilled and a very slight underestimation by the highly skilled
• the greater the knowledge, the lesser the amount of error in estimating that knowledge

Of course some of these biases are so obvious as to be uninteresting and others are contradictory. The latter suggests this will remain a fruitful area of study for those seeking academic standing. But the lack of finding of any large biases suggests that none of this has any practical ramifications and thus we might as well quit talking about it.

 

[yle]

Well, I have an important question then. So the DK effect is not scientific, i.e. there is not truly a general correlation between lack of experience and overestimation of ability. But there certainly are some individuals that display this behavior. Can we instead claim they have "DK syndrome"? Acknowledge that it only affects some people, so at least we still have a convenient label for it?

I'm asking just so we can all communicate effectively moving forward.

 

[Cthippo]

I think this may evolve into a case where we are using the term wrong, but since we all agree what it means, even if that was not the original meaning, it works.

 

[lowwall]

Well, I have an important question then. So the DK effect is not scientific, i.e. there is not truly a general correlation between lack of experience and overestimation of ability. But there certainly are some individuals that display this behavior. Can we instead claim they have "DK syndrome"? Acknowledge that it only affects some people, so at least we still have a convenient label for it?

I'm asking just so we can all communicate effectively moving forward.

I think "teenager" already covers it.

 

[wetb4igetinthewater]

I think "teenager" already covers it.

When I had my heavily modified WRX, I was still driving like I did when I was a teenager. But when I was a teenager, I didn't know about heel-toe.

 

[-JD-]

When I had my heavily modified WRX, I was still driving like I did when I was a teenager. But when I was a teenager, I didn't know about heel-toe.

Teen ride - 80HP Diesel in a 2-ton car.
Long term benefit - You have to learn all of the tricks 'cause speed ain't commin' easy ...

 

[tursiops]

This has been debunked. The method used to produce their results was flawed. This has been graphically demonstrated (both literally and figuratively) by showing that completely random data will produce the same result.

I don't think this is a fair summary. Even Wikipedia explains it better than that.

Also, much of the criticism of D-K comes from ignoring the empirical results because no reason for them can be agreed on. No mechanism, so the data must be wrong. That's bad science.

 

[lowwall]

I don't think this is a fair summary. Even Wikipedia explains it better than that.

Could you provide a better one?

Also, much of the criticism of D-K comes from ignoring the empirical results because no reason for them can be agreed on. No mechanism, so the data must be wrong. That's bad science.

That is not the issue here. Did you read the autocorrelation link provided by the OP? This criticism is based on their statistical methodology. The conclusions D-K have drawn are based on a completely flawed method.

I can show the flaw in their method with a simple thought experiment. But first we have to look at one of the actual charts on which their claims were based.

The horizontal axis is formed by taking the test scores and breaking then up into 4 groups. The vertical axis is used for two things, both expressed as percentiles. First is the test score. Second is the average of student's guess about what their score would be for each quartile.

D-K's findings are based on the difference between the two lines. When the solid line is above the shaded line, they declared it to be overconfidence. When it's below, they found underconfidence.

The problem is that the shaded line has no inherent meaning. Because it uses the same data points to form both its axes, it is simply a plot of x=x. Another way to put it is that the shaded line actually is the x axis. it's just been redrawn at a 45 degree angle from where it belongs.

Contrast this to the solid line which has two independent variables as axes and thus gives you potentially valid data. In this case it shows that perceived ability increases as domain knowledge increases.

If you haven't grasped it yet, here's the thought experiment. Take the same students and their same guesses on their perceived ability and replace the test they took with a toss of a 12 sided die (borrowed from my old D&D set). Assign those who rolled 1-3 to the bottom quartile, 4-6 to the 2nd quartile, etc.

Now what will your graph look like? Well, it will look something like this (please pretend the solid line is both straight and horizontal).

So Dunning-Kruger triumphs again, right? Those dumb bottom quartile kids sure are overconfident. And those top kids once again showed their good breeding by slightly underestimating their ability.

But what are we really seeing? Let's start with the easy stuff. Comparing the solid line to the x axis, we see nothing. There is no change in perceived ability compared to quartile. This is expected because there is no difference in each of the groups, they were assigned randomly. Why doesn't the solid line bisect the middle of the shaded line? That's actually a well known cognitive bias called above-average effect (which is discussed in the original paper). It turns out that the average human thinks he or she is quite a bit above average, so the solid line is shifted above 50%.

Now what about that difference between the solid and shaded line? The one that D-K uses to show how overconfident the dumb kids are? But there aren't any dumb kids here, just a meaningless line that should never have been drawn on the chart.

 

[happy-diver]

Gareth lock flys around the world supervising his minions on
not knowing what they don't know and how stupid they are.

101 Human Factors In Diving With Gareth Lock - ScubaGuru

In this episode of The Dive Locker Podcast we learn about human factors in diving with Gareth Lock.

scubaguru.com

 

[lexvil]

Does this thread prove or disprove the theory?