Is the assessment of scuba skills objective or subjective?
All skill assessment is mostly subjective. I have in the past conducted an education workshop in which I spoke about what I call "the illusion of objectivity," which shows that supposed objective measurement systems are really subjective, but they hide that subjectivity enough that the tester is unaware of it. This can lead to truly skewed results. Research has repeatedly shown for nearly 100 years that student performance on supposed objective examinations will be scored anywhere from an A to an F by different examiners. In my workshop I actually recreated such an examination as a part of the demonstration.
An analogy is to the supreme court decision decades ago regarding pornography. One of the justices wrote that he could not create an accurate written description of pornography, but he knew it when he saw it. What he meant by that is that when he evaluated something, he compared what he saw with a mental model of what his experience told him was pornography. The problem with that is that we all have a different experience base for this comparison. A Wahhabist in Saudi Arabia will come to a very different evaluation from most of us.
In formal skill evaluation systems, such as grading SAT essays, the process calls for
calibrating the examiners. They are trained by reading a number of previously scored essays and practicing on them. It takes surprisingly little time for a group of new examiners to reach the point that they are consistently giving the same scores to the same new essays because they are all using the same mental models for comparison.
That is what the IDC is supposed to do for scuba instructors. They are supposed to get the experience that gives them the same mental models of new OW students performance with which to compare new student performance. Like pornography, it is not possible to define scuba standards so clearly that they can be used independent of those comparisons. Yes, new OW students are supposed to be able to hover, but they don't need to do it at the level of a skilled technical diver who can remain nearly motionless in perfect horizontal trim for 30 minutes.
The problem comes when those mental models change over time. In low performing high schools, teachers can get so used to seeing poorly written essays that their mental models of what constitutes an acceptable performance at that grade level can slip. In very high performing, affluent schools, the opposite can happen. The same can be true of scuba. An instructor's mental image of a passing performance can slowly change over time. In professional evaluations, such as SAT exams, experienced professionals are constantly on the alert for this, and they will recalibrate an examiner when it becomes necessary. Unfortunately, most scuba agencies have no way of doing this well. The only real way to do it would be in a fairly large shop with a very active course director on patrol, but that is pretty rare.