A Philosopher's Blog

Adventures in Assessment

Posted in Universities & Colleges by Michael LaBossiere on August 22, 2017

I’ve fallen behind on my usual schedule of posting and replying to comments. The reason is, of course, my adventures in the realm of assessment. This began in 2004 when I was assigned to eternal membership on the General Education Assessment Committee. I am now a co-chair of the committee. I was also assigned to do the unit assessment for philosophy and religion. The basic idea of assessment is to assess using various direct and indirect measures. As might be imagined, this has little to nothing to do with philosophy, although I do try to sneak in the occasional philosophical bits. These are, as you might guess, typically edited out when the documents are reviewed.

My university is in the process of re-accreditation, something all schools do on a regular basis. My task is to complete a major document for a specific standard–the document is currently at 11, 184 words.

I have some upcoming essays that I hope to complete tomorrow and perhaps the assessment grind will permit me to get back to my usual writing a reply cycle. And, you know, teaching and stuff.

But, here is a look at what sort of stuff I write for assessment with some philosophy. Philosophy that will be excised in the final version, of course.

Overview of Target Levels and Measure of Success

The establishment of target levels and measuring competence requires addressing two basic concerns. One is determining what counts as competence in each assessed area.  The second is setting a percentage goal for student competence.

The second is easy to address. In the United States educational system (broadly construed), 70% has been established as the minimal level of adequacy. As such, adopting the broad standard that 70% of the students assessed will perform at a level of adequate competency or better is justified by this established measure. Justification for this measure, in general, can be sought in whatever theoretical, practical and philosophical foundations were used to make this the national standard. The first is rather more challenging to address.

Justifying a standard of competence is difficult because of an epistemic problem raised by the ancient Greek Skeptics. If a standard is not self-justifying, it must be justified. If the justification is not self-justifying, it must be justified. Philosophically, this must lead to either a regress (infinite or circular) or a self-justifying foundation. As there seem to be no self-justifying foundations for standards, the regress problem wins the day and all standards are ultimately arbitrary. Fortunately, there is a pragmatic solution to this problem: presenting a plausible narrative for the standards that convinces the relevant authorities to accept them. This is what follows.

To measure the competence of an individual student in an assessment area, there must be an established standard of what counts as competent. To use the obvious analogy, to measure the height of a person, there must be an established and consistent means of measuring. One way to define competence in education is in terms of how the average student performs in that area. This is analogous to sorting out what is “normal” height—it is based on what is average in the relevant population.  As such, assessing the competence of Florida A&M University students required knowing the national average for comparable students in the relevant competency areas. To this end, the ETS Proficiency Profile (EPP) was utilized to set the standard—specifically the national mean. This standard is used in the areas the EPP tests: Communication, Critical Thinking, and Quantitative Reasoning. Since this method is accepted by the relevant authorities in assessment, it is justified.

While the use of standardized tests solves some of the assessment problems, it does not solve all of them. Specifically, it does not solve the problem of assessing areas that are not well-covered by standardized tests (such as Social/Ethical Responsibility) and it does not solve the problem of assessing individual artifacts, such as philosophy papers. Fortunately, there is an established solution to this problem, namely the use of rubrics. The main challenge with a rubric is developing it so that it properly and consistently sorts students into the specified levels of competence. While all rubrics are flawed in some manner, Florida A&M University began in 2004 with established rubrics from other universities and refined them over the years in accord with both national and local findings to ensure that best practices were being used. Since these rubrics are accepted by the experts in the field of assessment, they are justified as means of assessment.

Other methods of assessment, such as focus groups and surveys, are also established as accepted methods by the relevant experts in the field of assessment. These methods are, of course, crafted and deployed in accord with the best-practices as established by the relevant experts in the field. Thus, these methods are also justified.

Advertisements
Tagged with:

Florida & Performance Pay for Teachers

Posted in Politics, Universities & Colleges by Michael LaBossiere on April 14, 2010

The Florida senate recently passed a bill that gets rid of seniority based pay  for teachers and replaces it with a performance based system. Apparently half of the performance assessment will be based on test scores.

Since I am a professor and not a teacher, this will have no impact on me. Thanks to budget cuts, merit based pay increases are a thing of the distant past. However, I am generally in favor of tying pay to job performance. This, one might say, just makes sense. One reason is that it provides a motivation to do a better job (or a motivation not to do a worse job. Another reason is that it seems to be just and fair.

The concept of performance pay is simple: better performance yields better pay. The challenge is working out how to objectively measure performance in education.

As noted above, the plan is that half of the assessment will be based on test scores. On one hand, this would seem to provide an objective measure of how well a teacher is doing. On the other hand, using tests as a measure is fraught with problems.

One concern is whether the tests actually measure true learning as opposed to the ability to take such tests. Having taught classes on standardized tests, I learned that there are numerous ways to teach students how to do well on these tests that really have nothing to do with what might be considered “real” education. Teachers could simply train students to well on these tests without actually teaching what should be taught.

A second concern is developing a fair, objective and secure system for such tests that adequately controls for factors that are beyond a teacher’s control. To use an obvious example, a teacher teaching at a well funded school located in a rich neighborhood who happens to have the best students in her class will no doubt get some very good test scores. In contrast, a teacher at a woefully underfunded school in a terrible neighborhood who has been assigned the worst students at the school will have rather bad test scores. As such, the testing methodology will need to sort out these sorts of factors if it is to be used to determine teacher pay.

Based on my own experience in education, test scores do have some  limitations as measures of teaching performance. For example, I teach two sections of Ethics in the fall. I use the same material, tests, and papers. Yet I have ended up with very different test averages in the classes, although my teaching ability is obviously the same in both. The main variable factor seems to be, obviously enough, the students.

If such factors can be adequately controlled, then the test scores would be useful measures, to a degree. However, resting so much of the evaluation on a single factor does seem problematic.

However, I do think that linking pay to performance is a good idea, provided that a fair system of assessing performance can be established and instituted.

Reblog this post [with Zemanta]