KPI – Part V: Scoring Rubric for Guiding Principles Factor

This post is a continuation of my previous post [KPI – Part IV: Guiding Principles]. It will describe the scoring rubric we selected to use to assign points to institutions on the Guiding Principles Factor of the Baylis/Burwell Vitality/Morbidity Model.

There are two different directions which we could have taken to develop our scoring rubric. The first way was an ultra-quantitative, spreadsheet approach attempting to measure the quality of the statements of institutional Mission, Vision, and Core Values, and the institution’s efforts to live out those statements in their actions.

This type of approach is typically called the objective approach. However, if by objective you mean “not influenced by personal feelings or opinions in considering and representing facts,” this direction is far from being truly objective. There are many points of subjectivity present in the quantitative scoring of the various components and in the weighting factors used in combining component scores to obtain a final score, where the raters’ biases and opinions enter into the equation.

The second approach is a more holistic approach which is typically labeled as a subjective approach. It relies on the use of higher education experts, who have had years of training and experience in the field of higher education, to evaluate the institution in a number of ways.

The first thing these experts are asked to do is to read the institution’s published documents and judge whether they believe the institution has selected values and behaviors that represent those of a quality institution of higher education. The institutions are scored on the following three-point scale:

-1   Totally inadequate for a quality institution of higher education

0   Barely adequate for a quality institution of higher education

+1  Describes a high performing institution of higher education

The higher education experts are then asked to judge whether the behavior of a given institution matches its stated beliefs using the following scale:

-1  Behavior doesn’t come close to its stated values. The institution fails to meet its own stated standards

0   Behavior barely meets its stated values or standards.

+1  Behavior exceeds the expectations set by its stated values.

A quality institution of higher education should be beyond reproach. In light of this, the panel of higher education experts is asked to make two more judgments.  The first judgment involves the institution’s track record with those entities and agencies to which the institution is responsible. Does the institution meet all of its required reporting deadlines and fulfill all obligations to federal and accrediting agencies? Institutions will be scored on the following scale:

-1  The institution has failed to meet more than one reporting obligation or legal requirements.

0  The institution has met all requirements and obligations but has occasionally been late or hesitant in making results public.

+1 The institution has gone of out its way in meeting requirements and obligations. It has made been completely transparent in all of its operations.

The final area of concern for the panel of experts deals with the reputation of the institution. The panel will judge whether the institution is held in high esteem by various entities such as higher education as a whole, the general public, students and prospective students, and employers of the institution’s graduates.

The scoring scale for this area of concern is as follows:

  -1  The reputation of the institution is tarnished in a number of areas with a number of groups.

0  The reputation of the institution is considered “run-of-the-mill.” It is not outstanding in any area.

+1  The reputation of the institution is stellar with all groups with which it deals.

To determine a factor score for Guiding Principles, the sub-factor scores are summed. Total scores are assigned as follows:

If the total sub-factor score is -3 or less, the assigned factor score is -1. Any institution in this area should be considered in trouble and possibly dying.

If the total sub-factor score is -2 to +2, the assigned factor score is 0. An institution with a score in this area is just hanging on and should be considered just surviving.

If the total sub-factor score is +3 or more, the assigned factor score is +1. An institution with a score in this area is doing well and should be considered to be thriving.

With the institutions we have examined we have found a predisposition away from the thriving side of the scale. It should not be surprising. Most observers will readily say that the overwhelming majority of colleges and universities are either in trouble or just surviving. There are few elite, or top tier institutions that are really thriving.

Next Tuesday, March 19, I will take a break from this series of post on Key Performance Indicators and publish a special post inspired by the scores of birthday wishes that I received this past week. It may be unusual to throw a big celebration for someone’s 73rd birthday. However, after a series of traumatic brain incidents more than a decade ago, scores of doctors wouldn’t have given you a plug nickel that I would make my 73rd birthday. Thus I will publish a post celebrating an unexpected decade of extra life. What would you do with an extra decade of life?