New Zealand’s tertiary institutions have some strange ways of measuring academic performance

The Tertiary Education Commission (TEC) is gaining a reputation as a Mickey Mouse government organisation that harms the reputation and integrity of New Zealand’s tertiary institutions.

In April, I called publicly for the acting chair of the TEC, Kaye Turner, to resign. Her defence of the teaching of homeopathy at New Zealand’s tertiary institutions is an embarrassment. She insisted that the TEC would continue to support the teaching of homeopathy because students want it and there are jobs for people with qualifications in the subject. Thus, skeptics and other taxpayers are forced to support courses in mumbo-jumbo if the courses sell. Dr Turner expressed indignation that I called homeopathy nonsense, and she called it an alternative to antibiotics. The TEC thus helps to spread dangerous twaddle; quackery can be a matter of life and death.

The TEC is also responsible for another abomination afflicting our tertiary institutions: the Performance-based Research Fund (PBRF). The PBRF process ranks academic staff from “A” for world class to “R” for research-inactive. It then allocates millions of dollars to polytechnics, universities, private training establishments, wananga and colleges of education, according to their rankings.

The most important part of the PBRF round requires staff to list their Research Outputs over the previous six years. A panel of experts then examines each list and awards a grade of A, B, C or the dreaded R. There are major problems with this procedure that generate unfair and invalid results.

First, the PBRF process assumes that there is a single model of what constitutes good research. This assumption may hold true in mathematics, medicine and the hard sciences, where other scholars can replicate work. In these subjects, the process of peer-review can work well.

But there are other areas of academic endeavour, such as the Arts and Social Sciences, where there is no agreed model. Scholars who rely on empirical research clash with radical postmodernists and academics who insist that there are ways of knowing other than the rational. Politically Correct fashions come and go. A peer-reviewer rejected one of my history articles on the grounds that it did not contain ‘a feminist/pacifist perspective’.

And on what grounds will the panellists rate the research? The history of intellectual activity is full of examples of work that was dismissed by experts at the time, only to be recognised later as insightful. On the other hand, peer-reviewed journals have published garbage. The hoax perpetrated in 1996 by Alan Sokal is telling; he deliberately submitted a nonsensical article to the peer-reviewed journal Social Text, which published it. And in September 2001 the Journal of Reproductive Medicine, a peer-reviewed medical journal, published a paper about how a Columbia University study showed the power of Christians’ prayers to help patients at a fertility clinic to conceive (see NZ Skeptic 75).

The PBRF process distorts research activity because it rewards academics for churning out quick projects, such as articles, at the expense of big efforts. Books in, say, history, often take years to produce. The scholar has to read first all the books and articles relevant to the field, then piles of documents that may be kept in far-flung collections. Then the writing begins, with each key assertion carefully footnoted to a reliable source. It is not uncommon for such efforts to take ten or more years, and sometimes a lifetime. But the PBRF process focuses on a six-year period. Scholars who are devoting time to long-term projects receive the inaccurate and offensive label of Research Inactive, as if they spend their time snoozing in a hammock. A Waikato University senior manager pointedly urged a historian to abandon a lengthy biography because it was an unwise research choice in the PBRF environment.

The PBRF process is distorting work at universities because it pressures academics to play a game. The university tearooms are full of talk about how to work the system. Some academics team up with friends who are referees in order to publish a couple of articles each year. The PBRF discourages in-depth projects and those aspects of the job such as teaching preparation and community service that do not help PBRF scores. It frustrates academic freedom by shaping research plans to fit its arbitrary scheme.

The PBRF also encourages departments to downgrade the importance of teaching so that the trend will be for them to become like some overseas departments I have seen, where graduate students do much of the teaching while lecturers lock themselves away to turn out articles. The TEC’s latest plan, to introduce ratings for teaching, inspires no confidence. Perhaps lecturers will score well by teaching homeopathy to lots of students.

While thinking about this subject I benefited from discussions with Dr Ron Smith of Waikato University.

Recommended Posts