Disputes in British Universities

In November 2006, the BBC program 'More or Less' carried a feature on "Multiplre Choice" questions of the type common in British exams whereby you have to match up items from two columns. The identified problem was that if, just by way of example, you had four options to match up, then you could never get 3 out of 4 correct - you could only get 4 out of 4 or 2 out 4. The BBC were not impressed with the reply they received from the examination board AQA.

Dear Mr Daugherty,

Here's the statement from the AQA that we carried on the programme yesterday. We were sincere in our request for comments, principally because we don't see how the answer from AQA addresses the specfic problem identified by our listener, namely that two candidates of equal ability - at the 75 per cent level - can score differently. AQA seems to be saying here that it doesn't matter that you can score 100 per cent by knowing only 75 per cent of the answers, since the last two questions could be scored as one mark only. Fine - as far as it goes. However, that's no solution if some people at the 75 per cent level score 100 per cent and some score 50 - there will still be a false difference between their scores.

Best wishes, Michael Blastland.

(The Statement from AQA)          

The argument that there is a mismatch between ability and marks is based on the false assumption that...  the difference between scores tells us something about the difference between candidates of a different ability.

The number of marks awarded on a particular question cannot be interpreted on a linear scale of ability. Thus scoring 1 mark on one question may imply greater ability than scoring 4 marks on another question.

AQA uses Item Response Theory models to ensure that the mark scales they use do indeed represent coherent measures of ability and are satisfied that these question types contribute positively to these measures. The fact that you cannot score 3 marks on these questions is irrelevant as the difference in ability implied between scores of 2 and 4 may well be less, for example, than that which exists between 0 and 1.    

1. University departments are immediately accountable to an external examiner. Unfortunately, external examiners are appointed by the very same departments they are supposed to be supervising. Presumably, it would be likely that the external examiner and members of the department are already known to each other, either from conferences, or guest seminars, and the like, and are possibly likely to have a very cosy relationship.
   
   
   • It reminds me a lot of the way that owners of oil tankers are able to choose which country to register their ships in. At places like Newcastle University the identity of the External Examiner is kept an absolute secret.
   
   
   
2. It is not unnatural to expect that any major grievance about exam results would be dealt with in the usual way i.e. before some hearing carried out in the accepted manner, with experts to represent students, cross-questioning allowed etc. (as per the standard model offered by the judiciary of this country). This is not the way most British Universities operate - it varies from University to University.
   
   
   At Newcastle, the sole method in which a student can express a grievance is to write a letter to the Dean of Faculty, who then deals with it, in secret, without any further communication with the student.
   
   
3. The only other action you can take is to appeal to the University's Visitor. To quote The Guardian : 'For students caught up in the Byzantine process of making a complaint about their university, the "visitor" system may seem Dickensian. In fact it's not that modern'.

   For the last few years, there has been an ongoing campaign to get rid of the Visitor, boosted by the European Human Rights Act. The Government even said at one stage that it was goiing to abolish the post, but this does not seem to have been fully achieved yet.

Here is a particular opinion offered on the nature of the Ombudsman.

Abolition of the External Examiner system. Steps in this general direction were proposed in the Labor Party proposals when they were in opposition. We all know that this is a different thing from actual implementation by a Labor Government but at least the subject is in the air, it is not just an unrealizable idealization.

The institution of a Watchdog for the Universities. Since Watchdogs already exist for many areas such as Gas, Electric, Telecommunications etc.etc., it would be no great innovation to set up such a body for the Universities. In fact, Universities are notable for being about the only public bodies immune from this type of accountability (as already mentioned we are awaiting how recent developments in this field turn out).

And along the way, institute an appeal structure that bore more of a resemblance to the standard model set by the British Legal System.