Lies, damned lies, and statistics
The figures released by the MoJ three months into its controversial Framework Agreement fall a long way short of revealing the full picture, says Kasia Beresford in the ITI Bulletin.
The Ministry of Justice has recently released a report called ‘Statistics on the use of language services in courts and tribunals’ (www.justice.gov.uk/statistics/courts-and-sentencing/language-services-in-use). This is the first set of statistics relating to the controversial Framework Agreement (FWA) between the Ministry of Justice (MoJ) and its contractor Applied Language Solutions (ALS), part of Capita Group plc. It covers the first three-month period during which ALS was charged with providing interpreting services for all courts and tribunals nationwide.
An incomplete picture
Despite the fact that nationwide implementation was made easier for the contractor by the Tribunals Service booking its own interpreters in advance for most jobs in February (only 1,711 service requests were made to ALS by Tribunals in February, as opposed to 4,711 in March), it failed to deliver so spectacularly that after two weeks the MoJ changed its initial stance of not allowing any bookings other than via the FWA.
The new guidance issued in mid-February stated: ‘With immediate effect HMCTS will revert to the previous arrangements for all bookings due within 24 hours at the Magistrates’ Courts. Magistrates’ Courts bookings should be made direct with the interpreter under the terms of the National Agreement. It has also been decided that we will revert to previous arrangements for urgent bookings required for bail applications, deports and fast track applications in the First Tier Tribunal Immigration and Asylum and urgent bookings in the Asylum Support Tribunal.’
Clearly a significant proportion of the total workload is not being handled under the FWA, yet the MoJ has presented a report based solely on the data in ALS’s web-based portal as at 8 May (p9 of the report). This report does not reflect the use of language services in courts and tribunals as a whole, only those requests processed by ALS.
Not revealing the total number of cases requiring interpreters makes a mockery of any claim to openness and transparency, as without figures for the courts’ total requirement the true picture of ALS’s underperformance on its nationwide contract is conveniently obscured.
Performance well below target
The report includes findings, supporting tables and base data focusing on two sets of information:
■ Language service requests by outcome, by type of court, by month and by language and geographical region
■ Complaints by reason for complaint, by type of court, analysed by region and separately by month (no base data provided)
Even within the limited parameters set by this report the contractor’s performance does not meet the criteria set out in the FWA. The target for fulfilment of all assignments in the FWA is 98%. The ‘success rate’ (more later) is 81% over the quarter. The actual fulfilment rate is only 72%… and neither of those percentages is actually based on the total requirement.
There were 2,232 complaints over the quarter, of which 177 had not been resolved by 8 May (p7). The Key Performance Indicator in the FWA states that complaints should be resolved within three working days. Clearly the contractor clocked up another abject failure in terms of both the number of complaints and speed of resolution.
Interestingly there is no table showing total complaints. If you aggregate the tables provided you will find the total is 177 short of the quoted figure of 2,232. The figures given in the tables would be about 9% worse had they been included… so how valid are the rates shown?
One other obvious anomaly is that only 3% of the complaints actually included in the tables are categorised as relating to ‘Interpreter Quality’. Interestingly the MoJ’s Chief Statistician distances himself from the categorisations used in the report by saying (p9): ‘The classifications used in this bulletin… are taken directly from the management system [ie ALS’s system], and are decided according to the rules laid down by the contractor.’ Perhaps that is why issues relating to the tier of interpreter are classified as ‘Operational issues’, which make up 21% of the complaints reported in the tables. This is rather convenient for the government minister Mr Crispin Blunt, who has recently quoted the 3% figure as evidence of a lack of problems with quality.
Manipulation
The statistics have been manipulated to provide ‘improved’ headline figures for the press and all those who may not have the time or inclination to examine them in detail. Let us examine one of the key statistics – the overall ‘success rate’ of 81%.
The fulfilment rate is shown as 72%, so how does that become 81%? Well, you add 0.1% of requests when the ALS interpreter was present but the ‘Customer Did Not Attend’ and then you take the 11% so-called ‘Customer Cancellations’ out of the calculation. Well, it’s not the contractor’s fault if the customer cancelled, is it? So it wouldn’t be fair to include them in the total requests, would it?
The clue is on page 10 of the report: ‘Requests may also fail because… the supplier… does not attend (or arrives so late that the job is cancelled).’ If the interpreter arrives so late that the court can no longer proceed, that is by any normal person's standards a failure, hence those requests should be included in the total number of requests for interpreters when calculating the ‘success rate’.
I suspect that a large number of failures are hidden in these so-called ‘Customer cancellations’, because the rate of cancellation seems very high at 11%. In my experience the only type of court with a very high interpreter cancellation rate is an Employment Tribunal, as the parties often settle prior to a hearing. However, Employment Tribunal cancellations are only 3% of the total cancellations, so how does one explain the rest? Sometimes postponement of a hearing will lead to an interpreter cancellation in advance, but it does not happen that frequently. Certainly the cancellation rate of 17%, or 863 Immigration & Asylum Tribunals (IAT), appears extremely high. I have interpreted at quite a number of IAT cases and I can't recall a single booking being cancelled in four years.
Clearly the real ‘success rate’ should be closer to 72% than the 81% headline, i.e. 26 percentage points short of the 98% target!
Obfuscation
There is a file of record-level data accompanying the report, which contains the data for the 26,059 language service requests on which all the tables and statistics are based… well, not quite.
The base data has been doctored to make independent analysis difficult: 4,446 records (17% of the total) have had the actual ‘Language’ replaced by ‘Not disclosed’.
The notes provided state that this has been done ‘to protect the privacy of individuals’ for any instances where there is only one interpreting request for a single combination of court, month and language.
This ‘justification’ doesn't bear examination. First and foremost there is no data relating to individuals included in the data set. How does the fact that a specific court (or group of courts) had a language service request which was fulfilled, not fulfilled or cancelled in a particular month impact on any individual's privacy? It should be borne in mind that the data is being provided from one to three months after the event. In any case, the interpreter's identity in any specific case is not secret: the vast majority of court hearings are open to the public and the interpreter will commonly give his or her name to the judge in open court.
So what does this restriction achieve? It makes many additional analyses of the data either incomplete or impossible. In particular it would assist in covering up high failure rates in certain languages.
For example, say I wanted to calculate the ‘fulfilment rate’ for Tamil, as opposed to the dubious ‘success rate’ shown in Table 2. By summing data from Tables 8, 9 and 14, I can calculate that there were 621 requests for Tamil in total, but the record-level data contains only 541 records for Tamil, i.e. 80 requests or 13% of the Tamil data are hidden in ‘Not disclosed’, so full analysis is impossible.
The situation for Lithuanian, Vietnamese and Latvian is even worse because these languages are not included in Table 14, so there is no way to calculate the total requests for each of these languages. Hence we don't even know what % is hidden in ‘Not disclosed’.
Even for very frequently used languages such as Polish the ‘Not disclosed’ category is problematic. I wanted to look at the figures for Polish for my region, the North-West, but the number of Polish records which can't be allocated as they are concealed in ‘Not disclosed’ – 310 – is greater than the 239 Polish requests definitely associated with the North-West!
I fail to see any justification whatsoever for restricting the record-level data in this fashion. The only individuals protected by this obfuscation are those who are desperate to fly in the face of the evidence and maintain that the FWA is a success.
Revealing omissions
Finally, the most important thing about this report is all the statistics that are missing.
There is not a single figure giving information about the tiers of interpreters used, although the FWA clearly states this information is to be supplied. Perhaps it is now convenient to forget about quality in relation to the FWA? Perhaps this reflects the reality that there are no minimum standards for court interpreters anymore? It is certainly noteworthy that the detailed criteria set out for each tier in the FWA have been diluted into ‘tier-based needs’ in this report and contain no mention of any qualifications or experience at all. Similarly, the number of available interpreters, gaps in availability of languages by region and all the other Key Performance Indicators listed in the FWA are conspicuous by their absence.
These statistics do not even address the limited scope set honestly. While the MoJ turns a blind eye and refuses to accept that the FWA is fundamentally flawed, the outlook remains bleak. Proper evaluation of the FWA would reveal very poor stewardship of both the public interest and the public purse.
