Last time, we looked at the question of test security in general. Today, we’ll tackle a key (and much-discussed!) issue for many testing programmes: Test content exposure and theft – what it means, how it happens and what you can (and can’t!) do about it.
What do we mean by test content exposure and theft?
A reasonable definition would be: Memorising, copying or stealing live test content with a view to enabling test takers to improve their test score.
How does theft happen?
There are many, many ways in which it happens, and for large-scale high-stakes testing programmes, ‘content-harvesting’ is a lucrative industry in its own right.
A few of the most common methods are:
Opening test packets: In pen-and-paper tests, test packets can be intercepted, opened and resealed before the test (at customs, in the courier network or in the test centre).
Photographing during the test: Button-hole cameras (and now smart glasses) can photograph pen-and-paper and digital content. Remote proctored tests provide additional opportunities to hide cameras in the background.
Memorising during the test: ‘Harvesters’ sign up for the test with the sole purpose of memorising the content for onward sale.
Recalling content when prompted: ‘Surveyors’ stand outside test centres and ask test takers to recall whatever they can of the content. The more test takers they can ask, the better they can piece together the content to a reasonable standard.
Photocopying/photographing test papers after the test: Unless test papers are immediately re-sealed in tamper-proof bags in the test room following the test, centre staff could have full access to them for several hours.
Can we prevent theft?
There are plenty of steps which testing programmes can take to make content exposure and theft less straightforward.
For pen and paper tests, ‘tamper-proof’ bags and test-day inspections help.
Strict proctoring processes can make it a bit more difficult to photograph content.
We may be able to spot harvesters who take the test multiple times, or who behave suspiciously during the test.
Together, measures like this may be enough for smaller-scale or lower-stakes testing programmes, but in contexts where there are strong incentives for cheating (recruitment to a prestigious company or university programme, or the granting of a visa or citizenship, etc), there will still be people making money from selling content.
Even if you conduct all tests in test centres and carry out airport-level security scans, there will still be theft.
So in a high-takes testing programme, it is safest to assume that once content has been used once, somebody will have a copy of it somewhere.
Can we detect leaked content, or the use of it?
To a degree, yes. We do have methods and tools to detect some leaked content and some cheating resulting from prior familiarity with content. Is it foolproof? Not at all!
Can we provide details here? No, because if the gamekeeper reveals all his/her secrets, the poacher will have a field day!
So where does that leave us?
It depends on the testing programme. If it’s low stakes, then perhaps standard proctoring coupled with some post-test detection processes is enough.
For a high-stakes programme, once we have taken reasonable steps to reduce theft, and introduced processes to detect the most obvious uses of stolen content, we have to assume that there will still be test takers who have memorised content and can recall it without detection.
We then have a choice to make. Do we cross our fingers and hope for the best? Or do we design our test so that it doesn’t matter even if the test taker does memorise the content?
That’s what we’ll cover next time.