Designing a usability test
We set out to compare the usability of this map-based interface with the conventional paper questionnaire, viewable online, produced by the PBNI.
Selecting the testers
There is no point getting students to test the interface since what we wanted to know was whether ex-offenders, some with low literacy levels, found the map-based interface more or less usable than current consultation techniques.
NIACRO
We needed to recruit ex-offenders to test the system. The Northern Ireland Association for the Care and Resettlement of Offenders (NIACRO) runs IT courses for ex-offenders and others. They have a small computer room in which trainees can develop IT skills that may increase their employability. A number of probationers take these courses. So NIACRO arranged for some of their trainees to test the interface when they came in for their regular session on 13 April 2006.
Selecting the tasks
We designed a sequence of tasks that started with simple familiarization with the map-based site, then got progressively more involved as the testers gained confidence, until they were entering comments on the consultation topic.
In brief, the sequence of tasks were:
- Explore the PBNI e-consultation website.
- Find your probation office or reporting centre on the map.
- Find the all offices or reporting centres you have been to, and add comments on each place.
- Find your new office and reporting centre, and comment on how it would affect you.
The control task was to complete as much of the PBNI questionnaire as they could in 15 minutes.
Collecting test data
Before the tests started, we asked testers to complete a questionnaire on their familiarity with computers, the Internet and consultations.
We set up Camtasia Studio, a computer activity recorder, on one PC. This software produced a video of the screen, including the position of the mouse at any time, and what the tester saw.
The tester spoke aloud during the test, explaining what the user was doing, and noting any difficulties in using the interface. This was recorded on the same Camtasia video, using a microphone plugged into the computer. In addition, the comments entered were automatically stored on the server, so at the end of the session we copied the comments and deleted them from the server.
After completing the test tasks we asked the testers what they thought of the interface, using the post-test questionnaire (See Figure ). We used a similar questionnaire to collect their assessment of the control task, completing a paper questionnaire.
Finally, we invited the testers, in a group, to discuss their experiences in expressing their views through the computer map and the paper questionnaire. We took notes of points raised. The questions asked in the group included:
- Now that you have completed all the tasks, please give us your opinions.
- How did you feel about your performance on the tasks overall?
- What would you say was the best thing about this website?
- What would you say the worst thing about this website?
- What do you think of visual design of home page? Navigation? colour?
- Describe your experience of posting your comment on location. Describe what happened.
- What do you think of geographical map for locations in estate review consultation?
- Tell me about what happened when [problem]?
