Difference between revisions of "Designing a usability test"
| Line 1: | Line 1: | ||
We set out to compare the usability of this map-based interface with the conventional paper | We set out to compare the usability of this map-based interface with the conventional paper | ||
| − | questionnaire produced by the PBNI. | + | questionnaire produced by the [http://www.pbni.org.uk/ PBNI]. |
==Selecting the testers== | ==Selecting the testers== | ||
| − | There is no point getting students to test the interface since what we wanted to know was | + | There is no point getting students to test the interface since what we wanted to know was whether ex-offenders, some with low literacy levels, found the map-based interface more or less usable than current consultation techniques. |
| − | whether ex-offenders, some with low literacy levels, found the map-based interface more or less | + | |
| − | usable than current consultation techniques. We needed to recruit ex-offenders to test the | + | ===NIACRO=== |
| − | system. The Northern Ireland Association for the Care and Resettlement of Offenders | + | We needed to recruit ex-offenders to test the system. The [http://www.niacro.co.uk/ Northern Ireland Association for the Care and Resettlement of Offenders (NIACRO)] runs IT courses for ex-offenders and others. They have a small computer room in which trainees can develop IT skills that may increase their employability. A number of probationers take these courses. So [http://www.niacro.co.uk/ NIACRO] arranged for some of their trainees to test the interface when they came in for their regular session on 13 April 2006. |
| − | (NIACRO) runs IT courses for ex-offenders and others. They have a small computer room in | ||
| − | which trainees can develop IT skills that may increase their employability. A number of | ||
| − | probationers take these courses. So NIACRO arranged for some of their trainees to test the | ||
| − | interface when they came in for their regular session on 13 April 2006. | ||
==Selecting the tasks== | ==Selecting the tasks== | ||
We designed a sequence of tasks that started with simple familiarization with the map-based | We designed a sequence of tasks that started with simple familiarization with the map-based | ||
site, then got progressively more involved as the testers gained confidence, until they were | site, then got progressively more involved as the testers gained confidence, until they were | ||
| − | entering comments on the consultation topic. In brief | + | entering comments on the consultation topic. |
| − | + | ||
| − | + | In brief, the sequence of tasks were: | |
| − | + | * Explore the [http://www.pbni.org.uk/ PBNI] [http://www.pbni.org.uk/index/publications/consultation.htm e-consultation website]. | |
| − | + | * Find your probation office or reporting centre on the map. | |
| − | The control task was to complete as much of the PBNI questionnaire as they could in 15 | + | * Find the all offices or reporting centres you have been to, and add comments on each place. |
| − | minutes. | + | * Find your new office and reporting centre, and comment on how it would affect you. |
| + | |||
| + | The control task was to complete as much of the [http://www.pbni.org.uk/ PBNI] questionnaire as they could in 15 minutes. | ||
==Collecting test data== | ==Collecting test data== | ||
Before the tests started, we asked testers to complete a questionnaire on their familiarity with | Before the tests started, we asked testers to complete a questionnaire on their familiarity with | ||
| − | computers, the Internet and consultations. We set up Camtasia Studio on one PC. This software | + | computers, the Internet and consultations. |
| − | produced a video of the screen, including the position of the mouse at any time, and what the | + | |
| − | tester saw. The tester spoke aloud during the test, explaining what | + | We set up [http://www.techsmith.com/camtasia.asp?CMP=KgoogleCStmhome Camtasia Studio], a computer activity recorder, on one PC. This software produced a video of the screen, including the position of the mouse at any time, and what the tester saw. |
| − | any difficulties in using the interface. This was recorded on the same Camtasia video, using a | + | |
| − | microphone plugged into the computer. In addition, the comments entered were automatically | + | The tester spoke aloud during the test, explaining what the user was doing, and noting any difficulties in using the interface. This was recorded on the same Camtasia video, using a microphone plugged into the computer. In addition, the comments entered were automatically stored on the server, so at the end of the session we copied the comments and deleted them from the server. |
| − | stored on the server, so at the end of the session we copied the comments and deleted them from | + | |
| − | the server. After completing the test tasks we asked the testers what they thought of the | + | After completing the test tasks we asked the testers what they thought of the interface, using the post-test questionnaire. We used a similar questionnaire to collect their assessment of the control task, completing a paper questionnaire. |
| − | interface, using the post-test questionnaire. We used a similar questionnaire to collect their | + | |
| − | assessment of the control task, completing a paper questionnaire. Finally, we invited the testers, | + | Finally, we invited the testers, in a group, to discuss their experiences in expressing their views through the computer map and the paper questionnaire. We took notes of points raised. |
| − | in a group, to discuss their experiences in expressing their views through the computer map and | ||
| − | the paper questionnaire. We took notes of points raised. | ||
Revision as of 19:38, 13 January 2008
We set out to compare the usability of this map-based interface with the conventional paper questionnaire produced by the PBNI.
Selecting the testers
There is no point getting students to test the interface since what we wanted to know was whether ex-offenders, some with low literacy levels, found the map-based interface more or less usable than current consultation techniques.
NIACRO
We needed to recruit ex-offenders to test the system. The Northern Ireland Association for the Care and Resettlement of Offenders (NIACRO) runs IT courses for ex-offenders and others. They have a small computer room in which trainees can develop IT skills that may increase their employability. A number of probationers take these courses. So NIACRO arranged for some of their trainees to test the interface when they came in for their regular session on 13 April 2006.
Selecting the tasks
We designed a sequence of tasks that started with simple familiarization with the map-based site, then got progressively more involved as the testers gained confidence, until they were entering comments on the consultation topic.
In brief, the sequence of tasks were:
- Explore the PBNI e-consultation website.
- Find your probation office or reporting centre on the map.
- Find the all offices or reporting centres you have been to, and add comments on each place.
- Find your new office and reporting centre, and comment on how it would affect you.
The control task was to complete as much of the PBNI questionnaire as they could in 15 minutes.
Collecting test data
Before the tests started, we asked testers to complete a questionnaire on their familiarity with computers, the Internet and consultations.
We set up Camtasia Studio, a computer activity recorder, on one PC. This software produced a video of the screen, including the position of the mouse at any time, and what the tester saw.
The tester spoke aloud during the test, explaining what the user was doing, and noting any difficulties in using the interface. This was recorded on the same Camtasia video, using a microphone plugged into the computer. In addition, the comments entered were automatically stored on the server, so at the end of the session we copied the comments and deleted them from the server.
After completing the test tasks we asked the testers what they thought of the interface, using the post-test questionnaire. We used a similar questionnaire to collect their assessment of the control task, completing a paper questionnaire.
Finally, we invited the testers, in a group, to discuss their experiences in expressing their views through the computer map and the paper questionnaire. We took notes of points raised.
