Another white paper on Usability Testing...
ABSTRACT
Usability testing is a dynamic process that can be used throughout the process of developing interactive multimedia software. The purpose of usability testing is to find problems and make recommendations to improve the utility of a product during its design and development. For developing effective interactive multimedia software, dimensions of usability testing were classified into the general categories of: learnability; performance effectiveness; flexibility; error tolerance and system integrity; and user satisfaction. In the process of usability testing, evaluation experts consider the nature of users and tasks, tradeoffs supported by the iterative design paradigm, and real world constraints to effectively evaluate and improve interactive multimedia software. Different methods address different purposes and involve a combination of user and usability testing, however, usability practitioners follow the seven general procedures of usability testing for effective multimedia development. As the knowledge about usability testing grows, evaluation experts will be able to choose more effective and efficient methods and techniques that are appropriate to their goals.
Usability Testing
Usability can be defined as "a measure of the ease with which a system can be learned or used, its safety, effectiveness and efficiency, and attitude of its users towards it" (Preece et al., 1994, p. 722). Based upon this definition, the usability of a multimedia software could be measured by how easily and effectively a specific user can use the multimedia program, given particular kinds of support, to carry out a fixed set of tasks, in a defined set of environments (Chapanis, 1991).
Usability testing determines whether a system meets a pre-defined, quantifiable level of the usability for specific types of user carrying out specific tasks. Traditionally, software products including information materials and multimedia software have been evaluated by means of marketplace reviews, magazine reviews, and beta tests, but these approaches leave too little time for major modifications and improvement of products (Reed, 1992; Skelton, 1992). As the process of observing and collecting data from users while they interact with multimedia prototypes, usability testing can be used to address and solve a system’s usability problems before it goes into production.
The aim of usability testing is not to solve problems, or to enable a quantitative assessment of usability (Patterson, 1994). It provides a means of identifying problem areas, and the extracting of information concerning problems, difficulties, weaknesses and areas for improvement. Even if usability testing should reveal difficulties or faults that cannot be corrected in the model under development, the information is still important for the designers in planning for the future release of a product (Chapanis, 1991; Dieli, 1989).
Usability testing may serve a number of different purposes: to improve an existing product; to compare two or more products; to measure a system against a standard or a set of guidelines (Lindgaard, 1994). It can also be used as a comparison test: usability of a product is compared against competitors’ products, and serves as a verification tool- a way to check user reaction to new features (Reed, 1992).
Usability testing is concerned with ‘fitness for use of a system,’ and as such it can be a powerful instructional systems development (ISD) tool for identifying problems with multimedia interface as defined by the specific user rather than the interface as designed by the instructional systems designers (Davies, 1995). With usability testing, rapid prototyping in the multimedia production process is beginning to emerge as a way to test design approaches and user interfaces, and will reduce the software development cycle while at the same time increasing effectiveness (Henson & Knezek, 1991; Northrup, 1995).
Reed (1992) indicates maxims of usability for software developers: (a) design for the software end user, not for the designers/clients; (b) test the multimedia software, not the user; (c) test usability with real users early and often; (d) don’t test everything at once; (e) measure performance of real-world tasks with software, not functionality of the program; and (f) test usability problems that software designers never imagined.
Assessment Testing
At the intermediate point of the web site development cycle, real time trials may be conducted to evaluate user performance rather than user thought processes. Assessment tests allow designers to focus on specific usability deficiencies while gathering other task-pertinent data. Although hints may be provided to a user under performance duress, there is little interaction between the user and the test administrator. This affords a greater opportunity to collect quantitative data that may bolster the results gathered from an exploratory test. The number of hints provided by the test administrator, errors committed by all users, successful task completions among all users, and the elapsed time for task completion are examples of documented critical appraisals. Given any specific goal of a web site, an error free sales transaction for instance, the assessment test may be used to measure an initial benchmark for user performance within a web site.
Validation Testing
Validation tests gauge how a web site compares to a competitor, a company historical standard, or to a project specific objective. This test measures consistency among all users against a predetermined benchmark and is therefore quantitative in nature. Because the validation test is used to accredit the site's ability to meet its designers' goals, it is conducted near the end of a web site's development cycle. The test is capable of determining which criteria are being met and reasons for those not met. For validation testing, the parameters of performance among targeted users typically exceed speed and accuracy. They can include ranking preferences both within the site itself and amongst similar competitor sites. In addition, all site components are tested as a complete package. For example, evaluating a search engine's efficiency works in conjunction with the amount of information available within the site. If there are deficiencies, the validation test affords management the opportunity to delay going live in order to fine-tune the site, prepare a response for public relations purposes, or effectively train a support team for predicted user difficulty.
A usability test is a formal evaluation process that has as its goal improvement of the usability of the product being tested. It differs from a quality assurance or quality test, which has as its goal assessing whether the product works according to specifications. It differs from a customer assurance test, a pilot test, and a beta test because the usability test ensures the collection of systematic, recorded, quantifiable data and observation of behaviors.
A usability test has these five characteristics:
Each test has specific goals and concerns that are tested
The participants represent real users (6 to 12 participants are typical)
The participants do real tasks
The participants are observed and recorded
The data is analyzed, problems diagnosed, and recommendations made
A usability test consists of these activities:
• Planning the test, developing participants profiles, identifying participants from user pool, creating test materials, writing task scenarios, determining usability criteria and measures
• Preparing the test location, pilot testing materials and procedures
• Introducing the participant to the situation, the product, and the procedure
• Running of the task-based test, where participants are asked to complete a series of tasks that address the specific goals and concerns being tested.
• Participants are asked to "think aloud" (articulate their thoughts, feelings, and actions). This data, and the recorded video images, helps target areas that re confusing, unclear, or misleading during the analysis stage.
• Debriefing the participant to get final thoughts, subjective feelings about the product, and suggestions for improvement.
• Analyzing the data, making recommendations, and documenting findings
The deliverable from a usability test is a report that details the problems encountered by the participants and recommendations for change based on known human factors, cognitive, and behavioral principles, and recognized best practices.
© 2003, Bill Tchakirides
Usability testing is a dynamic process that can be used throughout the process of developing interactive multimedia software. The purpose of usability testing is to find problems and make recommendations to improve the utility of a product during its design and development. For developing effective interactive multimedia software, dimensions of usability testing were classified into the general categories of: learnability; performance effectiveness; flexibility; error tolerance and system integrity; and user satisfaction. In the process of usability testing, evaluation experts consider the nature of users and tasks, tradeoffs supported by the iterative design paradigm, and real world constraints to effectively evaluate and improve interactive multimedia software. Different methods address different purposes and involve a combination of user and usability testing, however, usability practitioners follow the seven general procedures of usability testing for effective multimedia development. As the knowledge about usability testing grows, evaluation experts will be able to choose more effective and efficient methods and techniques that are appropriate to their goals.
Usability Testing
Usability can be defined as "a measure of the ease with which a system can be learned or used, its safety, effectiveness and efficiency, and attitude of its users towards it" (Preece et al., 1994, p. 722). Based upon this definition, the usability of a multimedia software could be measured by how easily and effectively a specific user can use the multimedia program, given particular kinds of support, to carry out a fixed set of tasks, in a defined set of environments (Chapanis, 1991).
Usability testing determines whether a system meets a pre-defined, quantifiable level of the usability for specific types of user carrying out specific tasks. Traditionally, software products including information materials and multimedia software have been evaluated by means of marketplace reviews, magazine reviews, and beta tests, but these approaches leave too little time for major modifications and improvement of products (Reed, 1992; Skelton, 1992). As the process of observing and collecting data from users while they interact with multimedia prototypes, usability testing can be used to address and solve a system’s usability problems before it goes into production.
The aim of usability testing is not to solve problems, or to enable a quantitative assessment of usability (Patterson, 1994). It provides a means of identifying problem areas, and the extracting of information concerning problems, difficulties, weaknesses and areas for improvement. Even if usability testing should reveal difficulties or faults that cannot be corrected in the model under development, the information is still important for the designers in planning for the future release of a product (Chapanis, 1991; Dieli, 1989).
Usability testing may serve a number of different purposes: to improve an existing product; to compare two or more products; to measure a system against a standard or a set of guidelines (Lindgaard, 1994). It can also be used as a comparison test: usability of a product is compared against competitors’ products, and serves as a verification tool- a way to check user reaction to new features (Reed, 1992).
Usability testing is concerned with ‘fitness for use of a system,’ and as such it can be a powerful instructional systems development (ISD) tool for identifying problems with multimedia interface as defined by the specific user rather than the interface as designed by the instructional systems designers (Davies, 1995). With usability testing, rapid prototyping in the multimedia production process is beginning to emerge as a way to test design approaches and user interfaces, and will reduce the software development cycle while at the same time increasing effectiveness (Henson & Knezek, 1991; Northrup, 1995).
Reed (1992) indicates maxims of usability for software developers: (a) design for the software end user, not for the designers/clients; (b) test the multimedia software, not the user; (c) test usability with real users early and often; (d) don’t test everything at once; (e) measure performance of real-world tasks with software, not functionality of the program; and (f) test usability problems that software designers never imagined.
Assessment Testing
At the intermediate point of the web site development cycle, real time trials may be conducted to evaluate user performance rather than user thought processes. Assessment tests allow designers to focus on specific usability deficiencies while gathering other task-pertinent data. Although hints may be provided to a user under performance duress, there is little interaction between the user and the test administrator. This affords a greater opportunity to collect quantitative data that may bolster the results gathered from an exploratory test. The number of hints provided by the test administrator, errors committed by all users, successful task completions among all users, and the elapsed time for task completion are examples of documented critical appraisals. Given any specific goal of a web site, an error free sales transaction for instance, the assessment test may be used to measure an initial benchmark for user performance within a web site.
Validation Testing
Validation tests gauge how a web site compares to a competitor, a company historical standard, or to a project specific objective. This test measures consistency among all users against a predetermined benchmark and is therefore quantitative in nature. Because the validation test is used to accredit the site's ability to meet its designers' goals, it is conducted near the end of a web site's development cycle. The test is capable of determining which criteria are being met and reasons for those not met. For validation testing, the parameters of performance among targeted users typically exceed speed and accuracy. They can include ranking preferences both within the site itself and amongst similar competitor sites. In addition, all site components are tested as a complete package. For example, evaluating a search engine's efficiency works in conjunction with the amount of information available within the site. If there are deficiencies, the validation test affords management the opportunity to delay going live in order to fine-tune the site, prepare a response for public relations purposes, or effectively train a support team for predicted user difficulty.
A usability test is a formal evaluation process that has as its goal improvement of the usability of the product being tested. It differs from a quality assurance or quality test, which has as its goal assessing whether the product works according to specifications. It differs from a customer assurance test, a pilot test, and a beta test because the usability test ensures the collection of systematic, recorded, quantifiable data and observation of behaviors.
A usability test has these five characteristics:
Each test has specific goals and concerns that are tested
The participants represent real users (6 to 12 participants are typical)
The participants do real tasks
The participants are observed and recorded
The data is analyzed, problems diagnosed, and recommendations made
A usability test consists of these activities:
• Planning the test, developing participants profiles, identifying participants from user pool, creating test materials, writing task scenarios, determining usability criteria and measures
• Preparing the test location, pilot testing materials and procedures
• Introducing the participant to the situation, the product, and the procedure
• Running of the task-based test, where participants are asked to complete a series of tasks that address the specific goals and concerns being tested.
• Participants are asked to "think aloud" (articulate their thoughts, feelings, and actions). This data, and the recorded video images, helps target areas that re confusing, unclear, or misleading during the analysis stage.
• Debriefing the participant to get final thoughts, subjective feelings about the product, and suggestions for improvement.
• Analyzing the data, making recommendations, and documenting findings
The deliverable from a usability test is a report that details the problems encountered by the participants and recommendations for change based on known human factors, cognitive, and behavioral principles, and recognized best practices.
© 2003, Bill Tchakirides
posted by btchakir at 8:50 AM
0 Comments:
Post a Comment
<< Home