Usability Evaluation Plan TTE40

Usability Evaluation Plan TTE@40 Prepared for: TTE@40 Usability Test Mag. (FH) Bettina Thurnher This document is based on a template of the: GP&IM/OASIS/Usability and Technical Documentation. Author: Denise D. Pieratti, April 19, 1995. Usabil i ty Evaluation Plan for TTE@40 March 3, 2005 This is a sample of a usability test/evaluation plan. Information that needs to be added for your specific situation is underlined. 1. Introduction This document describes the usability evaluation plan for TTE@40. The document has been elaborated by the Institute of Software Technology and Interactive Systems (IFS) of the Vienna University of Technology and shall serve as a guideline for the structured usability evaluation examined by our partners. The purpose of a usability evaluation is to predict the expected performance of the actual customer using the current product and materials, as well as detect any serious problems prior to the release of the product. This plan includes the following sections: • Purpose of the usability evaluation • Target audience • Design of the usability evaluation • Data collection methodology • Deliverables • Resources • Schedule Our partner, Senun 40, has developed a preliminary questionnaire (see Appendix) in June 2004. This questionnaire has been adapted and can be found in the appendix as well. 1.1. What is TTE@40? The main objective of TTE@40 (Technological Toolkit for Entrepreneurs @ 40) is to develop a multimedia training resource for mid-career entrepreneurs in order to support them to start their own enterprise and to enhance their entrepreneurial performance. The target groups of our project are people in the age of forty and above, who think of starting their own enterprise or are already in the start-up process. We specially focus on unemployed, underemployed and inactive men and women. TTE@40 addresses the need to provide information and training for this target group to gain a more in-depth understanding of the qualities contributing to the success of an entrepreneur. It seeks to: • encourage a positive attitude toward innovation, • develop the mindset to stimulate ideas with commercial potential • and strengthen the business planning capacity of older, would-be entrepreneurs to create and develop sustainable enterprise ventures. P a g e 1 Usabil i ty Evaluation Plan for TTE@40 March 3, 2005 The utilization of the platform: • individual learning, independent from location • online discussions with peers and tutors in the integrated forum or per email • obtaining topic-relevant country-specific information • embedding into blended-learning workshops (learning which combines online and face-to-face-approaches): concepts are developed at present in Spain and Romania • utilization in start-up counseling 1.2. Needed Equipment and Platform The current Version of TTE@40 is Version: 0.8. By the time of project launch we will have Version 1.0. The interface is optimized for 800x600 a resolution. We recommend: Internet: Internet Explorer 6.0 Operating System: XP Professional, Service Pack 1 The platform can be accessed through: URL: http://www.tte40.net/portal/web/index.php 1.3. Design Usability Goals The evaluation will focus on determining if the needs of the user are met in an easy to understand, useful, and productive manner. Specific measurable goals for the usability evaluation are outlined in the Usability Evaluation Goals section of this document. P a g e 2 Usabil i ty Evaluation Plan for TTE@40 March 3, 2005 2. Purpose of the Usability Evaluation The usability evaluation of the TTE@40 application will evaluate the potential for errors and difficulties involved in using the application. 2.1. Evaluation Scenarios This section outlines the specific scenarios the user will have to execute during the usability evaluation. “A scenario is an encapsulated description of an individual user, using a specific set of computer facilities to achieve a specific outcome under specified circumstances over a certain time interval (this in contrast to simple static collection of screens and menus: The scenario explicitly includes a time dimension of what happens when).”[Niel93] We will execute the evaluation using three scenarios as described below: 1. Scenario The test user has to create a new account, login and select a language (English). 2. Scenario How should links be displayed? – Color, highlighting… Are the different concepts of links, bookmarks and notes clear to the user? 3. Scenario Is the information logically organized? Can the users easily locate the information they are looking for? The users will examine a task/case study and save the result in their account. 2.2. Usability Evaluation Goals Specific usability goals were determined from the above concerns. These goals allow for the creation of evaluation scenarios and tasks that will let us know if our concerns are valid and what measures can help us determine if in fact the participants are having trouble completing the tasks. This evaluation will be based on the following usability goals: • Participants will be able to begin using the application with no documentation. • Participants will be able to complete activities or locate specific information within specific time limits. A series of tasks will be designed and benchmark times established. We will time users during the usability evaluation. • Participants will be able to find related information with no expressed or visible frustration. P a g e 3 Usabil i ty Evaluation Plan for TTE@40 March 3, 2005 • Participants will have no more than two false attempts in finding specific information. We will also use a survey to determine subjective reactions based on the Technology Acceptance Model of Davis 1989: • Users feel that the pictures used on the icons are recognizable and do facilitate system use/understanding? • Users feel comfortable using only the on-line help, or if context-sensitive help is a requirement. • Users feel the on-line help provides them with all of the information necessary to use the system. • Users feel that the application facilitates his work: “perceived easy of use” and “perceived usefulness”. 3. Target Audience The selection of participants whose background and abilities are representative of the products’ intended end user is a crucial element of the evaluation process. Valid results will be obtained only if the participants selected are typical end users of the product, or are matched as closely to the criterion as possible. 3.1. Background Typical users of TTE@40 platform are users which want to start their own business and are 40 ore above. We assume that in general the users do have little IT experience. Therefore the platform has to be especially intuitive and logical structured that potential users are able to find the information relevant to them. 3.2. Subject Selection Criteria The selection of participants whose background and abilities that are representative of the products intended end user is a crucial element of a successful usability evaluation. The evaluation will be valid only if the people evaluated are typical end users of the product, or as close to a selected set of characteristics as possible. The following list shows the key characteristics of the end users that are considered as critical differentiators for successful adoption, and use, of tte40. These characteristics are the basis for participant selection for the usability evaluation. The participants will be selected to reflect the range of characteristics shown below. We will execute the test with as many “real” test users as possible. Additionally we will do the evaluation with students who have a Human Computer Interaction (HCI) background. They know what they have to look for. This allows us to evaluate the platforms´ navigation (students of HCI) and the content (“real” users). P a g e 4 Usabil i ty Evaluation Plan for TTE@40 March 3, 2005 Criteria for test user selection are: • Job Function • Computer literacy Low, medium, high • Microsoft Windows experience 0-6 months, 6-12 months, more than 12 months • Number of systems/applications used 1-2 systems, 3-4 systems, 5 or more systems • Use of electronic support tools (email, Internet access, PC or workstation, etc.) 0-2 tools, 3-5 tools, more than 5 tools • Educational level up to AA or technical college degree, baccalaureate degree, graduate degree 4. Design of the Usability Evaluation A single usability evaluation will be run in at least 5 individual participant sessions. Each individual session will consist of a set of tasks and an interview/questionnaire for the participants to complete. The individual evaluations will take place in the following order: • A performance evaluation in which each participant is asked to perform a series of real-life tasks • A questionnaire and an interview after each performance evaluation to gather additional insights from the participants about TTE@40. 4.1. The Evaluation Process The usability evaluation process is as follows: Participant greeting and background questionnaire Each participant will be personally greeted by the usability specialist and made to feel comfortable and relaxed as much as possible. The participants will be asked to fill out a short background questionnaire and the issue of confidentiality will be explained. Orientation The participants will receive a short, verbal scripted introduction and orientation to the evaluation. This material will explain the purpose and objective of the evaluation, the need for product anonymity until after the evaluation, and additional information about what is expected of them. They will be assured that the product is the center of the evaluation and not themselves or their computer literacy. P a g e 5 Usabil i ty Evaluation Plan for TTE@40 March 3, 2005 Performance evaluation The performance evaluation consists of a series of tasks that are evaluated separately and sequentially. The individual participants complete the tasks while being observed by the usability specialists. The scenario is as follows: • After the orientation, the participants will be asked to sit down at the computer. The evaluation monitor will give the participants the task scenario booklet and instruct them. • After the participants begin working through the evaluation scenario, they will be encouraged to work without guidance except for the provided material and the product itself. The usability expert may ask the participant to verbalize his or her thoughts if the participant becomes stuck or confused. These occurrences will be noted by the usability expert, and will help to determine the cause of the problem. Participant debriefing After all tasks are complete or the time expires, each participant will be debriefed by the usability expert. The debriefing will include the following: • Completion of a brief post evaluation questionnaire in which the participants share their opinions on the product’s usability, appearance of application screens, and general impressions of the product • Participant’s overall comments about his or her experience • Participant’s responses to probes from the usability expert about specific errors or problems encountered during the evaluation The debriefing session serves several functions. It allows the participants to say whatever they like, which is important if tasks are frustrating. It provides important information about each participant’s rationale for performing specific actions, and it allows the collection of subjective preference data about the application and its supporting documentation. After the debriefing session, the participants will be thanked for their efforts, and released. They will be given a small token of appreciation as they leave. 4.2. Logistics We will simulate a typical office environment during the usability evaluation. Because the application is a web application, the usability evaluation must take place in a location where a network connection is available. The office will be large enough to comfortably accommodate a desk for the participant to sit at while completing the evaluation. Additionally it is recommendable to videotape the user actions during the usability evaluation. In our case we will use scenario based testing and a structured questionnaire which provides rich enough data for the aim of our evaluation. 4.3. Requirements for the Evaluation The IFS will provide: P a g e 6 Usabil i ty Evaluation Plan for TTE@40 March 3, 2005 • Average office with desk and table, internet connection and evaluation personnel • Prepared evaluation documents • Support person for length of usability pilot evaluation and evaluation sessions 4.4. Materials Design The following materials will be designed and developed for use in the tte40 usability evaluation: • Questionnaire • Task scenario package • Evaluation participant debriefing materials 5. Data Collection Methodology Data will be collected through the use of a “thinking aloud” protocol. Measures to be collected include the following: 1. The average amount of time to complete each task 2. The percentage of participants who finished each task successfully 3. The number of cases in which the participants were not able to complete a launch due to an error from which they could not recover 4. The number of times the participant used the help line or on-line documentation for each task 5. The number of positive or critical statements about the application 6. Number of and types of errors, including: Observations and comments. The usability expert notes when participants have difficulty, when an unusual behavior occurs, or when a cause of error becomes obvious. Non-critical error. A participant makes a mistake but is able to recover during the task in the fixed time. Critical error. A participant makes a mistake and is unable to recover and complete the task on time. The participant may or may not realize a mistake has been made. 8. The number of indications of frustration or joy from the participant 9. The number of subjective opinions of the usability and aesthetics of the product expressed by the participants 6. Deliverables At the completion of the usability evaluation, a formal analysis will be performed. A final evaluation report will be provided, which will detail the significant problems and P a g e 7 Usabil i ty Evaluation Plan for TTE@40 March 3, 2005 observations detected during the usability evaluation, and recommendations to address the findings. This report will be provided by the IFS. We ask our partners to send the raw data of their usability evaluations to IFS: thurnher@qse.ifs.tuwien.ac.at at the beginning of February 2005. The IFS will summarize test data. 7. Resources The following is an estimate for the total resource requirements, in days. IFS # Activity Total Resource Time 1 Planning for the evaluation Creating evaluation materials 3 2 Arranging logistics 0.2 3 Piloting the materials and procedure 1 4 Running the evaluation 1 5 Analyzing data 3 6 Preparation of final report and peer review 4 Total Time ~ 12.2 Our partners only have to execute task 2 and 4. Therefore they only have a time effort of about 1.2 days. Partners Activity Total Resource Time 2 Arranging logistics 0,2 4 Running the evaluation 1 Total Time ~ 1.2 P a g e 8 Usabil i ty Evaluation Plan for TTE@40 March 3, 2005 8. Schedule The usability evaluation of the Institut of Software Technology and Interactive Systems will be conducted in February 2005. According to the concept each partner should execute a usability test according to the possibilities of the partners. P a g e 9 Usabil i ty Evaluation Plan for TTE@40 March 3, 2005 References NIELSEN, JAKOB (2000): Usability Engineering. San Diego: Academic Press. SHNEIDERMAN, BEN; PLAISANT, CATHERINE (2004): Designing the user interface: strategies for effective human-computer interaction. 4th ed. Boston: Addiso-Wesley. Davis, F. D. (1989). Perceived usefulness, Perceived Ease of Use and User Acceptance of Information Technology. MIS Quarterly, 13:3, 319-340. 1989. P a g e 1 0 Usabil i ty Evaluation Plan for TTE@40 March 3, 2005 Appendix Questionnaire of Senun 40 (QUESTIONAIRE OF ENTREPRENEURIAL ATTITUDES) LEONARDO’S PROGRAM QUESTIONNAIRE OVER ANTREPRENEURIAL ATTITUDES Dear Sir/Lady, It’s possible that you have been anytime along your life in a personnel or professional situation, where you should have had to take a difficult decision about your future, because your professional life has become difficult (unemployment). In this moment you will have to think in all your possibilities, and one of them can be to become an entrepreneur and create a small business. But in your mind will come many different questions, like, ¿What can I do?, ¿What company I’m going to create?, ¿From where I am going to obtain the money? ¿What kind of documents must I fill up ? and how?…. After this, you will doubt if you will become a good entrepreneur and if your attitudes are positive. Nevertheless, if your attitudes are to be an entrepreneur or not we desire you answer the following questions, as your answers will be very interesting for everybody, especially for you and for those persons that want to become entrepreneurs. 1. ¿What do you think on the idea that one must be of a special way to create its own business? 2. ¿Do you think that the entrepreneurs are born or they become? Why? P a g e 1 1 Usabil i ty Evaluation Plan for TTE@40 March 3, 2005 3. It’s said, that for women it’s more difficult to become businessmen, than for men? ¿What do you think about it? 4. ¿Why do you think that most of the people is not ventured to create their own business? 5. ¿In your opinion, what is the most difficult thing to create your own company? 6. ¿What do you think a person needs to become an entrepreneur? 7. ¿Where do you think ideas come from to set up a business? 8. ¿Do you think that business can be undertaken alone or a team of persons is needed? 9. ¿What kind of business would please you to create if you could do it? 10. ¿Where do you think good opportunities of business are found? 11. ¿Do you believe that with a good training a business can be undertaken successfully? 12. ¿Do you know what “e-learning” means and what it’s used for? 13. ¿Would you like to know your possibilities of becoming an entrepreneur if you handled ease with e-learning? P a g e 1 2 Usabil i ty Evaluation Plan for TTE@40 March 3, 2005 14. ¿Which are the first three questions that arise when you become an entrepreneur? a. b. c. Finally, imagine you are going to create your own business. Evaluate the following sentences scoring from zero (no important) to ten (the most important) according to the importance that has for you the questions that we propose: ¾ The idea of business …………………………………………………………………... ¾ The money to mount a company ………………………………………………………. ¾ Human Resources ……………………………………………………………………… ¾ The search of the market (looking for clients) ………………………………………… ¾ The infrastructure ( local, equipment, offices, furniture, ….) …………………………. ¾ The documents or previous “steps” (documents, l