Seminar 2 reading notes

Chapter 13 - An Evaluation Framework

This chapter introduces the DECIDE framework for evaluation. The framework has some steps but it is not necessary to do them in order, it's meant to go back and forth between steps.

Determine goals (of the evaluation).

• Can be different depending on where in the design process you are

Explore the questions

• Break down questions to more specific sub-questions

Choose evaluation methods

• Depends on what data is needed to answer the questions
• Use different methods to collect data

Identify the practical issues

• Do a pilot study - a small trail run of the main study, to make sure there aren't major problems that need to be fixed before doing the study with more people. 
• Time, money etc affect what you can do
• participants: experts, users - with the right level of experience!
• facilities and equipment: how you record the data needs to be right for the context
• schedule and budget: compromises need to be made
• expertise: have the knowledge to complete the study

Decide how to deal with the ethical issues

• Privacy needs to be protected
• What happens with the data collected?
• Tell people the goals of the study, the process, what kind of data is being collected?
• Share report with participants, esp when interviews and observations: ensure that you really understood what they were saying
• Incentives can be problematic

Evaluate, analyze and present the data

• How reliable are the results? Can the method produce the same results again (separate occasion, same circumstances)
• Validity - did the evaluation method measure what it was intended to measure?

Chapter 15 - Evaluation: inspections, analytics, and models

This chapter introduces methods that don’t require users present during the evaluation. Experts role-play the users the product is designed for.

Inspections: Heuristic evaluation. Walkthroughs (focused, suited for evaluating small parts of a product).

User experts instead of uses. The experts are knowledgeable about both interaction design and the needs and typical behavior of users.

Heuristics: usability principles.

• Visibility of system status
• Match between system and the real world (system should speak the user’s language)
• User control and freedom (support undo and redo)
• Consistency and standards
• Error prevention
• Recognition rather than recall
• Flexibility and efficiency of use (allow for tailoring frequent actions)
• Aesthetic and minimalist design
• Help users recognize, diagnose, and recover from errors
• Help and documentation

Common to tailor the heuristics with other design guidelines etc. to get more specific heuristics.

Multiple experts need to do the heuristic evaluation because one person might not catch all problems. The more the better.

Walkthroughs

Cognitive walkthroughs: simulating a user’s problem-solving process at each step in the human-computer dialog.

Typical user characteristics are identified and documented, sample tasks are developed.

Identifies specific user problems at a high level of details. Good when testing something that has very complex operations.

Pluralistic walkthroughs: several people (users, developers and usability experts) assume the role of a typical user. Write down what they would do in scenario, discuss it and move on to next round of screens.

Analytics

Evaluating user traffic through a system, eg Google Analytics.

Predictive models

No users present. Formulas to derive various measures of user performance.

GOMS model: Goals (the state that the user wants to achieve), Operators (cognitive processes and physical actions that need to be performed to attain those goals), Methods (learned procedures for accomplishing goals), Selection rules (what method to choose)

KLM model: Keystroke Level Model. Average time it takes to perform common actions when using something. There is time for “Mental preparation” (time to think about/decide what to do next), but can be difficult to know when to use.

The advantage of using these evaluation methods are that you don’t need to bring in any users. But isn’t it better to use users if you can? Do you use completely different methods if you have cheap access to user, or can these methods be used with users as well, or is the knowledge of interaction design required to do, for example, a walkthrough? 

Why don’t we bring in the heuristics earlier in the design process? Can it be useful to be more aware of them earlier?