IV. Evaluating Students' Work

Testing

Exams as Learning

Kenneth Elzinga teaching ECON 201 with an enrollment of 500.

A well-constructed exam can teach students almost as much about a subject as it tests their knowledge of it. If you are using tests merely to assign grades, you are missing an opportunity to facilitate learning. According to McKeachie (1999), students' learning is directly related to what and how you test; therefore, you must decide in what proportion you want to emphasize simple memorization of facts versus the ability to apply the material that students have learned to other situations.
Your examinations and quizzes will be fair, reliable, and defensible when you follow a few simple guidelines. Most importantly, your exam must reflect course goals and objectives. "Of course," you say. But studies have repeatedly shown that many exams require only the regurgitation of information, even when instructors embrace and proclaim on their syllabi higher-level cognitive goals, such as critical examination of ideas, analysis of principles, problem-solving, and inquiry. You may, of course, need to test students' knowledge of facts; if so, be sure to test important, fundamental concepts, principles, and generalizations rather than trivial details, and to construct good short-answer or objective items.
In many cases, you cannot usefully measure all course objectives with in-class exams; rather, you need to check on students' learning in a variety of ways. To construct a fair evaluation system, determine which goals you can best measure by tests and which by papers, projects, problem sets, in-class discussions, oral exams or take-home exams. Consider these factors:

• Time: How much "thinking" can be accomplished in a fifteen-minute essay?
  
• The type of performance desired: Can an in-class examination measure students' creative ability?
  
• The amount of course time you wish to devote to evaluation. o
  
• The types of evaluation, given the size of the class.

When you decide that you can best assess students' progress with an exam, follow these steps:

• List the topics you have taught and what students should understand about each.
• Determine how important each topic has been in class and in assignments.
  
• Write the exam to reflect the topics covered and their relative importance.
  
• Tell your students what to emphasize, so that they can devote the most energy to the most important material.
  
• Give an ungraded quiz during the first four weeks of class so students will know what to expect and how to focus their studying for future exams.
  
• Help students become test-wise by spending some time explaining test-taking strategies in advance (see also "Preparing Students for Exams" and McKeachie, 1999, pp. 95-8).
  
  
  

I teach students the concepts and the language to frame problems which they then have to answer for themselves.

—Ingrid Soudek, Technology,
                                                                 Culture, and Communication

Writing and Scoring Essay Items

After selecting your exam goals and objectives, choose the most appropriate format: essay, short answer, identification, multiple-choice, fill-in-the-blank, and so on. When skillfully crafted and carefully evaluated, essay and short-answer items can truly measure students' ability to analyze, apply, or synthesize ideas. It is recommended that you use at least one essay question per test (McKeachie, 1999). Here are a few guidelines and examples:

• Make sure each question clearly defines the task. Avoid the broad or ambiguous: "Summarize the Vietnam War." Phrase the question specifically and give enough details to explain without giving too much away:

  In designing a one-hour Culture Fair presentation to educate Americans about Arab culture, which three cultural aspects would you emphasize and why? And which three stereotypes would you challenge and how?

• When you want students to demonstrate their reasoning ability, word your questions clearly, as these examples show:

• Indicate the scope of the required answer by defining length in time or space or value in number of points.
  
  
• Remind students that their essays must have a thesis, a main point clearly stated and coherently supported.
  
  
• Use questions that have correct answers, even if more than one answer is correct. Essay questions should measure students' knowledge and reasoning ability, not opinions or attitudes. Instead of asking, "How do you think crime could be eliminated?" say, "Describe one proposed method for controlling crime. Give four reasons why this method would be effective or ineffective." Score on the accuracy of the information and the quality of reasoning.
  
  
• Allow for "thought time" in planning exam length. Students need time to assimilate the question, recall necessary information, choose a position, and organize their answer. Allow five minutes "thought time" for an answer requiring fifteen minutes to write.
  
  
• Consider the relative value of shorter versus longer test items. Shortanswer items are more specific, better define the task, and are easier to score. You can include more items and test more topics. Longer essays, however, better evaluate students' ability to integrate material or explain complex ideas.
  
  
• Use optional questions sparingly. Because questions are not equally difficult or clear, students' choices may unintentionally make the exam harder for them. Choice also penalizes students who waste time trying to answer all questions before choosing the one answered "best."
  
  
• Before administering the test, develop a scoring key or guide that weights content, reasoning, and style. Tell students your expectations. Writing the key will help you spot confusing, ambiguous, or awkward questions; having the key as you read the first papers will save you from grading them too easily or harshly. (From Brown, 1981.)

Essay items can help you see how students think about your discipline if you ask questions that provoke them to analyze, evaluate, or synthesize. Likewise, short answer questions can be used to test more than facts if they ask students to demonstrate that they can apply that information to other situations. If you specify the type of information you seek, you will prevent students from writing down everything that they know about a subject in hopes that the right answer is in there somewhere. Additionally, it will be easier for you to judge whether the student has actually supplied the required information.

To score essay answers fairly and reliably, follow these guidelines:

• Construct the exam so that you cannot see students' names while scoring. You might have students write their ID numbers on the exam and then match names to numbers after scoring.
  
  
• Read several students' answers to each question to verify that your scoring guide is appropriate. If not, revise accordingly.
  
  
• In addition to assigning numerical or letter grades, note positive features or flaws; make sure students know the reasons for the grade.
  
  
• Read all students' answers to one question before scoring the next. You will better remember your expectations and more satisfactorily compare student performances. Moreover, grading by question rather than by exam makes you less likely to be biased by a student who responds poorly to one question.
  
  
• After finishing each question, re-grade a few you graded first to make sure your criteria have remained consistent.
  
  
• Then shuffle the papers before starting a new question. Imagine being the student who is always graded after a particularly brilliant writer or reading an average essay after a terrible one. Either way, preconceptions warp grading.
  
  
• After grading all exams, review items that posed consistent problems to ensure that they were unambiguous and keyed correctly.

Writing and Scoring Objective Items

Of course, objective items or tests can be more reliably graded than essay answers because a good objective question has only one correct answer; scoring is not subject to personal impressions or biases. For the same reasons, objective multiple-choice, matching, truefalse, or fill-in-the-blank items normally allow students no opportunity to show how they arrived at an answer. Objective items can, however, measure students' ability to analyze, evaluate and apply course content to new situations. In addition, they do measure both simple knowledge and precise discrimination. However, while grading multiple choice exams is generally faster and easier than reading a pile of essays, constructing good questions can be difficult. To create such multiple-choice questions, you must imagine various logical interpretations of a given situation and offer students those as well as the correct explanation (Grzelkowski, 1987).

Here are a few tips:

• In order to measure students' understanding, use questions requiring them to predict the outcome of a situation rather than simply label a phenomenon.
  
  
• The suggested wrong answers should represent common errors.
  
  
• Ask students to apply information to a realworld situation using concrete rather than abstract terms.
  
  
• Use a series of related items in order to measure more complex thinking.
  
  
• If you can't think of a good distractor, don't waste both your time and your students' by constructing an answer that doesn't really test discrimination. Three possible answers can be as effective as four (Costin, qtd. in McKeachie, 1999, p. 93).
  
  
• Use "all of the above" and "none of the above" rarely. They are not particularly useful in testing discrimination or knowledge.

Testing with such items enables you to reduce grading time and improve reliability while maintaining some indication of how your students reason. If you use relatively innovative items, however, discuss samples ahead of time and, on the exam, group these questions separately from fact questions.

To construct problem sets that frequently occur on math and science exams, use these guidelines:

• Make sure your problems resemble those in practice exercises.
  
  
• Make the problems interesting by giving real applications or by combining two concepts for a more engaging (and more difficult) challenge.
  
  
• Concentrate on ideas rather than on long, detailed computations.

Although grading objective questions can be quick, writing them should take some time. As with essay questions, you need to match them to course objectives and maintain the balance of topics in the course. Use the checklist below to review your objective questions:

Constructing Exams

To ensure balanced exam coverage that reflects course content, construct test items for the most important concepts after each class. With two multiplechoice questions and one essay or problem per class, you begin an item bank for exam preparation. Select a good cross section of items to create the final test, and consider a mixed format of subjective and objective items.
You will probably not be able to test as much material as you would like; students must have enough time to attempt all questions. To check on exam length, take the exam yourself, or have a colleague take it before you administer it. Read each question thoroughly, including all options, and write out each essay completely. Students will require three or four times the amount of time an expert needs. As a rough guide for objective items, assume that a simple recall item will take 30 seconds to answer; one that requires an analysis or evaluation will take one to two minutes; and one that requires calculation will vary according to the complexity of the calculation. If scheduling allows, you may want to consider untimed exams, allowing students to begin early or continue beyond the class time; you are then more certain to measure what students know rather than how fast they can process information and record answers.

Frequency of Testing

Undergraduate students need to know whether they are studying correctly and learning what they should; you need to know whether they're learning what you're teaching.

• To help your students learn best, test them (if only with a lightly weighted exam or quiz) no later than one month into the semester and return results promptly.
  
  
• Balance quizzes, exams, papers, and projects reasonably so that testing does not consume too much class time.
  
  
• To assess students' work regularly, announce tenminute quizzes at specific times and allow questions in the classes prior to quizzes. "Pop" quizzes and tests on reading or lectures that students have not been able to discuss with you may produce undue anxiety and an unpleasant learning environment unless they can have only a positive effect on a student's grades (see Wegner, 1996).

Preparing Students for Exams

You must prepare your students to take the type of exam you choose. Here are a few examples:

• If you are testing at an analytical and intellectual level, show during lectures and discussions how to reason towards solutions, analyze problematic circumstances, weigh pros and cons. Such activities benefit students who in the past have been rewarded for only encoding and recalling information and who don't know how to answer other types of questions.
  
  
• When you use multiple-choice items to assess thinking, hand out two previous exam questions in each class. After students have about two minutes to answer the questions, give the correct answers and respond to their queries. Not only do students learn how to think for the exam, they and you both see how much they understand.
  
  
• To prepare students for thought-provoking essay questions, try Ziegler's model (1989). A week before the first exam, explain your expectations for essays, including study techniques, ways to answer questions, and specific details on grading factors. Then, as students leave the exam itself, hand out three sample responses to exam questions (written at "A," "C," and "F" levels) for them to grade according to your criteria, with a short justification. As soon as possible, average the grades students assigned and discuss them. Students quickly learn to distinguish good essay answers.

I have found that collaborative learning techniques work best for me. My approach is to create a safe learning environment, where students are encouraged to take chances, experiment, and learn from each other.

—Brad Brown, Commerce

Administering Exams

When administering an exam, clarify the students' task and create a supportive atmosphere:

• Be sure that all copies of the exam are legible, paying special attention to graphs or diagrams. Bring extra copies in case pages are missing or illegible. If errors are discovered, write corrections on the board and draw all students' attention to them.
  
  
• Review instructions orally, and note exam page length.
  
  
• Suggest that students browse through the exam and ask necessary questions.
  
  
• To let students know how much time is left, write the time on the board and quietly update it each 15 minutes without interrupting.
  
  
• To avoid after-the-fact disputes about unintentionally ambiguous items or to permit variable interpretations, let students justify possibly questionable answers on the back of the test. Their responses will also help you to avoid similar problematic questions in the future.
  
  
• If you choose not to remain in the room, tell students where they can find you during the exam if they need to ask questions.

Scoring Services

If you use a multiple-choice format, your tests can be electronically scored at Computing Services in Carruthers Hall or Wilson Hall. To get details about how to do this, check with the Office of Information, Technology and Communication or look on the ITC web site: http://www.itc.virginia.edu/desktop/ unix/docs/u022.testscor.html. If you are giving several computer-scorable tests in a course, you can use a grading program to add subsequent scores and know the total points for any student at any time. You can also request that your data be mailed to your electronic mail address in ASCII format.

After the Test

Students learn from exams when feedback is timely and clear. Write specific corrections and responses on individual papers and distribute or post a key for objective answers. Offer feedback in marginal comments on essays to help students perform better on the next exam. Studies have shown that when marginal comments on earlier tests emphasized a particular skill (e.g., creativity or presenting coherent arguments), use of that skill was improved on the next exam (Johnson, 1975, qtd. in McKeachie, 1999, p.91). Return exams as quickly as possible (during the next class for objective exams), and clarify misunderstandings revealed during scoring, without wasting time by walking students through the entire exam. Set discussion guidelines to avoid unproductive debates: for example, set a time limit for questions raised in class; offer to consider an argument for a variant answer if the student submits it in writing. Exams can also facilitate learning if you allow failing students to take a make-up exam and earn up to the passing grade. Rewarding students willing to study material and gain mastery reduces test anxiety for most students.