Data Collection Plan

What: Self-perception of achievement, such as content understanding and skills (e.g., competence; confidence; knowledge)

Data Collection: Poll, survey, in-person questioning

Questions: Do you understand X? What do you find confusing? How confident do you feel about doing X? Is this too difficult?

What: Self-perception of course experience, such as what contributed to or detracted from learning

Data Collection: Poll, survey, in-person questioning, student evaluations of the course 

Questions: Was there too much content? Anything too easy?  Which areas did you need more time? Why?

What: Self-perception of instructional experience (experience of how the instructor contributed to and detracted from learning)

Data Collection: Poll, survey, in-person questioning, student evaluations of teaching 

Questions: Was the instructor talking too fast, confusing, approachable, helpful, gives clear examples, available for office hours, culturally responsive, etc.?

What: Achievement measures

Data Collection: Exams or quizzes; assignments; projects; portfolios; clinical field work; observation of work; DFRW rates

Questions: Specific to assessing learning outcomes

What: Design rubrics and self-assessment of experience

Data Collection: Observation of interactions with students; Inference from student work and achievement; Learning management system (LMS) user indicators such as student time spent in the system

Questions: What do I think is going well? What are students struggling with? What are they confused by? When did students seem engaged? Which assignments were done well? With which assignments did students struggle? When did students seem to have enough time/not enough time? When did students work well together?

What: Perception of teaching experience and direct observation (e.g., recording of teaching)

Data Collection: Observation of interactions with students; analysis of teaching using an instructional rubric

Questions: Did I feel students asked questions? Were students congenial with me? Did I see students were taking chances? Did students demonstrate trust? Did students give compliments or positive affirmation about my teaching? What might have contributed to the positive/negative interactions?

What: Observation of evidence of student achievement

Data Collection: Observations of student work and performances

Questions: Specific to assessing learning outcomes.

What: Assessment of course design; observation of course

Data Collection: Observation of syllabus, course plan, course content and materials

Questions: Same as instructor questions. Specific to observation instrument (e.g., active learning criteria, etc.)

What: Observation of instruction

Data Collection: Informal teaching observation; observation using observation instrument; separate analysis of student survey responses

Questions: Same as instructor questions; What did students like/dislike about the instructor from survey questions?

Questioning students about their perception of understanding or skill development can take a variety of forms. Formally by using mid-semester surveys or polls, or informally by asking students comprehension questions during class to assess their understanding. For several examples of this formative data collection see the following:

The Student Assessment of Learning Gains (SALG) survey directly asks students about their perceptions of what they improved on and what helped. It can be accessed on the SALG website and tailored for each course.

When test items and rubric steps are carefully aligned to course learning outcomes, data from summative assessments can be aggregated for each learning outcome to provide an indication of overall student success in achieving them. While it is hoped that all students achieve the desired outcomes, it is likely that not all will. Patterns of achievement can be used to determine which course learning outcomes are most problematic for students. For these outcomes, it may be necessary to revisit instructional delivery and student support.

Comparing improved exam and course grades to previous courses is not necessarily straightforward. Some key issues to consider are:

• Change in content
  With the redesign of both content and assessments, your course may be more or less difficult than the previous iteration. While higher scores may indicate improved student understanding, they also may indicate less challenging content or assessments.
• Change in assessment
  Changes in assessment may make it difficult to compare scores. For example, you may opt to have students create a final group project instead of take an exam. While ideally covering similar content, your exam may have assessed “understanding” while the group project now assesses “application.”
  
  Directly comparing performance between assessment versions may not be a useful comparison. In another example, you may keep an exam, but the number of topics covered may increase or decrease. Likewise, comparing these scores may not tell you whether students from previous courses learned less or more than from your redesigned course. You will need to make individual judgements about what you can compare.

You may need to use the new assessment and improve it several times before it is calibrated and reliable for comparing courses.

Having a peer faculty member or instructional designer review student achievement can give insights about the success of student outcomes (i.e., are these outcomes expected or satisfactory) as well as why patterns in grades or responses are occurring. These are the same measures used by the instructor but may also include observation of teaching and learning.

Examining DFRW (drop, fail, resign and withdrawal) rates compared to rates from past offerings of the course, or to other sections of the course, can provide important information about course and instructor accessibility. While the goal is to increase the number of students passing the course, patterns may not always be clear. For example, there may be an increase in the withdrawal rate from a revised course if the syllabus and what is expected of students is now clearer causing some students to realize that the course is not a good fit for them. On the other hand, increased DFRW rates can signal content or assignments that are misaligned to student needs, or to instructor differences in providing adequate student support, as opposed to course design differences. Interpreting DFRW rates should always occur in the context of what occurred in the course and should be used in conjunction with other indicators, within and beyond that particular course.

Like surveys and polls for students to self-assess their achievement (see above), you can ask about specific design questions such as the difficulty of certain topics or the amount of time given for projects.

The Teaching Practices Inventory is a faculty self-assessment instrument to determine how many best practices for course design are followed by the instructor. This instrument gives an overall score for design and can be used as a guide to improve several aspects of your course.

While the OSCQR is a non-evaluative assessment, it does provide categories for reviewing courses for quality and effectiveness and provides support for improvements.

Like surveys and polls for students to assess course design or self-assess achievement (see above) you can ask about specific instructional questions such as the clarity of instruction, approachability of the instructor, or whether the instructor provided enough support.

TDOP is an instrument that can be used to code observations of your course for what students and faculty are doing, the types of technology used, how often students are interacting and asking questions, amongst a variety of other important in class actions.

COPUS is an instrument that allows observers of your course to code how students and faculty spend their time in the classroom.

• Aleamoni, L. M. (1999). Student rating myths versus research facts from 1924 to 1998. Journal of Personnel Evaluation in Education, 13(2), 153–166.
• Marsh, H. W., & Roche, L. A. (1997). Making students’ evaluations of teaching effectiveness effective: The critical issues of validity, bias, and utility. American Psychologist, 52(11), 1187–1197.