University at Buffalo (UB)
Empowering Instructors | Supporting Excellence
To improve your course, you need to review what happened, determine what to change and decide how to improve it. This process requires not just collecting the right data but knowing how to make sense of it, especially when different types of information might lead to contradictory conclusions. For example, what should you do if some students loved an assignment while other students disliked it? How can you improve instruction if many students fail an exam? This page will focus on what to consider when using multiple types of data to make sense of course achievement, as well as how to use these analyses to make improvements.
Using a variety of types of data together is known as a “mixed-methods” design (Creswell, 2014). Mixed methods is an entire field of study, with many unresolved conceptual and design issues. Therefore, we provide a brief and simple overview here of what to be aware of when planning data collection and analyses for your class.
Broadly speaking it may be helpful to consider data types as either quantitative data (numeric) or qualitative data (descriptive). While these are not strict categories, they may help you think about the variety of data you might consider collecting.
These may consist of course grades, answers to close ended surveys, specific test item results, number counts, and a variety of other numeric data. Analysis of these results might consist of finding averages for your students, correlations, variance, etc. In most cases quantitative data will be represented using tables, charts, or graphs.
These are often more open ended and descriptive types of responses. For example, a discussion asking students why they found an exam difficult, a survey asking how they hope to use their knowledge in the future, or an evaluation of students’ proposed course improvements. Qualitative data are often exploratory to elicit information and help explain why something is happening. Analyzing this type of data requires reading responses, coding into categories (e.g., negative and positive responses; suggestions vs. complaints), and looking for patterns that explain what is happening (e.g., students seem happy about a project) and why (e.g., they found it would contribute to their future employability) and presenting this information to form a narrative of what occurred. Instructors may find reading such data challenging or personal; remember that all student responses can provide important information or signals about where improvements in teaching, course design or assessment can occur.
You may choose to collect different types of data at the same time (concurrent design) or in phases (sequential design). There are advantages to each of these and a variety within each type.
With a concurrent design, all data are gathered around the same time, making data collection more feasible for both students and faculty. An example might be a single survey that asks students about their prior experience with the content, feelings towards the coursework and future plans in the subject area.
With this design, data is collected in phases, and most importantly, earlier data results may affect subsequent data collection. For example, you may hold a focus group with some former students to find out which aspects of your course might benefit from improvements. You might then use focus group information to create a survey for the entire class based on the focus group’s original answers. In this way the initial data collection influences subsequent questions. You may also begin with a quantitative survey first and then have a focus group to help explain the results from the survey.
If you gather more than one type of data, you will need to determine the time or resources for each data collection (e.g., how many students will you have in a focus group? How many in the survey?). You may also care about some findings more than others, and some types of data will only be supporting the other data.
You need to determine when and how mixing your different types of data will occur.
This is the easier of the two decisions and should be determined initially in your data collection design. If you are collecting data in phases, and expect your focus group to influence your survey, then mixing will occur when you create the survey (the focus group influences the survey) and after the survey is given (survey results now say something about focus group explanations.) If you are collecting data, concurrently mixing will probably occur after data collection.
Deciding how to mix data from different sources can be difficult, especially if you need to determine meaning if results for different sources do not match. If the focus group found an exam difficult, but the class survey found it easy, how will you explain these differences? Will the survey results trump the focus group results? Choices you make here will be part of the larger argument you represent with your data.
If you are interested in better understanding how to create a mixed-methods data collection design, a good place to start is the book Research Design: Qualitative, Quantitative, and Mixed Methods Approaches by John Creswell.
Now that you have considered how data may be combined, the next step is to plan and conduct analyses and determine improvements for your course.
Make important or impactful changes first. Try to limit the number of changes to create fewer variables for future comparisons. If you make too many changes at once, it may be difficult to identify what caused new effects.