How Children and Adults Think About Data
Assistant Professor, Department of Psychology
If the time on the wall clock is different from the time on your watch, how do you determine the “correct” time? If you roll a ball down a ramp and measure the distance it travels, will it travel the exact same distance if rolled a second time? If you see two golfers driving a golf ball many times, how can you determine which golfer is better? Children and adults regularly encounter data in a range of contexts. Although elementary school children are formally introduced to the concept of collecting and analyzing data in the science classroom, they encounter these concepts in informal contexts on a regular basis. Adults also need to reason about data in the context of their professions (for example, as scientists, policy makers, or teachers) and their daily activities.
Some of the data decisions people make are about identifying a “true” value – what is the exact time, or what is the exact measurement of the distance a ball has traveled? Engineers must determine the maximum weight that an airplane can carry. In formal contexts, scientists conduct experiments to answer such questions, taking repeated measurements to assess an average or mean value. Repeated measurements are important because researchers can be more confident if they obtain similar results on multiple trials. In addition, researchers are more confident in their results when the repeated measurements are similar rather than widely varying.
Sometimes in decision-making, the goal is to compare two sets of data to determine if they are different. For example, how can you determine which of two golfers is more skilled at driving a golf ball? If you are given a list of the lengths of six drives from each of the golfers, you have two sets of data to compare. In the simplest case, if all the drives for Golfer A are farther than all the drives for Golfer B, it’s easy to make a decision about which golfer is likely more skilled. However, as in considering measurement for assessing one single value, other data characteristics can play an important role. If the data for Golfer A are all within a 20-yard range while for Golfer B they are within a 100-yard range, you may feel more confident in ranking Golfer A’s ability than Golfer B’s. Without knowing anything further about the golfers, you may even begin to develop a story to explain the pattern: perhaps Golfer A is more disciplined and swings a golf club consistently with each drive, while Golfer B is more impulsive and therefore more erratic. If Golfer A’s drives are usually but not always farther than Golfer B’s, you may still be confident that Golfer A is a better golfer, but perhaps not quite as confident as you would be when Golfer A always drives the ball farther. If I then tell you that even though Golfer A on average drove a golf ball farther, Golfer A is actually an amateur hobbyist golfer and Golfer B is a professional golfer, you would likely quickly reassess the situation. Although the data are still very important, you would most likely conclude that the professional golfer was having an off day when you collected the data, or that the amateur golfer is an undiscovered prodigy.
Determining when a difference in two data sets is meaningful and when it is not (e.g., assessing whether there is a real difference in ability between two golfers) is an important but challenging task. How do children and adults address such questions of data interpretation? Those with training in statistics can use formal approaches that allow them to predict the likelihood of a future event or to state with confidence whether a result is likely to have occurred by chance. However, children (and many adults) do not have such formal training. In addition to lacking formal statistics training, children have less background knowledge than adults about the world in general, which makes it even more difficult for them to detect patterns in data or even to know what factors to consider in data assessment.
How can children determine when variation in the data matters? Data and background knowledge interact: the background knowledge people bring to understanding data helps in evaluating the data and, at the same time, evaluation of data helps form the theoretical background knowledge that is used to form new predictions and interpret subsequent data. This interplay leads to a bit of a chicken and- egg question: do we only build up our knowledge as we see and learn from new data, or do we only pay close attention to data when it does not fit with our past knowledge? For example, an adult drawing conclusions about weak shots from a professional golfer begins to come up with an explanation for the data (“He must have been having an off day,” “Perhaps he's injured”). Similarly, children may begin to pay attention when they see data that conflict with their expectations. If a child understands that she is growing in height as she ages, she may be confused if her home measurement of being 100 cm tall one week is followed a week later by a measurement at the school nurse's office of only 99.5 cm. Surely she cannot have lost height in the course of a week, so perhaps there is something flawed about one or both of the measurements. These inconsistencies between data and background knowledge can often inspire children to become curious about data. At the same time, there is some evidence that even when children directly observe phenomena, they do not always correctly process what they see. Thus, for example, when children believe – incorrectly – that two objects of different weights will fall to the ground at different speeds, they may simultaneously drop two objects of different weights, watch them fall at the same speed, and yet still be convinced that the heavier object fell faster (Chinn & Brewer, 2001).
Over the past several years, my colleagues and I have worked to learn more about how children and adults reason about data and outcomes from scientific experimentation.
Children’s understanding of error in scientific experimentation
In collaboration with Dr. David Klahr at Carnegie Mellon University, I looked at second and fourth grade children (average ages of about 8 and 10) and how they reasoned about data in a familiar context: reasoning about how far balls travel when they are rolled down ramps (Masnick & Klahr 2003). Experiments with ramps are common in elementary school science classrooms, and children are usually accurate in predicting which factors will affect the distance and speed with which balls travel. In our task, children worked with ramps that could be set to a low or high steepness, with a smooth or rough surface, and they could roll a golf ball or a rubber ball down the ramp. They were asked to set up two ramps in such a way that they could determine if a particular factor (such as the steepness of the ramp) made a difference in how far a ball rolled after it came down the ramp. We also asked the children to make predictions, then run the test and interpret the results, providing reasons for their conclusions. Children also answered questions about what might happen if the experiment were to be rerun and what might cause similar or different results in a replication.
To perform an ideal test, children should set up an unconfounded design; that is, they should set up the two ramps such that they are identical in all settings except for the one variable being tested. Thus, a good test of ramp steepness would consist of setting up one ramp at a high (steep) height and one at a low height, while setting up both ramps with smooth surfaces and using a golf ball to roll down each ramp. As found in earlier work (Chen & Klahr, 1999), a large portion of children set up confounded experiments (84 percent of the second graders and 60 percent of the fourth graders).
However, in providing reasons for why replications of experimental trials might vary somewhat, children demonstrated a more sophisticated understanding in that nearly all of the children provided many potential external reasons for variation in the data. They proposed factors such as error in the experimenter’s use of the stopwatch to time the run, or the ball knocking against the side of the ramp, or an accidental knock on the table that could have affected the outcome. They recognized that even though the basic setup was the same, there were minor variations that were likely to influence the outcome. In addition, they were confident that although the precise distance a ball traveled might differ from trial to trial, which ball went farther overall was far less likely to vary with a repeated test. That is, they distinguished between small variations attributed to “random” error, and larger variations attributed to more systematic factors. At the same time, despite expressing complex ideas, children often had difficulty integrating these ideas with their conclusions, and rarely mentioned such causes for variation as reasons for confidence (or lack thereof) in their conclusions.
Recently, we have been collecting data for a follow-up project in which we will look at how children and adults reason in a domain in which prior beliefs are often inaccurate: the pendulum task. This task is also familiar in elementary science classrooms, but most children (and many adults) have inaccurate beliefs about which factors affect a pendulum’s speed. In fact, the length of a pendulum is the most important factor in determining its speed (as many do believe), but the mass of the bob does not play a role (although many expect it to). Data that contradict one’s belief can often be processed differently than data that fit with one’s belief, and so we have been exploring how people reason about variations in data that fit or do not fit with prior beliefs. We are also looking at how different approaches to the testing of variables affect how much people learn from data. Thus, some participants are guided systematically by the experimenter in testing the variables in a controlled manner, and others are given the freedom to choose which combinations of variables they would like to test. Some recent research suggests that there are cases in which direct guided instruction can lead to better long-term learning outcomes than more open-ended discovery learning (Klahr & Nigam, 2004), and we are interested in learning some of the boundary conditions for that finding: when might a lot of structure help learning, and when might it hinder learning?