by John Hood
Former President, John Locke Foundation
Smart Start, Gov. Jim Hunt’s early childhood development program, was created in 1993 to help get North Carolina preschoolers ready to learn and thus boost educational performance. It was been a successful program from the standpoint of gaining national attention, but until recently its effectiveness as a school-readiness intervention was impossible to estimate. This has not stopped some politicians from proclaiming Smart Start “successful.” Earlier this year, for example, Hunt attributed apparent gains in North Carolina reading scores on 1998 national tests to the existence of Smart Start, even though the highest grade that students from Smart-Start supported preschools could have reached by 1998 was third grade and the reading tests was of fourth-graders.1
Since early 1998, however, at least four studies have been published of Smart Start’s impact on school readiness that can provide fair-minded observers with critical information about the program’s effectiveness. Although portrayed by news reports and elsewhere as proving Smart Start’s success, the studies, when closely read, suggest a far different conclusion. Indeed, the most recent and most comprehensive study published in September by the Frank Porter Graham Child Development Center at UNC-Chapel Hill found that the vast majority of Smart Start expenditures had no statistically significant effect on participants’ readiness to learn in kindergarten.
A Closer Look
For the new study, researchers picked six counties where there were active Smart Start partnerships and recruited an experimental group of 214 children who had attended a Smart Start-supported child care center as well as a control group of 294 children who had attended child care centers not involved in Smart Start. The researchers then used four measures of readiness to learn: the Kindergarten Teacher Checklist (KTC), the Peabody Picture Vocabulary Test, the Social Skills Rating Scale, and the Social Skills Rating Scale for Problem Behaviors.2 There were a number of methodological concerns here, and the authors properly cited them. The largest concern was selection bias; children were not randomly assigned to centers with and without Smart Start involvement, thus raising the question of whether subsequent differences were due to Smart Start or to the skills and motivation of parents making these choices.
Nevertheless, taken as face value, this study deals a serious blow to the program’s perceived effectiveness. The researchers found no statistically significant differences between children who had attended Smart Start-participating centers and those who didn’t. The researchers did find a statistically significant, but very small, improvement in one of the four measures for a subgroup of Smart Start participants who attended centers where Smart Start had made expenditures in direct quality improvement. The study also found that, on three of the four measures, the percentage of very low performers decreased for the subgroup vs. the control group. Smart Start’s administrators and media reports on the study played up these last results, ignoring the main finding. That was a mistake. As the authors note, 75 percent of Smart Start expenditures fund activities not directly related to child care quality improvement.3 That means that three-fourths of Smart Start funds have been spent in ways that have been shown not to improve readiness to learn, according to the study.
Furthermore, even for the subgroup of Smart Start children receiving the most attention, gains that were statistically significant were simultaneously so small in magnitude that, based on past experience with preschool intervention programs such as Head Start, they are unlikely to last beyond first or second grade. For example, in the one measurement (the KTC) where the average score for the Smart Start subgroup was truly higher, the gain was only 0.2 points on a 1-to-5 scale (4.4 for the Smart Start Subgroup vs. 4.2 for the control). This is a 5 percentage point gain. As previous studies of Head Start and other preschool programs demonstrate, only very large differences in readiness to learn in kindergarten are likely to persist throughout a child’s academic career. Most short-term gains from Head Start, for example, disappear by second grade as the performance of students from similarly disadvantaged backgrounds tend to converge regardless of whether they attended Head Start.4 Absent any evidence to the contrary and no studies of Smart Start to date have tracked students into their first or second grades it would be reasonable to conclude that, based on past research, the few Smart Start recipients who have actually increased their readiness to learn will lose this edge over their peers as they progress through school.
Previous research points to similar conclusions. In 1998, the Graham Center conducted a study of Smart Start in Orange County. A flawed study, since it lacked a true control goup, it still found no statistically significant impact on non-poor children and a modest gain for poor ones.5 A more valid study was conducted in 1998 in Mecklenburg. It found no statistically significant impact on those who spent a single year in a Smart Start center. It did find a measurable gain for kids who stayed in such a center for three years, but once again, the gain was relatively small 2 percentage points on one measure and 7 percentage points on another.6 Finally, in early 1999 the Graham Center released a study of 200 Smart Start-supported centers that showed gains in child care quality butno impact on kindergarten readiness.7
Conclusions for Policy Makers
These four studies of Smart Start outcomes, taken together, provide a rationale for reforming the program as follows:
John Hood, President