Group items tagged

The past decade has seen increased testing of students and the concomitant proliferation of computer-based systems to store, manage, analyze, and report the data that comes from these tests. The research to date on teacher use of these data has mostly been qualitative and has mostly focused on the conditions that are necessary (but not necessarily sufficient) for effective use of data by teachers. Absent from the research base in this area is objective information on how much and in what ways teachers actually use student test data, even when supposed precursors of teacher data use are in place. This paper addresses this knowledge gap by analyzing usage data generated when teachers in one mid-size urban district log onto the web-based, district-provided data deliver and analytic tool. Based on information contained in the universe of web logs from the 2008-2009 and 2009-2010 school years, I find relatively low levels of teacher interaction with pages on the web tool that contain student test information that could potentially inform practice. I also find no evidence that teacher usage of web-based student data is related student achievement, but there is reason to believe these estimates are downwardly biased.

The data summit is part of the Data Quality Campaign, which is a national effort by dozens of organizations and funded by grants and contributions from a variety of foundations including the Bill & Melinda Gates Foundation, the Michael & Susan Dell Foundation, the Lumina Foundation for Education, AT&T, and the Birth to Five Policy Alliance. The campaign, the website says, works to "encourage and support state policymakers to improve the availability and use of high-quality education data to improve student achievement." There's nothing wrong and there can be a lot right with using high-quality education data to improve achievement, of course, but data can never be the whole story. Ensuring that data is high quality, knowing how to use it - and understanding its limitations - is still not the science. A lot of the data we have is junk, but we use it to inform important decisions anyway.

Imagine for a moment that data isn't becoming a dirty word. Let's imagine that when done correctly, and with integrity, data can provide useful information about students. Jonathan Cohen from the National School Climate Center once said, "Educators are now used to data being used as a hammer rather than a flashlight." What if we took some time to turn that around and made the data a flashlight instead of a hammer? Yes, it would take a collaborative and trusting relationship between administrators and teachers. Those educators reading the data would have to read the data with an open mind, even if it was telling them something they may not want to hear. Those numbers represent the lives of our students. Using data requires many important conversations. First and foremost, when we have those conversations, we need to see the faces of the students.

"This post is about rolling out some of the left over data I have from my various endeavors this summer.  These data include data from New York State personnel master files (PMFs) linked to New York City public schools and charter schools, NYC teacher value-added scores, and various bits of data on New York City charter and district schools including school site budget/annual financial report information. Here, I use these data combined with some of my previous stuff, to take a first, cursory shot at characterizing the teaching workforce of charter school teachers in New York City. All findings use data from 2008 to 2010."

We frequently present quick analyses of data on this blog (and look at those done by others). As a close follower of the education debate, I often get the sense that people are hungry for high-quality information on a variety of different topics, but searching for these data can be daunting, which probably deters many people from trying. So, while I'm sure that many others have compiled lists of data resources relevant to education, I figured I would do the same, with a focus on more user-friendly sources. But first, I would be remiss if I didn't caution you to use these data carefully. Almost all of the resources below have instructions or FAQ's, most non-technical. Read them. Remember that improper or misleading presentation of data is one of the most counterproductive features of today's education debates, and it occurs to the detriment of all. That said, here are a few key resources for education and other related quantitative data. It is far from exhaustive, so feel free to leave comments and suggestions if you think I missed anything important.

What counts as a "fact"? New York State Supreme Court Justice Cynthia Kern's ruling on the release of the New York City Teacher Data Reports reflects a view very much at odds with the social science research community. In ruling that the Department of Education's intent to release these reports, which purport to label elementary and middle school teachers as more or less effective based on their students' performance on state tests of English Language Arts and mathematics, was neither arbitrary nor capricious, Kern held that there is no requirement that data be reliable for them to be disclosed. Rather, the standard she invoked was that the data simply need to be "factual," quoting a Court of Appeals case that "factual data … simply means objective information, in contrast to opinions, ideas or advice." But it is entirely a matter of opinion as to whether the particular statistical analyses involved in the production of the Teacher Data Reports warrant the inference that teachers are more or less effective. All statistical models involve assumptions that lie outside of the data themselves. Whether these assumptions are appropriate is a matter of opinion.

There is enormous interest in information about how learners interact with educational materials. Being able to collect data about student progress in real time is a very powerful idea. Essentially, it enables us to observe and understand the process of learning. Questions such as 'how do people learn?' or 'what exactly influences learning outcomes?' have always been critical, and adaptive learning may help to address them. But who will have access to student data? At the moment most of these and similar data belong to the companies that collect them. In fact, to some extent, public money is indirectly financing data collection for the development of a for-profit product. Ironically, when such software products are finished, they may be sold back to students - much like those students whose data helped fine-tune the starting algorithms.

"By year's end I will have entered 8,892 data points into my district's data collection systems - Gradebook and Reading 3D. This data is from homework, assessments, and report cards. Which of these 8,892 data points are the important ones? I mean, which of these data points will count towards my evaluation? And what problem are you trying to address by including student assessments into teacher evaluations? "

A decade ago, it was disconcertingly easy to find education leaders who dismissed student achievement data and systematic research as having only limited utility when it came to improving schools or school systems. Today, we have come full circle. It is hard to attend an education conference or read an education magazine without encountering broad claims for data-based decision making and research-based practice. Yet these phrases can too readily morph into convenient buzzwords that obscure rather than clarify. Indeed, I fear that both "data-based decision making" and "research-based practice" can stand in for careful thought, serve as dressed-up rationales for the same old fads, or be used to justify incoherent proposals. Because few educators today are inclined to denounce data, there has been an unfortunate tendency to embrace glib new solutions rather than ask the simple question, What exactly does it mean to use data or research to inform decisions?

This technical report contains four main sections:  1) Data. Description of the data used to implement the student growth model, including data processing rules and relevant issues that arose during processing.  2) Model. Statistical description of the model.  3) Reporting. Description of reporting metrics and computation of effectiveness scores.  4) Results. Overview of key model results aimed at providing information on model quality and characteristics. It is important to note that results presented in this report are based on 2010-11 and prior school years' data. The model will be re-estimated with 2011-12 data when they are available"

"Yesterday, Matt Di Carlo over at Shankerblog put out his April fools post. The genius of the post is in its subtlety.  Matt put together a few graphs of longitudinal NAEP data showing that Maryland had made greater than average national gains on NAEP and then asserted that these gains must therefore be a function of some policy conditions that exist in Maryland. In the Post-RTTT era, Maryland has been the scorn of "reformers" because it just won't get on board with large scale vouchers and charter expansion and has resisted follow through on test-score based teacher evaluation. Taking a poke a reformy logic, Matt asserted that perhaps the low charter share and lack of emphasis on test score based teacher evaluation… along with a dose of decent funding might be the cause of Maryland's miracle! Of course, these assertions are no more a stretch than commonly touted miracles in Texas in the 1990s, Florida or Washington DC, most of which are derived from making loose connections between NAEP trend data and selective discussion of preferred policies that may have concurrently existed.  The difference is that Matt was poking fun at the idea of making bold, decisive, causal inferences from such data. Such data raise interesting questions. What I found so fun and at the same time deeply disturbing about Matt's post is that the assertions he made in satire… were nowhere near as absurd as many of the assertions made in studies/reports, etc. I discussed here on my blog over the years. Here are but a few examples of "stuff" presented as serious/legit policy evidence, that make Matt's satirical assertions seem completely reasonable."

Last week, I reported that the city's charter school sector was on the verge of releasing a trove of data about its schools. I began my reporting after I learned about the plan in February, and a week ago, I learned that the organization in charge of the report had big plans for the report's release. The organization, the New York City Charter School Center, sent an advisory a week ago announcing Monday as the big day and inviting reporters to an 11 a.m. press conference to learn about the report, which would compile data about the schools' performance and their students. But those plans were scrapped over the weekend. On Sunday afternoon a spokeswoman for the charter school center emailed to say that the "State of the Sector" report was being delayed because all of the data had not been verified.

Readers of my blog know I'm both a data geek and a skeptic of the usefulness of Value-added data specifically as a human resource management tool for schools and districts. There's been much talk this week about the release of the New York City teacher ratings to the media, and subsequent publication of those data by various news outlets. Most of the talk about the ratings has focused on the error rates in the ratings, and reporters from each news outlet have spent a great deal of time hiding behind their supposed ultra-responsibleness of being sure to inform the public that these ratings are not absolute, that they have significant error ranges, etc.  Matt Di Carlo over at Shanker Blog has already provided a very solid explanatory piece on the error ranges and how those ranges affect classification of teachers as either good or bad. But, the imprecision - as represented by error ranges - of each teacher's effectiveness estimate is but one small piece of this puzzle. And in my view, the various other issues involved go much further in undermining the usefulness of the value added measures which have been presented by the media as necessarily accurate albeit lacking in precision.

Though I was opposed to the release of this data because of how poorly it measures teacher quality, I was hopeful that when I got my hands on all this data, I would find it useful.  Well, I got much more than I bargained for! In this post I will explain how I used the data contained in the reports to definitively prove:  1) That high-performing charter schools have 'better' incoming students than public schools, 2) That these same high-performing charter schools do not 'move' their students any better than their public counterparts, and 3) That all teachers add around the same amount of 'value,' but the small differences get inflated when converted to percentiles.

"Last month, the U.S. Census Bureau released its annual public K-12 school finance report (and accompanying datasets). The data, which are for FY 2009 (there's always a lag in finance data), show that spending increased roughly two percent from the previous year. This represents much slower growth than usual. These data are a valuable resource that has rightfully gotten a lot of attention. But there's a serious problem within them, which, while slightly technical, hasn't received any attention at all: The vast majority of public charter schools are not included in the data."

I've commented in the past about the difficulties of obtaining reconcilable data on finances of New Jersey Charter Schools. What do I mean by reconcilable? Well, when I'm looking at financial data on charter schools in particular, I like to be able to see some relationship between expenditure and revenue data reported on IRS 990 filings (Tax returns of the non-profit boards/foundations/agencies that operate the charters) and state government (department of ed) reported expenditures and/or any annual financial report documents that might be required by charter authorizers. This really is an authorizer/accountability issue. A financial reporting requirement issue.

Statistical models, such as linear or logistic regression or survival analysis, are frequently used as a means to answer scientific questions in psychosomatic research. Many who use these techniques, however, apparently fail to appreciate fully the problem of overfitting, ie, capitalizing on the idiosyncrasies of the sample at hand. Overfitted models will fail to replicate in future samples, thus creating considerable uncertainty about the scientific merit of the finding. The present article is a nontechnical discussion of the concept of overfitting and is intended to be accessible to readers with varying levels of statistical expertise. The notion of overfitting is presented in terms of asking too much from the available data. Given a certain number of observations in a data set, there is an upper limit to the complexity of the model that can be derived with any acceptable degree of uncertainty. Complexity arises as a function of the number of degrees of freedom expended (the number of predictors including complex terms such as interactions and nonlinear terms) against the same data set during any stage of the data analysis. Theoretical and empirical evidence-with a special focus on the results of computer simulation studies-is presented to demonstrate the practical consequences of overfitting with respect to scientific inference. Three common practices-automated variable selection, pretesting of candidate predictors, and dichotomization of continuous variables-are shown to pose a considerable risk for spurious findings in models. The dilemma between overfitting and exploring candidate confounders is also discussed. Alternative means of guarding against overfitting are discussed, including variable aggregation and the fixing of coefficients a priori. Techniques that account and correct for complexity, including shrinkage and penalization, also are introduced.

The Bill & Melinda Gates Foundation's "Measures of Effective Teaching" (MET) Project seeks to validate the use of a teacher's estimated "value-added"-computed from the year-on-year test score gains of her students-as a measure of teaching effectiveness. Using data from six school districts, the initial report examines correlations between student survey responses and value-added scores computed both from state tests and from higher-order tests of conceptual understanding. The study finds that the measures are related, but only modestly. The report interprets this as support for the use of value-added as the basis for teacher evaluations. This conclusion is unsupported, as the data in fact indicate that a teachers' value-added for the state test is not strongly related to her effectiveness in a broader sense. Most notably, value-added for state assessments is correlated 0.5 or less with that for the alternative assessments, meaning that many teachers whose value-added for one test is low are in fact quite effective when judged by the other. As there is every reason to think that the problems with value-added measures apparent in the MET data would be worse in a high-stakes environment, the MET results are sobering about the value of student achievement data as a significant component of teacher evaluations.

After years of work and more than $250 million in federal support, nearly all states and many districts have established longitudinal student data systems for accountability, yet many of those systems, even within the same state, still can't talk to each other, nor easily provide data to answer daily instructional questions from educators and policymakers. The Austin, Texas-based Michael and Susan Dell Foundation is hoping its new Ed-Fi data standard, released this morning, will allow educators and researchers to access information on kindergarten through 12th grade from state and local systems even before the systems have been aligned.

We've been "data driven" for at least a decade in education, with many a fortune made on assessment training for educators. I have no problem with using data to inform instruction, but I am starting to think we've gone too far in demanding that instruction be driven by data.