CREDO is not credible, and never has been

Recently the CREDO institute based in Stanford has published national and state by state comparisons of charter schools that indicated charter schools may perform slightly better than traditional public schools in certain circumstances. Despite the lukewarm conclusions, privatizers and education reform advocates immediately touted these findings as proof positive that their draconian agenda was working. Following closely upon the national release was a local release evaluating charter schools in the New Orleans area. After Katrina, most of the city of New Orleans’ schools were turned into charter schools of one flavor or another. Anecdotal success stories have abounded in the wake of this charter subversion an invasion, and the New Orleans model/miracle has been touted far and wide as cure-all that should be adopted by anyone wishing to remake their education system to one that can ignore the systemic and generation impacts of poverty and instead focus pure “education choices.” Despite substantial evidence to the contrary, “school choice” has erroneously become synonymous with “school quality” and advocated as the elixir that has made dramatic improvement in New Orleans possible.

This is where CREDO comes in. Anecdotes make nice stories to tell at workshops, and great fodder for the people sections of newspapers, but Reformers were realizing they needed their own “evidence based research” reports to tout the miraculous claims they were making to lawmakers, legislators and civic organizations not already sold on the privatization movement and the dubious narrative that schools and student performance as declining (especially compared to international standards) and that “bad” teachers, uncaring teachers unions and inflexible traditional education venues were responsible for this decline.

CREDO received its first set of data from Louisiana January of 2012. I was responsible for helping pull all their data together, but I left the Louisiana Department of Education shortly after this data was sent to them. When I left, I took a lot of institutional knowledge with me – specifically much knowledge about flaws in this data, what this data contains, and what it doesn’t, and some of the limitations of using this data to evaluate different aspects of our student populations. I contacted the head of the research team, Devora Davis, not long after leaving DOE to offer my help and insight, but was rebuffed at the time.

From: devdavis@stanford.edu
Date: Mon, 27 Aug 2012 09:26:04 -0700
Subject: Re: Louisiana Education Data
To: jasonfrance@hotmail.com

Hi Jason,

Thank you for reaching out to us. Collaboration isn’t a possibility, since our agreement with the L-DOE does not permit us to share data with other researchers.

Best regards,
Dev


Dev Davis
CREDO at Stanford University

I assumed this was because CREDO was simply trying to observe the letter of the MOU they had signed with the LDOE as they claimed (however despite their assertions at the time, I discovered later that there were no exclusions to consulting with outside sources about the dataset.) When I contacted Devora, or “Dev”, as she told me she preferred to be called, this was not a cold call. Dev and I had discussed the possibility of working out a data sharing project for several years, but the logistics and legal framework was not there. Additionally quite a few resources would have to be committed for some time and at no small expense to satisfy their needs. It came as a quite a surprise then that John White, someone who has become notorious for refusing to release any data to anyone (except where there was money or free positive publicity involved) had agreed to such a large, resource intensive project before even officially signed on as the State Superintendent of Education. Nevertheless “Dev” confidently informed us that “arrangements” had been made and “John” was happy to share the state’s data with them as soon as we could get it. Devora called us weekly to see if we’d made any progress and a task force was set up of 5 or 6 folks to work on pulling all the data CREDO wanted from all of our different databases. I was consulted to link the data all together with a random identifier and to make sure the most sensitive FERPA protected elements like name and SSN were removed from the final dataset.

However what is even more interesting is this blurb I found on Devora’s boss, the director at CREDO and the “project director” of this study, is none other than Margaret E. Raymond:

Margaret E. Raymond is a research fellow at the Hoover Institution. At Hoover, Raymond serves as director of the Center for Research on Education Outcomes (CREDO), which analyzes education reform efforts around the country. CREDO’s mission is to improve the quantity and quality of evidence about the impacts of education innovations on student achievement in public K–12 education. Raymond, who has done extensive work in public policy and education reform, is currently researching the development of competitive markets and the creation of reliable data on program performance.

In partnership with the Walton Family Foundation and Pearson Learning Systems, Raymond is leading a national study of the effectiveness of public charter schools. The public-academic-private partnership helps public charter schools adopt information technologies as a means to both support their operations and generate information required by the study design. More than 250 public charter schools have joined the study to date.”

And moreover Mrs. Raymond is married to another more famous (infamous) senior fellow at the Hoover institution:

Eric Alan Hanushek (born, 1943) is a Paul and Jean Hanna Senior Fellow at the Hoover Institution of Stanford University. He is an expert on educational policy, and the economics of education. His research spans both the economics of school policy and the impact education on individuals and on economies. Major lines of research have focused on controversial areas of education policy including class size reduction, high stakes accountability, and the importance of teacher quality. He is perhaps best known for the controversial assertion that “money doesn’t matter”—that is, he says that the amount of money spent in an American school district is not related to the amount of student learning in that district—and he is often called to testify in court about school funding schemes

Hanushek is famous for his bizarre claims that class size doesn’t matter and money doesn’t matter in terms of educational gains for children that has made him a favorite of the Walton Family Foundation, individual Waltons, and Pearson Learning systems and Michael Bloomberg who have donated funds (usually the maximum allowable by law) to pro-charter state and local school board candidates and organizations like New Schools for New Orleans.

And if that weren’t enough to question the credibility of CREDO on charter school evaluations there’s also the situation that CREDO runs a charter school leadership institute in conjunction with many major charter school associations.

The PMI was initially developed as part of Building Charter School Quality, a three-year National Leadership Activities Project funded by a grant from the U.S. Department of Education’s Charter Schools Program. Under this grant, CREDO, in partnership with the National Alliance for Public Charter Schools, the National Association of Charter School Authorizers, and the Colorado League of Charter Schools, identified and developed best practices in the measurement of student and school performance, the management of performance data, and the use of performance measures to increase school and student accountability. The PMI has attracted more than 100 participants to date, and an interactive online version of the Institute was launched in June 2009.

I think this explains why CREDO was able to convince John White to turn over Louisiana’s complete dataset for a secretly pro-charter school organization to do their best to draft a propaganda piece masquerading as a credible study. CREDO is a complete sham. These are the same folks that promote TFA, John White, elimination of public education, for profit charters, and that poverty is irrelevant to academic performance.

Would you like to see how this played out in this pathetic study? The sad part is with all the gerrymandering of the data, the best they could come up with is a modest endorsement of charter schools with a low confidence of reliability.

Even if concerns over the study’s analytic methods are set side, however, Maul and McClelland point out that the study itself shows only a tiny real impact on the part of charter schools: “less than one hundredth of one percent of the variation in test performance is explainable by charter school enrollment,” they write. Specifically, students in charter schools were estimated to score approximately 0.01 standard deviations higher on reading tests and 0.005 standard deviations lower on math tests than their peers in traditional public schools.

“With a very large sample size, nearly any effect will be statistically significant,” the reviewers conclude, “but in practical terms these effects are so small as to be regarded, without hyperbole, as trivial.”

CREDO came out with a report that said charter schools do better than traditional schools, but the schools they compared to are RSD schools, state run schools, and taken collectively the worst district in Louisiana filled with all the students charter schools rejected, and even so, many of the charters did no better than the worst schools in the state, and many did even worse than the average of the worst district.

CREDO did not disclose that many of the charter schools that did better have selective admissions processes, specifically related to performance and test scores like Benjamin Franklin or socio-economically favorable geography and an admissions test like Lusher.


Benjamin Franklin High School Admissions Policy

PROCEDURE

1. Complete applications with all required documents must be submitted to the Admissions Office. • You may print it from the website (www.benfranklinhighschool.org and click the Admissions tab), or pick one up from the school. • Applications are accepted during school hours beginning in October. • Applications may be submitted in person, by mail, email, fax, or online (when available). The timely application deadline will be in January and the date specified in the Admissions Calendar. We will continue to accept applications after that date as long as space is available.

2. If the student does not have Iowa test scores, they will be scheduled to test when the application is submitted.

3. Test scores are mailed to the applicant when they are available (approximately a month after testing).

4. Acceptance letters are sent beginning in February of the year of application when we determine that the applicant is qualified to enter Benjamin Franklin High School and will continue on a rolling basis.

5. The final admission letter is sent from the principal when all required documents have been submitted including final report cards and LEAP scores where applicable.

VISITS TO THE SCHOOL

• Our Admissions Open House is held in the fall every year; check the Admissions Calendar and/or the website for the date and time. Our students, teachers, and administrators will be here to provide tours of the school, explain our academic and extracurricular programs and to answer questions.

• Families may schedule tours of the school by calling or emailing the Admissions Office. We are happy to provide guided tours but we do not provide for student “shadow” days.

ADMISSIONS REQUIREMENTS

Applying for 9th grade:

1. Residency in Orleans Parish.

2. 88 points on the admissions matrix which come from the reading, language, and math portions of the Iowa Test of Basic Skills taken in 7th or 8th grade and the 1st trimester or mid-year report card from 8th grade.

3. Promotion by the current school to 9th grade with no failing grades in academic subjects.

4. Passing score on the LEAP exam if you reside in Louisiana.

Applying for 10th grade:

1. Residency in Orleans Parish.

2. 88 points on the admissions matrix from the reading, language, and math portions of the Iowa Test of Educational Development taken in 8th or 9th grade and the grades from the 1st trimester or mid-year report card from 9th grade.

3. Promotion by the current school to 10th grade with no failing grades in academic subjects.

4. One credit each in English, math, science, social studies, and foreign language from your 9th grade school.

Applying for 11th grade:

1. Residency in Orleans Parish.

2. 108 points on our matrix from the reading, language, and math portions of the Iowa Test of Educational Development taken in the 9th or 10th grade and the student’s transcript showing all high school grades.

3. Promotion by the current school to the 11th grade with no failing grades in academic subjects.

4. Two credits each in English, math, science, social studies, and foreign language.

Applying for 12th grade:

Applications are not accepted for senior year.

Lusher Admissions

In-District Admissions
The in-district (neighborhood) process is available for those families who live within the Lusher school district (neighborhood).  To view the In-district address list, please visit our website at http://www.lusherschool.org – forms and downloads page.  In-district applicants must complete the application packet, provide the required documentation, and provide proof of residency.  

The in-district (neighborhood) process runs from February 18th to March 14th from 9 a.m. to 3 p.m. and on March 15th from 9 a.m. until noon. Lusher strongly encourages parents living in the neighborhood district to complete a community application as well.

Community Admissions

The Community process is open to all those living in Orleans Parish. All community applicants must complete an application packet, take an admissions test, attend a curriculum meeting (K-1 only), and submit other required documentation.  Community applicants, who also live within the Lusher district, may submit an in-district application in addition to the community application.  

Lusher Middle School (6-8) first admits current 5th grade Lusher students. Remaining seats are filled with candidates who have applied through the community application process.

Lusher is close to 50-60% White while RSD averages 98% poor and African American.

Of course many of the remaining schools that perform better have stricter “unenrollment” criteria which filter out the lower performing students. For instance, charters can decide that any disciplinary problem warrants expulsion, but they can offer the students a chance to leave the school voluntarily to avoid an explusion.


These factors were not considered by CREDO, nor were they even mentioned which is completely absurd. CREDO went into a big spiel about trying to find “matched pairs” of students based on identical demographics to mask this glaring deficiency. It sounds good, but one of the characteristics they did not match on is the fact that many of these students were segregated by test scores and performance. How could there not be a difference in achievement learning when charters are already pre-selecting and de-selecting based on the very metric CREDO is measuring?

Another laughable claim that CREDO makes relates to SPED achievement.

Special education students in New Orleans charter schools progress significantly more than their counterparts in New Orleans TPS in both reading and math. This amounts to 65 additional days of learning in reading and 43 more days in math for special education students in New Orleans charter schools. These results are slightly higher than were found statewide.

I would say that this study finding borders on the criminal and strongly caution parents not to pay attention to this finding. Charter schools do not take on the more significantly impaired students. Even though the CREDO folks has access to the severity of disabilities, the CREDO study relied on the most basic of Special Education indicator for their study. SPED = Y/N. This indicator also included gifted “Special Education” students in many years. Most of the disabled students charter schools do accept are the mild/moderate classification with speech and hearing impairments, not the severe profound students that may even be hospital bed bound that traditional schools must serve.

I sent multiple questions to CREDO for an detailed explanation of how they accounted for these issues, the charter schools that filter students based on high test scores, the SPED indicator, the disparity of severity in SPED enrollment, but they have refused to reply to me to date. Here is a copy of my most recent inquiry that has gone unanswered.

Date: Wed, 28 Aug 2013 01:55:35 -0500
Subject: CREDO and New Orleans Charters
From: jasonfrance@hotmail.com
To: devdavis@stanford.edu

Dev,

From the article I saw posted locally it appears you used a number of selective admission charter schools that select students based on high test scores. Were you aware of this and how did you account for this?

Did you do any studies on differences in severity of SPED diagnosis between charter schools and traditional schools. Our charters tend to turn away the more severe profound and serve more mild moderate disabled students.  Did you examine or account for this difference in student population or simply classify them all as SPED or not SPED?

Did you account for data being wildly inaccurate and incomplete for many charters and RSD schools in 2005-2006 and 2006-2007?

I’ve seen a rebuttal that whether simply being a charter school only accounted for one half of 1 % of differences on Colorado’s NEPC site.  Do you have a response?

The Louisiana Department of Education has likewise refused to release this data to any non-pro-charter front organizations to conduct true independent research (they have been fighting the release of this data in the courts for the last 18 months), but this has not prevented newspapers from pushing this propaganda paid for by the Walton’s as independent research. If I was a journalist I would be ashamed that I allowed myself to be fooled by such an easy thing to research and such a glaring conflict of interests.

CREDO is simply not credible, they are not a research institution, they are pro-charter propaganda churner and should be classified as such by anytime anything they produce is quoted in an newspaper or news program that claims to be unbiased and impartial. If you are a parent, please do not pay CREDO any more attention than you would a miscellaneous propaganda pamphlet handed out at neighborhood grocery store, or stuffed under you front door handle. You can see CREDO as a joke, a wolf in sheep’s clothing, a PR firm or a charter school pimp, but an independent research organization they are not.

About these ads

About crazycrawfish

I am a former Louisiana Department of Education employee. I was one of the principal student data folks for the better part of a decade. You can find my blog at crazycrawfish.wordpress.com
This entry was posted in charter school, education, jindal, John White, louisiana, politics and tagged , , , , . Bookmark the permalink.

12 Responses to CREDO is not credible, and never has been

  1. liznola716 says:

    Given that CREDO filtered out all meaningful data, i.e., the overwhelming number of students in RSD charters and direct-run schools, I’m astonished that our “elite” charter schools could only produce a 1/2 of 1% “edge” over traditional public schools. I’m also amazed that CREDO called their findings for NOLA so negligible as to be “trivial.” That seems to me to be the key point: stated in plain English in plain sight. That’s what any real journalist must report, so let’s see who does what with this egregious nonsense. Hey Crazy, was CREDO paid anything by “John” to do this study? That would seem significant as well. Keep up your awesome, important work. Just wish the average Joe in the street would WAKE UP!!!!

    • And independent researcher explained the significant behind the modest claims CREDO was making using days of learning and standard deviations.  They quantified their “study” without putting the findings in perspective, that essentially the differences were negigible. Considering charters have much more money, selective admisions, fewer LEP and disabled students, and crappy data this comes to me as an indictment of incompetence. 

      CREDO has not responded to my inquiry as to whether funding was involved for a special new Orleans specific study they did not do for any other region.  Reporters should have asked and published that before publishing story. Either way, the Walton’s and charter schools and operators fund CREDO, so it matters little anyways.  These guys are phoney. They should have disclosed those alliances up front if they were even going to have the slightest pretense of impartiality.  The fact that they did not renders their analysis bogus. Combined with the fact LDOE wont share that data with anyone else hammers home the final nail in the coffin.  This is a scam,one hand washing the other but both are irreversibly dirty.

      Sent from my Samsung smartphone on AT&T

  2. Old Teacher says:

    Even for statistical “noise” this is pretty quiet. They couldn’t even come up with a decent statistical artifact? I won’t even begin to wait for a confidence interval on these puny indistinguishable differences. Even CREDO can’t put enough lip stick on this pig to make it look good. I hope they are a little ashamed of themselves, they are smart enough to do a better con job than this.

    • This is the best job they could do with the reality they were given. They are not trying to be honest or accurate, merely as pro-charter as they possibly can be to serve their masters. It’s kind of sad this is the best they could come up with.

  3. Jason, when I read the report on this “special New Orleans” CREDO study, several questions popped into my head such as: How did they find matched pairs of students in New Orleans when it is well known that the elite charters pick and choose their students based on perfromance, motivation and discipline records and leave the “scrubs” in the direct run schools? I also wondered why they did not compare the results of the highly selective charters like Ben Franklin and Lusher to public magnet schools that use some of the same selection criteria? (They could still have controlled for poverty and other factors along with academic standards) Also when they looked at charters outside New Orleans, I wondered how they were able to compare type 1 charters in some of the parishes where it is known that student selection is also used . For example, how do you compare the Avoylles charter school fairly when they are attracting the higher performing students in the Parish? I just did not know who at CREDO I could call to ask those questions. It looks like you knew who to call but still cannot get the answers. Thanks for this analysis.
    Mike Deshotels

    • I’ve been contacting them for a while and it was a complete waste if time I now realize in retrospect. CREDO is merely a corporate pawn, a PR agency masquerading as a legitimate independent research institution. I plan to organize people to out them. I should have been more suspicious from the outset.

  4. Pingback: Crazy Crawfish Says CREDO New Orleans Study Is Not Credible | Diane Ravitch's blog

  5. Pingback: Louisiana Data=Berlin wall— What do they have to hide? | Cloaking Inequity

  6. Joe Nathan says:

    You and I agree that the CREDO approach is open to challenge, and I’ve done this in print. We do not agree that, as you wrote, “the New Orleans model/miracle has been touted far and wide as cure-all…”

    Please document that New Orleans has been “touted far & wide as cure-all.”

    Journalist John Merrow, who has considerable credibility, have said on balance, that what is happening in New Orleans is producing progress. He has not described it as a “cure-all.” I’ve heard a number of presentations about New Orleans and have not heard any describe it as a
    “cure-all.”

    Could you please share something in writing that says the New Orleans model cures all problems?

    • Hi Joe,

      Do you speak for the charter collective with your “We do not agree that. . . ” intro?  :)

      I object to your tepid remark that the “CREDO study is open to challenge. . .” it’s complete and utter crap, and I’m quite sure you know that.  If we were comparing magnet schools from Beverly Hills to charter schools in Ethiopia, but saying we did so by finding “virtual pairs or twins” you would be all over that.  In fact, I think I might do my own CREDO style study, in reverse, to show you how it works. . .

      I’ve seen numerous articles on RSD being touted as a great solution for poor urban districts.  Texas recently attmepted to create their own.  I got this answer to your comment from Dr Julian Vasquez Heilig.  You might read his included blog post about RSD. . .it was entertaining.

      Senator West pushed a recovery district in Texas. It got shot down in the legislature. Yes it [RSD] is being touted far and wide. I couldnt advise Senator West on whether it would work because NOLA won’t release their data. sb1718. >> >> >>http://cloakinginequity.com/2013/05/21/la-and-the-recovery-school-district-approach-sb1718-a-p-t-barnum-circus/ >> >>  >>Julian Vasquez Heilig, Ph.D. ▪ Associate Professor, Educational Policy and Planning ▪ Associate Professor, African and African Diaspora Studies (by courtesy) ▪ Faculty Associate, Center for Mexican American Studies ▪ Faculty Affiliate, Center for African and African American Studies ▪ Coordinator, M.Ed. Program in Educational Policy and Planning ▪ Associate Director, University Council for Educational Administration ▪ The University of Texas at Austin College of Education >>

      ________________________________

  7. Pingback: Is Student Privacy Hopeless? | Crazy Crawfish's Blog

  8. Pingback: The RSD and New Orleans miracle (of cheating) | Crazy Crawfish's Blog

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s