LDOE’s statistical methods and reports are beyond useless according to Harvard professor

(Provided by Mary K. Bellisario)

 

Question: Have press releases issued by the LA Dept. of Education been accurate,especially regarding specific analysis ofresults of student AP tests, possibly ACT tests,  PARCC preparation, and the VAM– Value-added Method for teacher evaluations? 

 

Further:  Should legislators base their decisions, and vote their support, oninformation contained in state press releases about public education whichcould be inaccurate?

 

An independent, nationally-recognized data analyst, Dr. A. J. Guarino, is doubtful that data from the LA Dept. of Education is being analyzed or presented accurately.  He gives his reasons below. He calls into serious question analysisprocedures and “selective” reporting in three specific areas:  AP test results (and possibly ACT results), PARCC preparation, and the VAM method of teacher evaluation.

 

Attached above and below is an unsolicited critical analysis of information that has recently come from the LA Dept. of Education, describing Louisiana’s purported progress in public education.  The analyst, Dr. A. J. Guarino,is a professor of biostatistics at the Harvard University teaching hospital, Massachusetts General Hospital of Health Professions. He and his graduate students routinely collect and analyze media reports from around the U.S. for their accuracy.   Dr. Guarino has ties to Louisiana, where he formerly served as president of the Louisiana Education Research Association (LERA). 

 

If you have further questions, or want to verify his credentials, Dr. Guarino can be reached at:  ajguarino@mgh.harvard.edu

 

Among Dr. Guarino’s conclusions are: 

“When looked at collectively, there is a troubling trend of either the misrepresentation of statistical outcomes or the incorrect application of statistical principles.”

 

Further, Dr. Guarino concludes: 

“Based upon my professional experience, I believe the Department of Education and Superintendent White have created both of these scenarios.  My intent is to bring attention to these problems so that legislators and policy makers can make informed decisions moving forward.”

__________________________________ 

Teaching, learning, and assessment: The closed circle

A. J. Guarino*

I certify that there is no conflict of interest with any financial organizationregarding the content of this letter.

The intent of this posting is to report on findings and policies that are based uponincomplete information or misapplication of statistical principles.

        I am a professor of biostatistics at the Massachusetts General Hospital Institute of Health Professions in Boston (teaching hospital for Harvard) where I teach basic and advanced quantitative methods to graduate students in the health sciences. To document the need to be “statistically literate,” I present reports from media sources purporting major findings while failing to provide essential information, which often contradicts their “major findings.” As the former president of the Louisiana Education Research Association (LERA), I continue to follow the educational developments in Louisiana.  Over the past weeks and months, I noticed “press releases” and other media containing only selective information, which made the reported findings suspicious. After securing the full information, my analyses did not replicate those of the press-releases.

        Decisions must never be derived from findings that fail to provide complete information. At this stage of my career, my community-service is to report possible data misrepresentations that appear in the media. I would never seek nor accept compensation to provide statistical consultation.   

1.      Reporting of AP Test Scores

A recent press-release by Superintendent White noted that AP passing rates increased 24.6%, the highest in the nation, from 5,144 in 2013 to 6,407 in 2014. What is conspicuously missing is the ratio between successful tests and unsuccessful tests for each year.  The first number (5,144) represents a reported 5.3% passing rate while the second number (6,407) represents a 4.1% passing; in other words, the AP passing rate actually Decreased 22.64% from 2013 to 2014.

(5.3 – 4.1) /5.3 = 1.2/5.3 = .2264 (22.64%)

The failure to provide the essential information grossly distorts the interpretation. In order to review the actual passing rates for the latest round of tests, I had to obtain that information from a newspaper article the next day.  Here is the summary:  Of the 55,000 or so additional tests in 2014, only 1,263 (approximately 2.2%) passed. These data were never mentioned in the press release. 

Calculating the percent increase from the raw data without taking into consideration the contexts of those numbers is complicity. This is like saying more people finished the Boston Marathon this year than last year only because more people entered the race.  Given the ever increasing monies expended by the state to provide for students to take AP exams, it would be appropriate for those holding the data to inform those who need the data to make informed decisions that, last year, approximately 156,000 total AP tests were given and around 6,407 passed (i.e., 4.1%).

6407/156000 = .0410 (4.1%)

One final note on this matter: It appears that the ACT results have been reported in a similar fashion to the AP scores.  If that is the case, and the increases in the number of tests that were scored at 18 or higher are due to increased numbers of students taking the tests then those results are equally distorted.

2.      Preparation for PARCC Exams

A recent press release from the Superintendent of the State Department of Education states: Since 2010, teachers and state assessment staff in more than a dozen states and the District of Columbia have developed questions, accommodations, and policies that make for an improved blueprint for standardized testing.

At first glance, the statement appears to be quite informative. However, my graduate students were quick to notice the missing essential information, i.e.,Curriculum. It is significant to note that a “Test” is a type of assessment that evaluates academic achievement. Downing & Yudkowsky (2009) explain, “…assessment and instruction are intimately related. Teaching, learning and assessment form a closed circle, with each entity tightly bound to the other” (p. 9) .

Superintendent White broke the “closed circle” by suggesting that neither Eureka Math nor any other curriculum is favored for state testing.  In a note to a high level government official who was expressing concerns about Eureka Math, Superintendent White replied:  “We will also continue to insist, as you urge, that state tests be aligned to high expectations for our students’ skills and not to any particular curriculum (emphasis added).” 

In other words, Superintendent White is suggesting that Louisiana should give high-stakes exams to studentsirrespective of the fact that students, teachers, and stakeholders have not been given a clear curriculum on the first day of school.  How do you test what you are supposed to know at the end of the year if you haven’t communicated those expectations at the beginning of the year?  Lack of clarity for curriculum guarantees that profit-driven entities will provide their own curriculum, all of which purportedly provide the best means of preparing students to meet expectations laid out by high-stakes exams.  This guarantees a process that is costly, inefficient, and ill-defined.

The DOE discreetly recognizes this unfortunate reality by designating certain curriculum derived by outside sources as “Tier 1.”   However, no outside entity can provide to parents and teachers on the first day of school what students should know and how they will be expected to demonstrate that knowledge; this responsibility falls squarely to the DOEand to this point in time it is a responsibility that has not been met. TheDOE tacitly acknowledges its failure to generate curriculum when it continues to insist that school systems are free to use whatever curriculum they choose. 

The lack of a state generated curriculum that is available on the first day of school short circuits end-of-course exams..Results are tainted since there has been no defined path presented to articulate the criteria, much less communicate how the criteria will be met. 

3.  The Evaluation of Teachers via a Value-Added Model (VAM)

The Louisiana Department of Education has implemented the Value Added Measures (VAM) as part of a teacher’s performance evaluation. However, implementation of any educational interventions must be supported by the best available research results (evidence) or more formally known as Evidence-Based Practice (EBP). For example, the Individuals with Disabilities Education Act (IDEA) and Elementary and Secondary Education Act (ESEA) require that schools use programs, curricula, and practices based on “scientifically-based research” “to the extent practicable.” The best available research from the American Educational Research Association (AERA)  reports the following, “…weak to nonexistent relationships between state-administered value-added model (VAM) measures of teacher performance and the content or quality of teachers’ instruction.” Implications of these resultsfail to support the utility of VAM data for teacher evaluations. Concurring with the AERA findings is the American Statistical Association , the largest organization in America that represents statisticians and related fields.  The Louisiana Department of Education’s implementation of theValue Added Measures (VAM) as part of a teacher’s performance evaluation is in conflict with Evidence-Based Practice (EBP).

Conclusion

When looked at collectively, there is a troubling trend of either the misrepresentation of statistical outcomes or the incorrect application of statistical principles.  If the results aremisrepresented, you do not have all the necessary data to make an informed decision (#1 above).  If the statistical principles are incorrectly applied, you have created a fundamentally flawed process that provides no meaningful interpretations (#2 and 3 above). 

Based upon my professional experience, I believe the Department of Education and Superintendent White have created both of these scenarios. 

My intent is to bring attention to these problems so that legislators and policy makers can make informed decisions moving forward.

________________________________________________ 

Bio:  Dr. A.J. Guarino

*A. J. Guarino presently teaches biostatistics at the Massachusetts General Hospital Institute of Health Professions in Boston. In 2011, Dr. Guarino was awarded the 2011 Nancy T. Watts Award for Excellence in Teaching – the highest prize given to a faculty member at Boston’s health sciences graduate school.  He received his bachelor’s degree from UC Berkeley and his doctorate in Educational Psychology with an emphasis in statistics and psychometrics. He has published over 50 refereed research articles in a variety of fields in health, education, psychology, assessment, and statistics and presented nearly one-hundred papers at national and regional conferences. Dr. Guarino has also coauthored five graduate level statistics textbooks.  A partial list of recent scholarship is provided below.  Dr. Guarino has also served as president of the Louisiana Education Research Association (LERA). 

He can be reached at: ajguarino@mgh.harvard.edu 

Recent Scholarship

Textbooks:

Meyers, L., Gamst, G. & Guarino, A. J. (2013). Performing Data Analysis: Using IBM SPSS. Hoboken, NJ: Wiley.

Meyers, L., Gamst, G. & Guarino, A. J. (2013). Applied Multivariate Research Design and Interpretation Second Ed. Newbury Park, CA: Sage.

Meyers, L., Gamst, G. & Guarino, A. J. (2006). Applied Multivariate Research Design and Interpretation. Newbury Park, CA: Sage.

Gamst, G., Meyers, L., & Guarino, A. J. (2008). Analysis of Variance Designs. New York: Cambridge Press.

Meyers, L., Gamst, G. & Guarino, A. J. (2009). Data Analysis Using SAS Enterprise Guide. Hoboken, NJ: Wiley Publication.

Refereed Articles and Presentations

Matthews, L. T., Ghebremichael, M., Giddy, J., Hampton, J., Guarino, A. J., Ewusi, A., Carver, E., Axten, K., Geary, M., Gandhi, R. T., Bangsberg, D. R. (2011). A Risk Factor-Based Approach to Reducing the Likelihood of Lactic Acidosis and Hyperlactatemia in Patients on Antiretroviral Therapy. PLoS ONE 6(4), 1-7.

Nahas, S. J., Young, W.B., Terry, R., Kim, A., Van, D. T., Guarino, A.J., & Silberstein SD. (2010). Right-to-left shunt is common in chronic migraine. Cephalalgia 30(5), 535-42.

Chesser, S., Forbes, S. A., & Guarino, A. J. (November, 2011).  Investigation of intra-individual response of the stress hormone cortisol to varying educational environments (single vs. mixed sex groupings). Poster presentation at the annual meeting of the Society for Neuroscience, Washington DC.

Lopez, R. P., & Guarino, A. J. (2013). Psychometric Evaluation of the Surrogate Decision Making Self-Efficacy Scale. Research in Gerontological Nursing 6(1), 71-76.DOI: 10.3928/19404921-20121203-02

Certain, L., Guarino, A. J., & Greenwald, J. (2011). Effective Multilevel Teaching Techniques on Attending Rounds: a Pilot Survey and Systematic Review of the Literature. Medical Teacher, 33, 644-650.

Hastie, P., & Guarino, A.J. (2013). The Development of Skill and Knowledge during a Season of Track and Field Athletics. Research Quarterly for Exercise and Sports.

Lopez, R.P., & Guarino, A.J. (May, 2011). Confirmatory factor analysis of the Surrogate Decision Making Self-Efficacy Scale (SDM-SES). Poster presented at the annual meeting of the American Psychological Society, Washington DC.

Eaves, R.C. & Guarino, A. J. (2006). Dunn’s multiple comparison test. Encyclopedia of Measurement and Statistics, 293-296.

Advertisements

42 thoughts on “LDOE’s statistical methods and reports are beyond useless according to Harvard professor

  1. Guarino’s first point does not appear to be accurate. He misreads what Whites claim is and calculates a different number.

    Rate this first claim as FALSE
    (perhaps misleading is a better word)

    – going of this claim and not going to White’s original source.

    REPORTED WHITES CLAIM: AP passing rate increased 24.6%.

    Supporting facts:
    2013 had 5,144 passing AP’s
    2014 had 6,407 passing AP’s.
    Improvement is 1,263 (6,407 less 5,144)

    improved number of passing tests is 1,263 / 5,144 * 100 % = 24.6%

    So Whites claim is TRUE. The number of passed AP exams increased by 24.6%. This is a good improvement.

    Guarino’s misconstruing this claim by defining ‘passing rate’ (tests taken that were passed) as ‘passing rate’ (rate of taken tests that were passed)

    So the reason Guarino needs ‘missing information’ is because he is calculating a different number.

    (Expletive deleted) Guarino doesn’t give all of his numbers. We have to back calculate.

    Taken Passed Passing rate
    2013 Not reported 5,144 5.3%
    2014 156,000 6,407 4.1%
    Delta 55,000 1,263 2.2%

    Numbers dont add up. There would be 101,000 tests taken in 2013 and the passing rate would be 5.4%.

    (Redacted). TP article by Jessica Williams from July 31,2014 states we have only 28,000 students in AP classes in 2014 and 23,000 in AP in 2013. So his number of tests is out of wack! She only reports a 4% drop in pass rate (defined as getting a 3 so that college credit is achieved) she reports an overall pass rate statewide of 34.1% in 2013 and 30.3% in 2014 !

    So his pass rate calculations are OFF by a frigging factor of ten from her report…

    Geez.

    Lets us the TP numbers. her numbers seem more in line with number of reported passing rates.

    Taken Passed Passing rate
    2013 15,085(calc) 5,144 34.1%
    2014 21,145(calc) 6,407 30.3%
    Delta 6,060(calc) 1,263 20.8%

    Looks like we are paying either $89 or $54/ low income test taken.

    ———————————————————

    “1. Reporting of AP Test Scores
    A recent press-release by Superintendent White noted that AP passing rates increased 24.6%, the highest in the nation, from 5,144 in 2013 to 6,407 in 2014. What is conspicuously missing is the ratio between successful tests and unsuccessful tests for each year. The first number (5,144) represents a reported 5.3% passing rate while the second number (6,407) represents a 4.1% passing; in other words, the AP passing rate actually Decreased 22.64% from 2013 to 2014.
    (5.3 – 4.1) /5.3 = 1.2/5.3 = .2264 (22.64%)
    The failure to provide the essential information grossly distorts the interpretation. In order to review the actual passing rates for the latest round of tests, I had to obtain that information from a newspaper article the next day. Here is the summary: Of the 55,000 or so additional tests in 2014, only 1,263 (approximately 2.2%) passed. These data were never mentioned in the press release.
    Calculating the percent increase from the raw data without taking into consideration the contexts of those numbers is complicity. This is like saying more people finished the Boston Marathon this year than last year only because more people entered the race. Given the ever increasing monies expended by the state to provide for students to take AP exams, it would be appropriate for those holding the data to inform those who need the data to make informed decisions that, last year, approximately 156,000 total AP tests were given and around 6,407 passed (i.e., 4.1%).
    6407/156000 = .0410 (4.1%)
    One final note on this matter: It appears that the ACT results have been reported in a similar fashion to the AP scores. If that is the case, and the increases in the number of tests that were scored at 18 or higher are due to increased numbers of students taking the tests then those results are equally distorted.”

    1. No. The passing rate is the percentage of students who passed based on tests admimistered. What you are calculating is the percent of total passing scores compared to previous year with a much smaller denominator. The passing RATE actually declined because many kids were given test who had no chance of passing. Test vendors made money though. More kids got passing scores but they tested many more kids to get them. That doesn’t mean they educated anyone better. That doesn’t show scholastic improvement. I took AP english in HS but could not afford test and never took. I could have passed some math and English AP exams 25 years ago but i didn’t take the tests. That doesn’t mean i was educated any more or less, merely that many kids who could have passed did not because they couldn’t take the tests. You can argue if this is a good or bad change based on gross increase in numbers but that doesn’t mean kids were any more or less educated, but that is how it was portrayed.

    1. LDOE likes to take their reports offline so it’s hard to verify their claims or counter their spin with verifiable data unless you manually archive everything they produce. Ain’t nobody got time for that!

  2. Got a Times of Shreveport article by Kevin McGill October 7, 2014. This cites the 4.1% number. But that number is for the ‘percentage of students earning college credit with AP courses’

    1. You just need tests administered for both years and tests passed. It’s a waste of taxpayer funds. Many charter schools give to all their kids in hope tgey have a few kids who pass (without taking AP) classes (because tests are free to them) because those give them points on SPS score.

  3. I’m not a big fan of AP tests (ie College in High School). Last article says in 2012 we were last in the nation in Kids taking AP courses. and now we are 38th in the nation. (Must be a per capita basis.) We are at 14 % participation in AP classes. national average is 22%.

    My daughter took a number of AP classes and took the exams back in 2008. Forget how many college credits she obtained, but it was a good number. but I would rather the students be taking college classes IN COLLEGE.

    1. Exactly. I tested out of math and English because of my ACT scores but i suffered in Math for it and ultimately had to switch majors. I knew ACT level math but not the College level foundation i needed.

  4. Article cites black students taking the test. 2,645 in 2013 increasing to 3,598 in 2014. passing numbers were 541 in 2013 and 707 in 2014.

    Black only
    Taken Passed Passing rate
    2013 2,645 541 34.1%
    2014 3,598 707 30.3%
    Delta 953 166 17.4%

    all Louisiana
    Taken Passed Passing rate
    2013 15,085(calc) 5,144 34.1%
    2014 21,145(calc) 6,407 30.3%
    Delta 6,060(calc) 1,263 20.8%

    similar to the overall pass rates.

    The article is extremely mistaken on its first point, so I don’t think I will review any of the other claims he makes.

    participation in AP is up.
    AP test taking is up.
    total AP’s with a 3 or better is up.
    The incremental rate of the additional test takers is 10% lower
    And presumably Louisiana is out either the $54 or $84 per additional test taken…

    1. I would prefer to know the 4s and 5s. Kids with 3s should probably still take the college courses. That is money that doesn’t pay for other programs. You can’t tell me we can’t do a better job of identifying who should be taking test. From what i saw from original report, many charter schools had zero kids pass.

  5. Tests taken from LEA.
    2012 9,644 4,112 passed 42.6%
    2013 15,070 5,144passed 34.1%
    2014 21,180 6,410 passed 30.3%

    What was Guarino smoking?

      1. Ahem… ‘a reported 5.3% passing rate’ and ‘a 4.1% passing’. He’s just off by a factor of TEN on his numbers.

        Just a wee error.

      2. In other words, if he is going to throw ‘statistical’ brick at someone. He should make sure he gets his (redacted) numbers right!

        1. Lol. I think he is conflating ACT tests with AP tests. His piece says 100 – 150k tests were administered. Could that be correct? If it is that was never revealed. I will email him the source of his test administrations. If he is accurate this is a huge boondoggle. I find it hard to believe he made such an elementary mistake. You could email him too if you have questions.

          1. LOL. Indeed, he made a major, major math error. It is not even close to correct. See one of my previous posts. I found the numbers for total test takers and used the newspaper breakdown for just blacks..

            The other two point he made don’t have numbers, so I can’t review their accuracy. so I actually stopped at the end. I thought the other points would have numbers to review also.

            The other two points seem to be more opinion than statistics.

  6. Schools get a boost in their grad index (new name — strength of diploma) if a kid even takes an AP test. More of a boost if they “pass” the AP test. They are changing that to “takes the class and takes the test.”

    Yes, the number of students passing increased 24.6%, and indeed the passing rates declined as Jason describes. 100 to 150 thousand tests administered sounds high. That info is on the LDoE website and easily checked.

  7. 21,180 AP tests administered in LA in 2013-14.
    45,305 ACT tests taken by the 2013 graduating class, but not necessarily in 2012-13.

  8. Excellent point Mickey. There are nowhere near enough ACT tests taken to account for the number. Perhaps he used the wrong State?

    Year State National
    2009 34,548 1,480,469
    2010 35,601 1,568,835
    2011 35,870 1,623,112
    2012 36,736 1,666,017
    2013 45,305 1,799,243

  9. Follow the money… “Those exam fees, however, continue to roll in. The nonprofit College Board, which also runs the SAT, reported net assets of $609 million at the end of fiscal year 2012, up from $491 million two years earlier.

    For decades, the AP division had been a drain on the organization, losing money because the tests are so pricey to grade. But surging volume has changed that; revenues from AP tests now exceed expenses by $20 million to $30 million a year, Packer said. The College Board spends the excess on teacher training, scholarships and test redesigns, he said.”

  10. Hmmm. Was this the article he was complaining about?

    They have a lot of cheek making the ‘first in the nation’ line for increase in ap tests passed.

    If you are last in the Nation, a surge in test taking will give you a big delta improvement. But looking at total passing rate still puts us way back in the pack.

    A lot of wasted money for weak students who need to be taking the college class.

    —————————

    “COLLEGE BOARD REPORT RANKS LOUISIANA FIRST IN THE NATION IN ADVANCED PLACEMENT IMPROVEMENT FOR THE CLASS OF 2013

    Feb 11, 2014
    Growth Among all Louisiana High School Students is Tops in the Nation

    BATON ROUGE, La. – The Department of Education today announced Louisiana high school seniors are outpacing the nation in the increase of graduates scoring a 3 or higher on AP exams and qualifying for college credit, according to the College Board’s 10th Annual AP Report to the Nation. The report shows the number of seniors scoring a qualifying score of 3 or higher increased from 1,531 in 2012 to 1,911 in 2013, representing a 25 percent increase, ranking Louisiana first in the nation. Nationwide, the increase from 2012 to 2013 was six percent. Additionally, Louisiana ranks third in the nation for the increase in the percentage of graduates taking an AP exam during high school, increasing 4.1 percentage points from 11.1 to 15.2 percent, representing an increase of more than 1,500 high school seniors taking an AP exam from 3,931 in 2012 to 5,516 in 2013. Likewise, the number of AP exams taken by high school seniors increased by 42 percent from 2012 to 2013, ranking Louisiana first in the nation.

    For all Louisiana high school students in grades 9-12, the number of college credits earned has increased by more than 1,000 from 2012 to 2013 – the greatest individual increase in state history. The number of students earning a qualifying score of 3 or higher rose from 4,112 in 2012 to 5,144 in 2013, representing a 25 percent increase. Likewise, the number of students taking an AP exam increased from 6,645 in 2012 to 10,553 in 2013, an increase of nearly 4,000 students or 59 percent. The number of AP exams those students took also increased, from 9,644 in 2012 to 15,070 in 2013, for an increase of 56 percent. Enrollment in AP courses has also grown. In, 2013, Louisiana students took approximately 6,000 more AP courses. There were 23,435 students enrolled in AP courses in 2013, compared to 17,496 in 2012, representing a 33 percent increase. With more students than ever before participating in AP, Louisiana is on track to being a leader in expanding college access to all students.

    “As more seniors in Louisiana leave high school with college credits already under their belt it is further proof that the hard work being done in our classrooms is paying off for our students,” said State Superintendent John White. “And, it’s not just the most recent cohort of high school graduates. We are seeing proven success and enrollment increases across all grade levels in Advanced Placement.”

    Advanced Placement courses allow high school students to earn college credit while working towards their high school diplomas. Students enrolled in the college-level courses receive high school credit for the subject in which they take the AP class, and college credit upon earning a qualifying score on a standardized AP exam.

    AP tests are scored on a 1 to 5 scale as follows:

    5 – Extremely well qualified
    4 – Well qualified
    3 – Qualified
    2 – Possibly qualified
    1 – No recommendation

    Increasing AP participation is a critical component of Louisiana Believes, the state’s comprehensive plan to put every student on track to a college degree or a professional career. With AP, students can get a feel for the rigors of college-level studies while they still have the support of a high school environment. When students take AP courses, they demonstrate to college admission officers they have sought out an educational experience that will prepare them for success in college and beyond. Research shows a strong link between taking AP courses and success in college, finding students taking AP courses are more likely to graduate in four years and have higher college GPAs.

    “Louisiana is poised to continue improving upon this success,” said Supt. White. “We are dedicated to providing not only increased opportunities for our students but the preparation and support educators need to offer these opportunities to students.”

    Louisiana is utilizing multiple strategies as part of its plan to increase AP participation including the Advanced Placement Exam Fee Program and the AP Summer Institute Fee Reimbursement Program. Through the AP Exam Fee Program, the Department will subsidize the $55 for each AP exam taken by students enrolled in both public and BESE-approved nonpublic school who meet the low-income eligibility criteria. In addition, the Department will reimburse the full $89 for each AP Exam taken by students enrolled in a public school which is implementing or expanding an AP program. The fee reimbursements are for students who were enrolled in a rigorous course preparing them for the AP exam taken during the May 2014 testing cycle, and were between the ages of 5 and 19 on exam day.

    Also, over the last two years, the Department has trained approximately 1,000 educators to teach rigorous college-preparatory courses. Through the 2014 AP Summer Institute Fee Reimbursement Program, the Department will offer the professional development opportunity to an additional 600 educators seeking to ensure a smooth transition to the redesigned courses that feature 21st century skills, inquiry-based approaches, and strong alignment to best practices in higher education. The deadline to apply for both programs is March 7. Applications can be found on the Department’s website by clicking here.

    The expanding of access to AP and student participation is leading to cost savings in college for students and families. In May 2013, Louisiana public and private high school students took a total of 8,617 AP exams that resulted in scores of 3 or higher. Based on students’ opportunity to earn at least 3 college credits for each AP Exam score of 3 or higher, this represents an estimated 25,851 college credits. At an average rate of $218.20 per credit hour, the total potential cost savings for Louisiana students and families was $5,640,688.

    To read the College Board’s 10th Annual AP Report to the Nation, please click here.

    To view 2012-2013 AP Results by District and School, please click here.

    # # # # # #

    1. Correct. That is the point. I think most of the increases were 3’s and not a result of any change or quality in education, just giving tests to more students who would have passed anyways in any year regardless and who should be taking the college courses anyways. The cost per additional passage is probably enormous, and there is also a loss of instructional time. The net effect is likely all negative. Too much money for too few kids who probably need the classes they technically qualify to get out of now and a distraction from real efforts and reforms that might actually mean something.

  11. There is no doubt in my mind that LDOE knows they are reporting data that is not accurate. They work very hard to hide as much as they can and to slant what they do release. Those who work in that area at the agency are not stupid, just devious. And are rewarded for their loyalty – follow the promotions.

    I think they believe that the end justifies the means. And the end for them is their own profit – monetarily and to be able to say “we told you we could improve outcomes for students.”

    In reality, they know nothing about what true education is. If they are the best and brightest, why aren’t they in jobs related to their college majors?

  12. I too feel dizzy from the LDOE spin here. However, here is the link to the LDOE press release on the matter that is still on the website. It does have the necessary data to support White’s angle on this. However, if you read it closely, you cannot make a clear conclusion about how many different students scored 3 or above on the AP. It shows that more students took the test, but we do not know how much of the increase came from students making 2 or more passing grades.

    http://www.louisianabelieves.com/newsroom/news-releases/2014/07/31/louisiana-students-achieve-top-advanced-placement-gains-in-state-history

    1. I’m wondering if we are missing something big. We all know the argument White is making is flawed. Most folks with any head for numbers would understand that from reading the release – or any of his releases for that matter. We also have anecdotal evidence he completely alters or outright lies evidence and numbers. Does anyone know where to find the actual amount the state spent on these tests? From that number we can ballpark how many tests were likely administered, even if the state got some sort of bulk discount. We can also then calculate the exact cost from one year to the next for those extra thousand or so passing scores. My guess is the cost will be over a thousand dollars per passing score. If the contracts are ginormously different we also know he lied to us about the denominator.

      1. I have already posted how many tests were taken from 2012 to present.

        What you need to know, however, is how many of these tests were paid for by the State or Feds. and the Pass rate for the State paid for tests.

        It does not appear that White misstated the numbers. he did however put a ridiculous spin on the numbers.

        Picture a footrace.

        The runner in 50th place just put on a burst of speed and passed a number of the slower runners and is now in 38th place.

        I guess that is a cause for some excitement. But it should be put in persoective of where the runner is on the track.

          1. I sent an email to Dr Guarino. Hope to hear back if his pipe was full or empty when he wrote. Not sure how those test numbers could be so high and such low passage rates. His conclusions and concerns are justified but his numbers may be way off. I hope to be enlightened. Thanks for pointing out the issue so we could investigate.

  13. I received a reply from the good Dr. Guarino. It appears he is unwilling or unable to confirm his denominators, which is a little ironic. However he stands by his results and analysis which does appear to conceptually correct. He is incorrect in believing we are not in need of additional backup from Harvard professors. Not sure why he felt obliged to write his letter sn then “not pursue this any further.”

    1. Wow. At least he got his calculations correct in his reply! 🙂

      What he claimed initially (with bad numbers) and reclaimed here (with good numbers) is A valid complaint. The ‘quality’ of the AP classes is decreasing.

      However, he continues attacking a claim that LDOE did not make.

      He uses the example of the Boston Marathon. So lets run with that… 🙂

      LDOE made the claim that they had a substantial increase in people running in their race (AP exams). LDOE is very excited about this increase and are seeking to coninue that increase. Validly so. More students are taking a very challenging class,

      LDOE also made the claim that they had a substantial increase in people finishing their race with a good finish time (3 or better). They made this claim correctly also. LDOE is also excited about this increase. More students are succeeding in the challenging class.

      For him to say that LDOE did their calculations incorreclty is wrong. They were appropriate on the above two claims.

      Even their claim that Louisiana is NUMBER ONE IN TOTAL INCREASE IN PASSING STUDENTS (my emphasis for this silly claim) is factually correct. it is just a silly claim to make. Our results are so far behind the pack that there is a whole lot more room for improvement. now from the standpoint of cheerleading on our teachers and encoiraging them to try harder – it is a good point. If the school system keeps working this hard, the eventually the pack leaders will be looking over their shoulders and starting to get worried. Right now, they can still mock us.

      Now his point restated is that we have a whole lot more walkers in our Boston Marathon than we used to. That certainly is a very true point.

      SUMMARY: three true claims
      ..More people are taking AP
      ..More people are passing the AP
      ..More people are failing the AP

      Now I wonder if we can get the accurate numbers to see if we are getting ‘value for money’?

  14. Dear Jason France (aka crazycrawfish),

               

                Please note that the thesis of my letter was that the AP passing rates are declining. Additionally, the calculation that yielded the passing rate of +24.6% is conceptually incorrect. The figures you provided substantiate the decline.

     

    Date                            Attempted                                Passed                         Percent

    2012                            9,644                                       4,112                           42.6%
    2013                            15,070                                     5,144                           34.1%
    2014                            21,180                                     6,410                           30.3%

                Just at first blush, the percents are attenuating. Please note the passing rates decreased 19.95% [(42.6 – 34.1) / 42.6] from to 2012 to 2013 and decreased again 11.14% from 2013 to2014. The decrease in the passing rate from 2012 to 2014 is 28.87% (which is actually greater than stated earlier)

                As previously stated, calculating the change by simply using the raw numbers (i.e., [(6410 – 5144) / 5144 = +24.6%1] is conceptually incorrect. To repeat, “This is like saying more people finished the Boston Marathon this year than last year only because more people entered the race.”

                I have no interest to pursue this matter further. I am aware of many very bright math professors in your state who will arrive at the same conclusion so my analyses are not necessary.  Hopefully this attention will force greater transparency from the board.

     

    Respectfully,

     

    A.J. Guarino, PhD

    Professor of Biostatistics

    Center for Interprofessional Studies and Innovation

    MGH Institute for Health Professions,

    Boston, MA  02129

    (617) 643-7479
    ajguarino@mgh.harvard.edu

     

     

    From: Jason France [mailto:jasonfrance@hotmail.com] 
    Sent: Wednesday, December 03, 2014 12:52 PM
    To: Guarino, Anthony J.
    Cc: Gene Bellisario; Mercedes Schneider; Lee Barrios; mikedeshot@aol.com
    Subject: Recent Letter about Louisiana Statistics

     

    Dr Guarino,

    I recently posted an open letter you made in regards to how Louisiana and LDOE presents its statistics on my blog.  I have been getting questions about some of your numbers related to testing denominators for AP tests.  Your numbers appear to be 100 to 150k, but the published numbers from the state and local stories are much lower, in the 10 and 20 thousands or so.  Can you clarify how you came with your passage rates and denominators for me and my readers?   

    https://crazycrawfish.wordpress.com/2014/12/02/ldoes-statistical-methods-and-reports-are-beyond-useless-according-to-harvard-professor/

    If the “official” denominators of total tests are vastly under-reported that would be a big deal and i could draw a lot of attention to that if you have documentation for that.

    One of my readers provided these numbers and commentary:

    Tests taken from LEA.
    2012 9,644 4,112 passed 42.6%
    2013 15,070 5,144passed 34.1%
    2014 21,180 6,410 passed 30.3%

    What was Guarino smoking?

    I assume you were not smoking anything and would be eager for a response that would clarify your study.  Any other figures or detailed reports would also be appreciated.  I have a lot of math professors that read my blog and they would be eager to investigate and verify your findings.

    Thanks!

    Jason France
    aka crazycrawfish

    The information in this e-mail is intended only for the person to whom it is
    addressed. If you believe this e-mail was sent to you in error and the e-mail
    contains patient information, please contact the Partners Compliance HelpLine at
    http://www.partners.org/complianceline . If the e-mail was sent to you in error
    but does not contain patient information, please contact the sender and properly
    dispose of the e-mail.

  15. I talked with a few more folks and received a subsequent response from Dr. Guarino that confirmed the our conclusions. Namely, that the number of test takers was extrapolated from the disclosed “passing rates.” That was an incorrect term to use and renders the articles invalid on the face. Dr. Guarino did not realize a subsequent article was written that disclosed numbers of test takers. His criticism is valid from that angle, however he overestimated the competence of the LDOE and newspaper to use correct terminology. Here is a breakdown of how this misunderstanding came to be: (Thanks to all who helped clear this situation up. Folks from out of state clearly underestimate the complicity and laziness of some in our local press corps.)

    these are quotes lifted directly from the article i sent to you:

    “However, the latest snapshot also shows that only 4.1% of juniors and seniors scored earned qualifying scores.”

    2 paragraphs down

    “Students have to score at least a 3 to earn college credit, and what schools accept varies.”

    “In Louisiana, those who did so (I am reading “so” as scored 3 or higher) rose from 5,144 to 6,407 this year amid a state push to boost the number of test takers through financial and other incentives.”

    OK…If you take paragraph 1 where it says 4.1% of juniors and seniors earned qualifying scores and you say there are 6,407 who earned qualifying scores YOU HAVE NO OPTION but to conclude that 6407 is 4.1% of the population which would be approximately 155K. It might be possible to say that 4.1% of the population means the entire high school population, or all juniors and seniors, or some other weird combination but that conclusion is not implied nor justified from the article.

    Now, we know those numbers are wrong, based on what you personally know to be true but there’s no way that truth can be derived from this article.

    1. Ah, so you are saying that he used the pool of all juniors and seniors as his denominator. Hmmm. I am a bit confused by that. It is a weird denominator to use.

      So this is measure is the percentage of eligible high school students who received a college credit for their AP class that year.

      Interesting. Well, no strange.

      1. It’s not clear what rates were being discussed. It appears the paper reported some sort of denominator from some large pool of high school students and called that a passage rate, which was really something closer to a percentage of students who earned credit, though not all students were tested and so not all could have “failed”. It mostly shows that LDOE has no idea what they are talking about half the time, and how The Advocate will print anything from an LDOE press release without any understanding or review.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s