SFUSD has
been hyping the success of SIG since the program ended in 2012-2013, last school year. To be sure I was skeptical about this $45M grant from
the get-go - what I considered to be an absolutely obscene amount of money to throw at nine schools over a 3-year period, especially with so little preparation time. That amount of money is more than the annual budgets of over 30 elementary schools combined. After having spent considerable time researching SIG during the first two years of the program,I commented repeatedly on another blog, the SF K Files, about the waste, fraud and abuse that I discovered. So, while I acknowledge I am not the most
objective observer to comment on this $45M program, I also did a lot of homework on the subject. I put this caveat up front so readers can review
these results with that in mind. Of course, I believe that any bias I may have is not reflected in the numerical results or comparisons drawn. If anyone thinks otherwise I would like to know and please feel free to comment with any concerns.
I'm not a statistician and is this not an in-depth analysis nor does this simple bread and butter statistical comparison require a higher level of scientific procedure. We are simply reviewing the numbers based on information gathered from California Department of Education API reports.
Results
In short, what
I gleaned from the API data was that 5 of the 9 schools did little to no better than the District API achievement averages and three did worse. Nevertheless SFUSD's review is
misleadingly positive. SFUSD pats itself on the back for a job well done though $45,000,000 in grant money for approximately 3,860 student participants did not raise achievement numbers in any significant way for most of the schools.
Method
The method
of analysis I use is simple. I compare the API of the last year
before the start of SIG (2009-10) with the final year API (2012-13). Many of
you know that the now defunct API index is a statistically adjusted achievement
metric. It is different from using STAR,
an absolute metric, though individual STAR results provide the raw data for API.
That data is adjusted for statistical
variants that would cause averaging errors. As examples, and there are many,
new students are not included in an API because they don't figure in a school's
growth and Students With Disabilities are not included as the kind and number
of SWDs varies considerably between schools and creates an apples to oranges comparison. Also, it should be noted that API has a subjective quality
that gives additional weight to growth at the lowest quintile because that is where
schools are expected to target remediation efforts and are statistically rewarded
for doing so which handicaps the results at low performing schools for the better. This gives low performing schools a leg up and leans in favor of increased
success when large grants are employed with the intent to focus on that
lowest performing quintile. Also, schools in the lowest quintile
typically show a potential for more upward mobility as there's more room to grow. Conversely,
high performing students show less upward mobility in a sort of law of
diminishing returns, if you will. In any case, API results yield more
favorable numbers than STAR test results. Since STAR is only assigned by individuals and
grades, not to schools overall, and because of the adjusted quality of API for school-wide analysis and comparison, API is a better tool than STAR and really the only tool, despite the preferential statistical treatment of lower performing schools.
I would be remiss not to point
out one problem with the STAR data used to create API. It has a Federally
mandated test participation rate of 95% to be considered scientifically viable and legal. Over the years few underperforming
schools have met that target participation rate and the Feds have looked the other way. Those of you who have
reviewed STAR results may be aware of this participation rate issue. If you
look at the results on the CDE's STAR site of high performing schools in SFUSD,
invariably you will notice that participation rates are well over 95%. However,
low performing schools often have much lower rates, sometimes as low as
80%. This is a variable that is hard to factor in, but the general rule is that test
no-shows are low-performers. In effect, if rates were higher at low performing
schools, it is likely that the results would be lower than they are. It's
something to keep in mind and bears mentioning since it also handicaps results to favor low performing schools.
SIG was in
effect for the school years 2010-11, 2011-12 and 2012-13. I compared the year (09-10) before the grant started with the year of the grant ended (12-13). This allowed for a comprehensive review since individual yearly
statistics varied and the important numbers were how the schools started out versus how they ended up. Also, SIG
got a late start in the first year and growth that year can't be compared
correctly with the next in a year-over-year comparison. Also, Chavez ES had its first-year tests invalidated
due to some funny business during test administration.
The nine SIG
schools are Bryant, Chavez, Carver, Muir, Buena Vista-Horace Mann, Revere,
Everett. Mission and O'Connell. I will not be including Buena Vista Horace Mann
in this analysis because that school became a K-8 and adjusted API data is not
available. Willie Brown received some SIG
funding for school closure, though it is reopening soon. SFUSD favorably
compared the SIG schools to the other five schools in the two
Superintendent Zones, Bret Hart, Drew, Malcolm X, Flynn and Marshall HS. Three
of those schools actually dropped over the 3-year period which meant that
SFUSD's comparison was really a joke. That is to say, it is easy to compare
favorably to schools that lost ground (and rather ridiculous). Below I have listed the nine SIG schools and the five other schools included in the SZ. The names are followed by the 2009-10 API,
10-11,11-12 and the 2012-13 API. The BVHM numbers are not accurate apples-to-apples
comparisons due to the merger of BV and HM.
DATA
Name 09, 10,
11, 12
Bryant 696, 701, 731, 703
Chavez 685, NA , 661, 690
Muir 635, 689, 715, 731
BVHM*...........653, 682, 727, 748
Carver.............701,
701, 740, 755
Revere 655, 683, 753, 772
Everett 607, 638, 693, 728
Mission...........
625, 642, 639, 641
O'Connell........603,
594. 667, 656
****************************
Bret
Hart 627, 650, 655, 648
Drew 710, 609, 677, 665
Malcolm
X 800, 790, 724, 711
Flynn 706, 710, 738, 694
Marshall HS 758, 774 ,768, 783
Summary of Data
The obvious first
conclusion that can be drawn is that the Bryant, Chavez and Mission results are
absolutely awful with or without an investment in millions. Underperforming
the District, their 7 and 5, and 16
respective point increases over the
3-year period are negligible and statistically insignificant. Small
variations are statistical noise as school APIs experience normal random
fluctuation year over year. Remember that these numbers do not represent a one
year change, but the total change over three years making these results very
disappointing indeed. These lowest performers I term Tier 3 schools.
In the
second tier, are the very mediocre results of Carver and O'Connell which rose
54 and 53 points over the period. This is mildly above the overall
district results, but only when the low quintile weighting is not factored in. These results also do not speak particularly well to the efficacy of the SIG
grants, though the results are better than the abysmal improvement of three Tier
3 schools.
In Tier 1,
are Muir (96 pts.), Revere (117pts.), and Everett (121 pts.). These schools far
outperformed the other SIG schools with the possible exception of BVHM which
was not included in this comparison and they represent the high point of the
SIG program. These schools showed steady
gains over the period. Bryant, Mission and O'Connell did not manage to post
increases in all three years and Bryant and O'Connell scores dropped significantly
in the final year, which is very disappointing considering that the
implementation should have shown the best results in that third and final year.
In regards
to the rest of the five schools also in the Superintendent Zones that were not SIG
recipients, their results were terrible in the main, especially considering that these
schools were also given many additional resources, though not nearly so much as the
SIG schools. Of the five schools three of them, Drew, Malcom X and Flynn
actually dropped over the same 3-year period. The other two, Bret Hart and Marshall saw
district- average size increases. What does this say about the benefit to SFUSD of Carlos Garcia's and Richard Carranza's Superintendent Zones - a program that has been the District's highest priority?
Conclusions
The School
Improvement Grant program over the 3-year period produced very
different achievement results among recipient schools. Three schools did very
well, over-performing district averages, and three did very poorly, underperforming district averages. Two schools
showed mediocre but around average compared to non-SIG schools, thereby under-performing as SIG recipients. One school was excluded from analysis. Approximately $12,000 -$13,000 was spent per student
on average in the nine schools over the 3-year period, factoring in attendance. It is difficult to concur with SFUSD that
the grant program was successful since at least half the schools didn't do any
better than the district as a whole and some did considerably worse, especially considering the statistical benefits enjoyed by low performing schools, as previously mentioned. Despite the extraordinarily large and unprecedented windfall funding that was SIG, the costs versus results point to a very uneven intervention with more poor results than good ones. The fact that some schools outperformed while others underperformed speaks
to poor implementation at the district level where money totaling in the millions was spent on administration.
I would be remiss not to point out that statistical comparisons are difficult for many reasons. Besides the bottom quintile advantage, the highest quintile sees the smallest gains year over year as it becomes more and more difficult to improve over-performance. So comparing districtwide averages gives a great advantage point for point to the lowest schools. For example, a school above 900 is hard-pressed to raise a score by 30 points in a year but a low performing school could easily see a similar point improvement. The real gains of the SIG schools, especially the five Tier 2 and 3 schools was bad, indeed, considering the unprecedented funding made available to them during the last three years. So why is SFUSD gloating?