Department of Education Study Is "Seriously Flawed"

In October 2005, the Policy and Program Studies Service of the United States Department of Education released a 148-page "review" of research on single-sex education.  This review was commissioned by the Department of Education and was performed, under contract, by five individuals at the American Institutes for Research (AIR).  AIR was selected for this task by RMC Corporation.

 

The review is not friendly to single-sex education, which is not surprising, considering what we know about RMC Corporation, but its not well-researched or well-executed, which is a bit surprising, considering that the Department of Education paid real money to have five people write this thing.

 

I (Dr. Sax) sent my criticism of this review to the five authors, since I thought it was only fair for them to have a chance to respond to my criticism before I posted it online.  Those authors were courteous enough to respond.  Their response follows below, after my criticism. 

 

The review is entitled "Single-Sex vs. Coeducational Schooling:  A Systematic Review."  You can read a summary of the report, and link to the full text in PDF format at this link:  http://www.ed.gov/rschstat/eval/other/single-sex/index.html

 

 

Criticism #1:  The "systematic quantitative review" is neither systematic nor quantitative.  

The authors describe their report as a "systematic quantitative review" (page 11).  They try to make their review appear quantitative by listing each study they reviewed as either "pro-SS, pro-CE, or null" (in favor of single-sex education, in favor of coeducation, or favoring neither).  They then add up the number of studies pro-SS, the number of studies pro-CE, and the number of studies which are null.  However, they do not take into consideration the number of students in each study.  

 

For example, the 2002 report by the National Foundation for Educational Research showed a dramatic beneficial effect of single-sex education in broadening educational horizons for girls:  girls who attended single-sex schools were much more likely to study advanced math and science than were girls of comparable ability who attended coed schools.  This study involved 369,341 pupils from 2,954 schools, and was published as a 108-page monograph.  But for the authors of the Department of Education report, it gets just one check mark in the pro-SS column, on page 51 of their report. 

 

The authors devote the entire previous page, page 50, to an unpublished, unreviewed study – a presentation at a conference of university administrators in Minneapolis in 1992.  This study involved fewer than 200 students at just TWO schools, and reportedly found a benefit of coeducation compared with single-sex education.  So that study gets one check mark in the pro-CE column. 

 

There is no weighting whatsoever for the fact that the published study favoring single-sex education involved 369,341 students at 2,954 schools, in hundreds of different cities and towns, while the study purportedly favoring coeducation was an unpublished presentation at an obscure conference, involving fewer than 200 students at just two schools in one community.  As far as these authors were concerned, it was a wash.  Quantitatively speaking.  One study "pro", one study "con."

 

This sort of writing is not systematic.  It is not quantitative.  It is not scientific.  It is poorly executed.

The Department of Education should demand their money back!

 

Criticism #2:  The "exhaustive, systematic review" is neither exhaustive nor systematic.    

The five authors of this 148-page report assert, three times, that they conducted "an exhaustive search" of the published scholarly literature. In other words, they are telling us that they included all relevant studies.  But they did not.  For example:  The authors pose the question of whether students at single-sex schools are more or less likely to develop eating disorders, compared with students at coed schools.  On this last question, the authors "exhaustive search" turned up just one study, according to which girls at single-sex schools were more likely to have eating disorders than girls at coed schools.  The authors say that they used the PsychINFO database as one of their principal sources.  However, a quick search of the PsychINFO database reveals not one, not two, but four studies which meet the authors quantitative criteria.  You will find the four studies listed (in PsychINFO format!) at www.singlesexschools.org/eatingdisorders.htm.  Two of those studies found that girls at coed schools wanted to look thin because they thought that's the way the boys wanted them to look.  Girls at single-sex schools were much less concerned with what the boys wanted.  None of this appears in the Department of Education report. 

 

The authors excuse (see the full text of their response, below) is that they were not able to get the full text of these other studies, so these other studies were not included.  That excuse is not persuasive.  The full text of these articles is readily available from services such as Infotrieve.com. 

 

Another example:  the authors list six studies which examined the effect of school format, coed vs. single-sex, on self-esteem.  However, the best-designed of these studies, the famous study by Jacqueline Granleese and Stephen Joseph, which I described in the closing chapter of Why Gender Matters, is not even mentioned anywhere in the report.  As you may recall, Granleese and Joseph found that for girls at coed schools, personal appearance, i.e. whether or not the girl believed she was pretty, was by far the most important determinant of self-esteem.  For girls at coed schools, if you are pretty, life is great.  But if your appearance does not qualify you as one of the cute girls, then life is not so great.   At coed schools, for girls, the focus is on how you look.  For girls at single-sex schools, though, appearance was not a key determinant of self-esteem.  The focus at girls schools is more on who you are rather than on how you look. 

 

This study is never mentioned in the Department of Education report, even though the study fulfills all the criteria for inclusion. 

 

Criticism #3:  The "exhaustive, systematic review" is not a review, but merely a bibliography.

The most important failing of this expensive study has nothing to do with the fact that the authors missed lots of important studies, despite their "exhaustive search."  The most important failure was the failure to address the key question raised by any review of this literature, namely:  WHY are some schools successful with this format, while others fail?  The authors list studies which show that boys did better at single-sex schools than at coed schools; they list studies showing that boys did worse at single-sex schools; they list studies showing that school format had no effect.  The authors never address the question:  why?  Why does single-sex education lead to tremendous gains at some schools, and disastrous outcomes at others?

 

That is why I consider this report merely an annotated bibliography, not a review.  A review includes some DISCUSSION of the variation in the results, and some attempt to consider various HYPOTHESES which might account for the variation.  This report contains none of those elements, merely an annotated listing of articles.

 

 

Criticism #4:  The authors of the study give no consideration whatsoever to the possibility that GIRLS AND BOYS LEARN DIFFERENTLY.  This possibility is never even mentioned anywhere in the 148-page study.

If teachers do not understand that girls and boys learn differently, then the single-sex format is not a recipe for success:  in fact it may lead to disaster.

This is NOT to say that all girls learn one way and all boys learn another way. We understand, respect and indeed cherish the very wide variation within the sexes. But differences between the sexes are real, just as differences between 8-year-olds and 11-year-olds are real. Again, one can assert that there are age differences in learning without necessarily asserting that all 8-year-olds learn one way and all 11-year-olds learn another way. In the United States, most schools segregate students on the basis of age. 8-year-olds are in one room, and 11-year-olds are in another. Sex differences in learning are much more robust than age differences, however (see Why Gender Matters for supporting documentation).

please consider two recent examples from American public schools.  On September 30, 2005, the Orlando Sun-Sentinel reported the success of a single-sex program at the Woodward Avenue Elementary School in DeLand, Florida.  Students were assigned either to single-sex or coed classrooms.  After one year, among 4th-grade boys assigned to coed classrooms, only 37% scored a grade of 3.5 or higher on the FCAT writing test.  Among 4th-grade girls assigned to coed classrooms, 59% earned a grade of 3.5 or higher on the FCAT writing test.  But, of the 4th-grade boys assigned to single-sex classrooms, an astonishing 86% earned a grade of 3.5 or higher on the writing test, while 75% of girls assigned to single-sex classrooms scored a 3.5 or higher.  The single-sex format not only improved academic performance; it eliminated the gender gap altogether.  This story made the front page of the Sun-Sentinel.  The figures just given here come not from the newspaper article but from JoAnne Rodkey, the principal of the school, who was kind enough to share them with us. 

 

On the very same day, September 30, 2005, the local newspaper in Lancaster, Pennsylvania recorded the failure of a similar program.  After two years, the gender-separate program at Reynolds Middle School recorded no academic gains, and the teachers voted to get rid of it. 

 

The authors of the Department of Education study would score the Florida result with a check-mark in the column favoring single-sex schools, and they would score the Lancaster study as "null."  But the important question is:  why the difference?

 

The authors of the Department of Education study might suggest that these programs are more likely to succeed in low-income, underprivileged neighborhoods serving primarily Black or Hispanic students.  That explanation will not work here.  Reynolds Middle School has a higher proportion of students receiving free or subsidized student lunches than the Woodward Avenue has, and the Reynolds school has an almost 100% minority population compared with less than 50% at Woodward Avenue.  Yet the programs at Reynolds failed, while the program at Woodward Avenue has been successful.

 

In fact, the explanation is simple and straightforward.  Teachers at Woodward Avenue Elementary School made a conscious effort to identify and implement effective gender-specific teaching strategies.  Teachers at the Lancaster school (Reynolds) did not.  Ms. Rodkey and her colleagues from the school and from Stetson University spoke led two sessions at our conference, on October 8th and October 9th.  Ms. Rodkey sent me an e-mail last week in which she made this key point:  “Our program works so well because our teachers want to do it.  After much study and discussion, they volunteered to take on the challenge.”

 

Right there, Ms. Rodkey has identified a key variable which the authors of the Department of Education study never address:  namely, the teachers' attitude toward the format.  Recall that in the Cambridge University study which I shared with you earlier this year (Warrington & Younger, 2005), the educators' attitude was the single most determinant of the success or failure of single-sex educational interventions.  The authors of the Department of Education study never mention teachers' attitude toward single-sex education – nor do they mention any of the three scholarly papers which the Cambridge University group (led by Warrington & Younger) have published on this topic over the past five years.

 

Girls and boys learn differently.  That's the key point, of which the authors of the Department of Education study show not the slightest awareness.  Indeed, their jargon is redolent of the 1970's sociologists who would dismiss such as a claim as “essentialist.”  Studying single-sex education without any awareness of gender differences in learning is like studying the ups and downs of the Dow Jones stock index without any knowledge of what stocks are.  You can make all sorts of charts and computations, but in reality you don't have any idea what you're talking about. 

 

That's why our Association generally, and I personally, have serious reservations about any study which compares "single-sex schools" as a category with "coed schools" as a category.  Just putting girls in one building and boys in another is not a guarantee of success.  If teachers don't have appropriate preparation, and knowledge of best practices for single-gender education, the single-gender format has no pedagogical advantages over the coed format, and may in fact result in educational shipwreck.  I made this point at some length in my essay for Education Week published earlier this year.  You can read that essay at this link:

http://www.singlesexschools.org/edweek.html

 

  

That concludes my criticism of the Department of Education report – at least for now.

 

© Leonard Sax, MD, PhD, Executive Director, NASSPE

19710 Fisher Avenue, Suite J

Poolesville, Maryland  20837

phone:  301 972 7600

fax:  301 972 8006

 

 

P. S.  As noted above, I sent an earlier version of this e-mail to the authors of this report.  The earlier version did not include criticism #1.

Here follows the response from the authors of the report, complete and unedited:

 

 

 

October 26 2005

 

Dr. Sax,

 

Thanks again for allowing us to reply to your criticisms about our review.  You raise some very relevant and good points about some potential omissions to the review, as well as some more general criticisms.  We have addressed these in the comments below.

 

1)       On your point about studies concerning student eating disorders, you obviously found additional studies than the single one that was included in our review.  There was a good reason for that.  We also found those additional three studies in Phase I of the review (see pages 3 and 4 in our review).  However, they appeared in journals that we were not able to locate (as we needed to review the full text of the articles, rather than merely the abstracts).  Our efforts involved a number of steps: 1) search for the full text of the article in PsycINFO, 2) search for the article using our organizational library resources (which involved contacting the journals directly about purchasing the article) 3) search for the article at local universities, and 4) attempting to contact the authors directly (if we were able to locate their contact information).  In all three cases, none of the steps resulted in obtaining the full texts of the articles.  Therefore, they could not be included in the review.

 

2)       On your other example of studies concerning student self-esteem, we did miss the Granleese and Joseph article that you mention.  This article did not turn up based on our searches of the research article databases (see page 3). 

 

3)       Your most significant criticism of our review is that it does not deal at all with the “why” question – why single sex schools would ever work better and under what conditions they would work better. That is correct; the report on the website does not cover “why” issues, as it was beyond the scope of the review. We actually authored a separate manuscript which addresses the potential reasons why single sex schools might work better or worse than coeducational schools, including the argument that boys and girls have different learning styles and are best taught in environments geared to their unique styles.  Although this paper was not put on the USED website, it is currently under review at a peer-reviewed journal.

 

Again, we appreciate the opportunity to reply to your criticisms and are happy that this important issue is receiving its due attention. 

 

Sincerely,

 

Fred Mael and the AIR team

 

 ----