The U.S. News Ratings Fiasco
The Math Behind how U.S. News and World Report Dinged NYU Stern
Keith Riegert, Managing Editor
Had I known that the average GMAT score for students accepted to Stern would be 560, I would have comfortably avoided the eve-of-test panic attack I succumbed to. I’m kidding, of course: While I may have panicked before the GMAT, last year’s average score for Stern was not anywhere close to 560—it was 720, right in the same spot it’s been for five years straight.
When U.S. News and World Report issued its 2017 Business School Rankings, dropping NYU Stern to #20 (thanks to a single missing value), the publication informed Stern’s administration that they had simply “estimated” the missing number. Well, in case you were wondering what that “estimate” was, here’s the math:
U.S. News ranks schools on a 100-point scale. This year, Harvard Business School earned a 100; Stern, a 72. The score, as we now know, is calculated from several hundred data points. But, publicly, U.S. News lists just eight data points with weights ranging from “Full-Time Acceptance Rate” at just 1.25% to “Peer Assessment” at 25%. Stern’s missing data point, which U.S. News “estimated” would have to have been “Full-Time Average GMAT Score”, weighted at 16.25%.
Although we don’t have access to the full scoring criteria, it turns out that these eight data points allow you to estimate each school’s scores with remarkable accuracy. The regression I ran of the top 50 schools (with Stern omitted) delivered a very handsome match when the data points were mapped into the predictive equation (with one glaring exception):
(In the spirit of full transparency, the regression equation and ANOVA output are included, so feel free to nerd out a little.)
Which brings us to the number 560. Plugging in Stern’s actual data to the predictive equation (including the real average GMAT score of 720) should have given the school an overall score between 83 and 84—placing Stern on the list between (#12-tied) University of Michigan’s Ross School of Business (score: 86) and (#14) Cornell University’s Johnson School of Business (score: 80). In order to peg Stern at a score of 72, the GMAT “estimate”, according to the statistical model, that U.S. News had to have used was right around 560.
As far as the ethics of data science goes when it comes to estimating missing values, U.S. News’ decision to use such a wildly inaccurate number is puzzling at best and strikingly egregious at worst. Though it’s had its own turbulent run, the U.S. News list is widely considered the premier ranking for most schools. I cannot even guess how many students rely on this list every year to accurately inform some of the most important decisions of their lives—where they apply and choose to attend both undergrad and graduate school. (I personally spent hours pouring over the list when applying to business school and happily shelled out the $30 to get full access online.) I’m not sure how a private publication like U.S. News ended up in a position of such influence over academia, but they own it now and should be held responsible for the integrity of these rankings.
To be clear, I get it, U.S. News didn’t really estimate Stern’s GMAT score; my guess is they just plugged in the national GMAT average in order to kick the school in the teeth. But Stern didn’t deserve such a low score and the “estimate” U.S. News used for Stern’s average GMAT score was a full 160 points off both the school’s real score and Stern’s five-year average. Such a careless treatment of data is baffling.
So, U.S. News, I’d like to make you an offer. It sounds like you could, maybe, use a fresh batch of data scientists. As it turns out, I graduate next year and would be more than happy to send along my résumé. My qualifications will include, among other things: An ethically sound ability to estimate missing data and a hard-earned MBA from what is, still and will be, one of the best business schools in the country.
If you’re interested in the raw data used for this article, I’m more than happy to share. Find me on Twitter @KeithRiegert