Making Sense of College Rankings

By David A. Tomar

For colleges and universities, reputation is everything. In the fiercely competitive and increasingly global higher education sector, perception is nearly as important as performance. With more than 183 million college students hitting the books across the globe, unnumbered eyes will behold countless college ranking publications.

These publications—which academics take special pleasure in deriding and universities take particular pains to satisfy—are a major driver of endowment decisions, enrollment figures, and employment opportunities. Rankings play a direct role in how aspiring college students select schools, how researchers build reputations, and how colleges position themselves in the marketplace.

This gives the leading rankers in the game considerable power in shaping student decisions, university priorities, and even national policy-orientation. To wit:

• Schools landing in the U.S. News and World Report's Top 25 will experience a 6-10% increasing in the volume of incoming applications.
• Schools making Princeton Review's Top 20 Best Overall Academic Experience will see a 3.2% surge in applications.
• And colleges that move just one percentage point upward in U.S. News rankings will see a corresponding 1% increase in applications received.
• These figures incentivize colleges to manipulate metrics such as such as selectivity, class rank, and standardized testing outcomes in order to game the rankings.
• Such manipulation can also have the effect of making schools less accessible to low income applicants. According to Salon, rankers like U.S. News “glamorize…selectivity, which creates a culture of exclusion that shuns low-income students the hardest.”
• But such concerns are overshadowed by PR interests. In a 2006 survey of global university leaders, nearly 50% of respondents said they used their institutional rank for publicity purposes.
• Moreover, a reported 68% of respondents used these rankings as a management tool to bring about strategic organizational, managerial or academic change, including embedding rankings in “target agreements” with faculty and other personnel.
• This orientation has contributed directly to the academic “arms race” whereby institutions and governments are locked in battle over “the quest for ever increasing resources.” This has had the effect of widening the gap between elite institutions and mass higher education, contributing to deeper socio-academic stratification.
• These economic implications explain why rankings even have the power to influence national policy orientation. According to a New York Times article on the Shanghai Ranking, “Indonesia, Malaysia, South Korea and Taiwan announced programs to lift at least one university into the top 100; Nigeria pledged to put two universities in the top 200.”

These quick facts merely scratch the surface. Clearly, though, there's a lot riding on college rankings.

But can we trust them? Do any of these rankings truly measure what they claim to? Can you judge your likely educational experience by using these rankings?

Well, that depends. No one ranking system is perfect. Each is vulnerable to critique and each employs a methodology that can be readily deconstructed and parsed for flaws. But if you understand these methodologies, if you recognize these flaws, and if you know how to distill the information that these rankings do have to offer, you may be able to learn quite a bit.

Whatever objections persist—and they are numerous—annual college ranking releases are stitched into the fabric of the university hierarchy. This means that you are better off learning how to navigate them than you are ignoring them.

Each year, the major players release their annual college ranking lists. The occasion of each release is marked by anticipation and media saturation. In the aftermath of each release, the winners celebrate their accomplishment, the losers issue apologies or rationalizations, and critics from all corners of academia scrutinize the rankers themselves.

In the face of this critique—and alongside the effort that colleges extend to perform well in these annual rankings—an industry of rankers battle it out to present what each sees as the most objective, meaningful, and engaging way to compare colleges.

The ranking game is dominated by a few major players, as well as a bevy of new entrants and innovators. In the coming months, we'll take a closer look at each of these players, their ranking methodologies, and some of the leading points of critique facing each of them. The goal is to create a growing set of resources on how to read, interpret and get the most out of each of these ranking systems both in light of their strengths and in spite of their flaws.

But first, let's get the lay of the land.

Hereafter, we offer a basic primer on college rankings, including a brief history of ranking, a glance at the major players in the game today, a look at the criteria driving the leading rankings, and a run-down of the leading points of criticism against them.

Full Disclosure

Transparency is an important part of ranking. Credibility extends in part from the ability of each ranking service to effectively show and defend its methodology. In that spirit, we feel it important to tell you that we are, in fact, in the ranking business as well. The Quad is the editorial arm of TheBestSchools.org, which provides reasonably objective, methodologically sound, and unbiased college and university rankings in a broad spectrum of categories.

As such, The Quad is also partnered with a tech startup, InfluenceRankings.com, which uses machine learning and search algorithms to rank universities and colleges based on influential faculty and alumni associated with them.

As the editorial arm of TheBestSchools.org, we at The Quad operate with complete and total independence and in accordance with our own Mission. Our opinions and the content we create at The Quad are neither dictated by, nor produced in coordination with, the rankings that drive either TheBestSchools.org or InfluenceRankings.com. The findings here are reported with a concerted emphasis on objectivity and with editorial freedom from the ranking apparatuses employed by TheBestSchools.org and InfluenceRankings.com.

A Brief History of Rankings

The college ranking business comes from humble beginnings. A century ago, rankings were largely honorary in nature and hardly carried the fanfare or economic consequence of modern ranking lists.

Among the first recognized rankings was a study called Where We Get Our Best Men, published by Alick Maclean in 1900. MacLean's text was mostly dedicated to profiling those so-called Best Men. However, its most consequential feature was an index which ranked universities based on “the absolute number of eminent men who had attended them.” Four years later, a fellow Englishman named Havelock Ellis compiled a similar list, this one based on the number of affiliated “geniuses,” as opposed to “eminent men.”

Neither list proposed to measure or rank the quality of universities based on this metric. However, as this methodology made its way across the Atlantic, the American spirit of competition took hold. In his 1910 publication, American Men of Science, author James Cattell made explicit the connection between alumni achievements and comparative university excellence, particularly in a table called “Scientific Strength of the Leading Institutions.”

Cattell even issued his publication with the advice that “students should certainly use every effort to attend institutions having large proportions of men of distinction among their instructors.”

This would be the guiding principal behind the dominant ranking strategy of the next several decades. Beginning in 1930 and lasting through the 1950s, Prentice and Kunkel published an annual report that ranked colleges based on how many of their alumni appeared “in the social bible Who's Who.”

Though their ranking was considered an outcomes-based way of evaluating colleges, Prentice and Kunkel were objectively honest about the inherent weaknesses in their approach. In a 1951 study, they conceded that their rankings were likely skewed because their primary source, Who's Who, suffered from an overrepresentation of ministers and college professors and an underrepresentation of engineers.

This concession would serve as prelude to a major transformation in the ranking business. Gradually, the outcomes-based approach to ranking would fall out of fashion as reputation-based measures became de rigueur. The Center for College Affordability traces this reputation-based approach to a 1959 ranking in which the University of Pennsylvania sought to measure itself against other American research universities. The study's author identified chairpersons at 25 top universities, all members of the Association of American Universities. These chairs were consulted as raters, a model which would ultimately lay the groundwork for the emergence of survey-driven, reputation-based rankings.

Over the next several decades, scholars from all around the globe offered their own methodological refinements to the process of reputational ranking. But for the better part of the 1960s and 1970s, these rankings were of greatest interest to academics. Rankings hadn't yet penetrated the mainstream consciousness yet. That was all about to change dramatically.

The Modern Ranking Sector

1983 was an inflection point. That was when a magazine called U.S. News & World Report published its first ranking list of “America's Best Colleges.” Driven entirely by survey responses, these reputational rankings would have an immediate and profound impact on the higher education marketplace. They would also set into motion the development of the broader ranking industry. The magazine began publishing the report annually starting in 1987 and since that time, has become the most frequently quoted of American college ranking outlets. Today, these annual rankings hold powerful sway over how colleges are perceived by students, parents, alumni, and employers. Each year, their ranking lists are met with equal parts excitement and critique.

At its inception, the US News & World Report ranking was entirely subjective, collecting its survey responses from university and college presidents. Starting in 1988, U.S. News undertook an effort to incorporate more meaningful quantitative data in its rankings. Since that time, chief data strategist Robert Morse has presided over an ever-evolving methodology.

By 2010, its lists had become so popular that U.S. News moved largely away from its news magazine format, making franchise ranking the dominant part of its business strategy.

For most of its history, U.S. News has focused solely on American colleges. Only in 2014 did the ranker unveil its Best Global Universities rankings. By this time, a number of leaders had already emerged in the international ranking game. The Academic Ranking of World Universities (ARWU) was first among them.

The ARWU was originally compiled and issued by Shanghai Jiao Tong University and is thus commonly referred to as the Shanghai Ranking. The inaugural Shanghai Ranking was issued in 2003 and, as an empirical ranking of universities on a global scale, was the first of its kind. It remains highly influential in the international ranking sector. Not only was Shanghai distinct from U.S. News in its global scope but it also forged a ranking methodology that eschewed reputational metrics in favor of a return to strictly outcomes-based metrics. Shanghai arrives at its rankings using criteria that are quite distinct from those used by U.S. News. Its outcomes are, likewise, quite different.

So too are the ranking outcomes produced by Quacquarelli Symonds (QS) World University Rankings, the next prominent player to emerge in the global ranking sector. QS began producing its rankings in conjunction with the publication Times Higher Education (THE) in 2004. Based out of Britain, QS is something of a hybrid, combining Shanghai's global focus with U.S. News & World Report's mix of reptutational and quantitative data. As critics are often quick to point out, QS relies more heavily on reputational data than any of its competitors.

Each year between 2004 and 2009, THE and QS collaborated to release an annual table of international university rankings. However, the two parties separated in 2009, with THE citing empirical weakness in the QS ranking methodology. Beginning in 2010, Times Higher Education began publishing its own THE World University Rankings.

THE would quickly emerge as another major player in the ranking sector, applying its own methodological refinements while employing a similar mix of reptutational and quantitative indicators.

Today, the field of college rankings abounds with competitors. Other notable ranking outlets include Forbes, The Princeton Review, Bloomberg Business, and the Washington Post, which only recently began producing rankings based on a composite of other rankings. There's also a ranker called PayScale, whose College ROI Report simply measures the cost of your college education relative to your likely earnings upon graduation.

The Department of Education has also gotten into the business, with its College Scorecard distilling certain practical indicators meant to evaluate the economic value of each college or university in its ranking.

Also, as noted from the outset, The Quad is affiliated directly with TheBestSchools.org. TheBestSchools.org is unique among ranking services for the sheer (and ever-expanding) variety of rankings it offers, across a broad range of categories relating to discipline, academic quality, affordability, and model of education.

InfluenceRankings.com, a new entrant in this sector and a partner with TheBestSchools.org, is also wholly unique among existing services. Using an algorithm that assigns influence scores to notable academic figures, and ranks university programs by their affiliation with these influencers, InfluenceRankings.com produces a ranking strictly quantified according to apparent influence across the internet.

This mix of major players and new entrants suggests a competitive and expanding college ranking sector.

Significance of Rankings

Why is this market so competitive and expansive? The short answer is that many students, parents, colleges and employers all take college rankings seriously and base important decisions on these rankings.

While we can debate the accuracy, reliability, or trustworthiness of rankings, their popularity is beyond dispute. To wit, the day that U.S. News & World Report issued its Best Colleges rankings for 2014, its website attracted 2.6 million unique visitors and 18.9 million page views.

And people aren't just looking. They're making active, life-altering decisions based on this information. According to a 2014 report from the American Educational Research Association, students do make critical application decisions based on these rankings. For instance, schools landing in the U.S. News and World Report's Top 25 will experience a 6-10% increasing in the volume of incoming applications. Schools making Princeton Review's Top 20 Best Overall Academic Experience will see a 3.2% surge in applications. And colleges that move just one percentage point upward in U.S. News rankings will see a corresponding 1% increase in applications received.

In spite of the considerable academic criticism aimed at them, college rankings make a lot of noise, and college students hear this noise as they send applications out into the world. Rankings have a direct impact on student decisions regardless of their empirical reliability.

But students aren't the only ones basing important decisions on ranking indicators. Institutions of higher learning are quite aware of the influence that rankings have on student applications and equally aware of the factors that enter into such rankings. This means that, regardless of how well a given ranking actually measures academic quality or student experience, most universities feel a certain pressure to compete.

An article in Christian Science Monitor notes that many administrators recognize a connection between rankings and the caliber of student their respective universities attract. The importance of reputation looms large, particularly when it comes to attracting international students who might lack informational resources beyond these rankings. Leading historian of American education and Harvard professor emerita Patricia Albjerg Graham observed that “rankings have become particularly important to college and university administrators, who are anxious for their school to rise in the rankings. They believe that such as rise will bring more and better prepared students.”

Of course, it isn't merely the pressure to look good that drives universities. It is the equal pressure to avoid looking bad. College rankings hold such influence on the public's perception that universities who slide in the rankings often find themselves compelled to rationalize such occurrences to their various publics. With the release of this year's U.S. News rankings, for instance, New York University saw a drop in the standings. In response, Peter Henry, dean of the university's Stern School of Business, issued a personal email apology to the university's students, taking responsibility for the failure to submit certain critical data to the rankers. He promised to tighten procedures and avoid such lapses in the future.

Henry's mea culpa underscores the pressure that colleges are under to at least participate in the ranking game. But this pressure extends even beyond what colleges experience. On the international scale, rankers like Shanghai have had a direct impact not just on the way that global universities make decisions but on the priorities that national governments emphasize in higher education.

According to the New York Times, some nations have even undertaken educational initiatives specifically designed to play to the Shanghai Ranking. For instance, “Indonesia, Malaysia, South Korea and Taiwan announced programs to lift at least one university into the top 100; Nigeria pledged to put two universities in the top 200. The ranking's biomedical bias makes it particularly influential in the developing world, where science, technology, engineering and math are seen as holding the key to economic prosperity.”

This suggests that rankings carry an influence that permeates higher education at every level, driving student decisions, college priorities, and even nationwide initiatives. Of course, this is merely a glimpse of the role that rankings have come to play in the higher education ecosystem, but we can already get a sense of their enormous power.

Ranking Criteria At a Glance

Is that power warranted? In order to answer that question, one must first understand what these rankings actually measure.

According to Simon Marginson, a professor from the Centre for the Study of Higher Education at the University of Melbourne, we can divide today's leading college ranking strategies into three categories:

• Those that rely on a multi-indicator system which assigns different weights to indicators both subjective and objective (such as U.S. News & World Report);
• Those that rely entirely on bibliometric or research-based metrics (such as ARWU's Shanghai Ranking); and
• Those that place a particularly high burden of support on reputational measures like survey responses (such as Quacquarelli Symonds and their ex-partners at Times Higher Education).

Since the three leading rankers mentioned above (U.S. News & World Report, ARWU/Shanghai Ranking, Quacquarelli Symonds) serve as prime examples of three distinct approaches, we've outlined their respective criteria here for your consideration. Check out our individual write-ups on these and other ranking services for more in-depth explanations of ranking criteria and methodologies. In the mean time, note the distinct variations that drive each of these ranking formulae.

The U.S. News & World Report Best Colleges

TheU.S. News ranking system is based on two pillars: quantitative measures proposed by education experts as reliable indicators of academic quality and the magazine's own research on the indicators deemed to matter most in education.

The following criteria drive U.S. News & World Report's rankings:

• Graduation and Retention Rates (22.5%)

-Six-Year Gradation Rate-80%
-First Year Retention Rate-20%

• Undergraduate Academic Reputation (22.5%)

-Academic Peer Surveys-66.6%
-Guidance Counselor Surveys-33.3%

• Faculty Resources (20%)

-Class size-40
-Faculty salary-35%
-Proportion of professors with the highest degree in their fields-15%
-Student-faculty ratio-5%
-Proportion of full-time faculty-5%

• Student Selectivity (12.5%)

-Admissions test scores (SAT, ACT, etc)-65%
-High School Class Rank-25%
-Acceptance Rate-10%

• Financial Resources (10%)

-Average spending per student-100%

• Graduation Rate Performance (7.5%)

-Comparative six year graduation rate-100%

• Alumni Giving Rate (5%)

Once these variables have been calculated, weighted scores are assigned and combined. All scores are then scaled to fit within a spectrum of 100 points.

ARWU/Shanghai Rankings

According to its own methodology, Shanghai evaluates every university which boasts Nobel Laureates, Fields Medalists, “Highly Cited Researchers,” and those universities with major papers published in the journals Nature and Science. Universities with significant numbers of papers indexed by Science Citation Index-Expanded (SCIE) and Social Science Citation Index (SSCI) are also considered.

• Quality of Education (10%): The number of alumni from a given institution who have gone on to win a Nobel Prize or Fields Medal
• Quality of Faculty (40%)

-Staff from the institution that have been awarded a Fields Medal or Nobel Prize-50%
-Number of faculty qualifying as highly cited researchers (as informed by Thomson Reuters' Highly Cited Researchers 2016)-50%

• Research Output (40%)

-Papers published in Nature and Science-50%
-Papers included in the Science Citation Index-expanded and Social Science Citation Index-50%

• Per Capita Performance (10%)-Derived from the weighted score of the above five indicators divided by the number of full-time equivalent academic staff of an institution

These weighted indicators are used to derive a composite score. The top ranking institution from the list is given a score of 100. All other scores are calculated as a percentage of the top score.

Quacquarelli Symonds World University Rankings

The QS World University Rankings identify four areas around which university performance is assessed: research, teaching, employability, and internationalization. Evaluation of these areas is informed by data relating to six indicators: four based on “hard data” and the other two, on “Major global surveys.”

• Academic Reputation (40%): Based entirely on the results of global survey
• Employer Reputation (10%): Based on the results of global survey of employers
• Student-to-faculty ratio (20%)
• Citations per faculty (20%)
• International Faculty Ratio (5%): Proportion of international faculty members to the overall faculty size
• International Student Ratio (5%): Proportion of international students to the overall size of the student body

Weighted items are consequently combined to determine a composite score. In 2016-2017, the QS World University Rankings assessed over 3,800 universities, ranking more than 900 of them. The first 400 schools ranked are given individual rankings while those ranked 401 and up are assigned numerical bands.

Here below, we've included a few helpful tables so that you can observe these rankings in action. We've compiled and compared ranking lists from all three of the above publications. Below are the three global ranking lists measuring overall rank, rank in mathematics, and rank in medicine, for the two most recent years available. Note the differences in ranking, both from publication to publication and from year to year. These variations can tell you much about the way that differences in criteria and methodology can affect outcomes.

Top 50 Universities Worldwide

Top 50 Mathematics Programs Worldwide

Top 50 Medical Programs Worldwide

Again, there are countless other ranking services in the game, each with its own unique formula for what constitutes excellence in higher education, be it post-graduate earning potential, faculty accessibility, or even campus beauty. We'll offer greater detail on each of their methodologies in our expanding series on college rankings.

The Trouble with Ranking

Now that you understand a bit of what goes into each of these rankings—and how distinct they are from one another—you might have a greater appreciation for some of the inherent challenges to the process.

Each ranking system has strengths, and each has weaknesses. Understanding and recognizing these strengths and weaknesses can truly illuminate the value of each ranking approach. Know what to look for (and know what you are unlikely to find), and you can learn quite a bit about a given college or university.

Ultimately, our critique is aimed not at discrediting the leading rankers but at identifying areas of need in the college ranking space and, more broadly, providing a reference for navigating this space.

Tinkering

Richard Holmes, an industry-leading ranking watchdog, administrates a reference site called University Ranking Watch. Here, he provides ongoing critique and analysis of the various university ranking systems in circulation today. He warns that for many ranking sites, methodologies are in somewhat constant flux. Either in response to academic criticism, or in the interest of refining existing strategies, or even with the intent of generating headlines, ranking services have a tendency to revise their strategies on a somewhat regular basis. This, said Holmes, can sometimes bring about notable and even problematic changes in outcome.

In some cases, these changes are modest on their face but can have significant real-world implications. According to the Atlantic, critics have charged U.S. News with imposing slight adjustments to its methodology every few years merely to give observers something new to talk about. This methodological tinkering can produce statistically significant movement in the standings for individual universities that have otherwise undergone no quantitative or material changes. This fluctuation can undermine our collective confidence in a ranking service's findings, particularly when they diverge significantly from prior findings.

Recent changes to the Shanghai Ranking offer a stark example of this pattern in action. Shanghai derives 50% of its ranking data from a source called Thomson Reuters' Highly Cited Researchers. This source underwent what would appear to be a dramatic revision over the last few years. Between 2014 and 2016, Shanghai made the gradual (though not gradual enough) transition to the updated source.

Richard Holmes notes that the rankings for 2013 reflected the old list of Highly Cited Researchers while the 2014 and 2015 rankings actually weighted the old and new lists equally. The goal in doing so was to ease the transition toward use of only the new list. All indications are that this was not the smooth transition they had in mind. In fact, the movement of some universities was so substantial that Shanghai might well have jeopardized much of the credibility it has earned to date.

Patterns throughout the Shanghai's Top 500 illustrate some dramatic ascents and descents. For instance, notes Richard Holmes, Rutgers University dropped from 39 to 96, University of Wisconsin Madison fell from 18 to 28, Virginia Polytechnic Institute plummeted from the range of 79-102 to 301-400.

Notably, Chinese schools fared particularly well in the transition. Whereas zero Chinese colleges were ranked in the top 100 in 2015, two broke through in 2016…and in a big way. Tsinghua University moved from the 101-150 range all the way to 58th spot in just one year. Peking University leapt from the same range up to 71st worldwide.

Overall, 54 Chinese schools made the Top 500 in 2016, which marked the addition of 10 Chinese schools in just one year.

Speaking free from cynicism, one could argue that most formula tinkering is done to prevent manipulation and gaming. Likewise, the web has given us an ever-expanding access to data. This means our ability to analyze these data are constantly improving. Thus, methodological changes are to be expected.

The Shanghai Ranking can at least be defended on the grounds that these changes will ultimately enhance the accuracy of its data analysis. Whether such dramatic one-year trends suggest bias is up for debate.

What is not up for debate is the empirical and psychological impact of such changes. These changes force us to question the trustworthiness of any ranking system that can undergo such dramatic data transformation in just on year. Even putting aside suspicion of bias, there is no statistical continuity between this year's rankings and last year's.

These kinds of sudden and dramatic movements can leave administrators scrambling for explanations before the public, their student bodies, their benefactors, and their alumni even when methodology and data sourcing—not academic changes—are at the root of these movements.

Lack of Scientific Rigor

In spite of their influence, most rankings lack proper experimental rigor or academic authority. For instance, in 1997, U.S. News commissioned the National Opinion Research Council to produce a comprehensive critique of its ranking methodology. The Council came to a problematic conclusion, finding that there was little apparent empirical justification for the weighting assigned to different variables. The importance ascribed to each seems to suggest little “defensible empirical or theoretical basis.”

In addition to questionable weighting of criteria, critics argue that the use of reputational metrics is itself inherently unscientific in nature.

QS uses its Global Academic Surveys to accumulate 40% of all the data underlying its `World University Rankings. With an additional 10% of its ranking attributable to Employer survey responses, a full 50% of the QS rankings are based on opinion. Granted, this is the opinion of a decidedly large sample population of respondents—the largest sample population of its kind according to QS—so the findings are not without validity. They are, however, highly vulnerable to empirical inconsistency.

In spite of the numerous steps that QS takes to control its findings, the unparalleled weight that it places on reputational survey responses makes it among the least empirically sound ranking methods in circulation. In fact, there are some startling anecdotal revelations from the data-gathering field that should give us pause. Indications are that its screening methods are somewhat porous, and inherently vulnerable to input from survey respondents that may be biased or downright unqualified.

Bias

Another common critique levied against college ranking is the inherent presence of bias, be it socioeconomic, cultural or even academic in nature.

For instance, the U.S. News & World Report rankings reward those colleges which, among other factors, demonstrate exclusivity. This, argues Salon, is used as a way to measure a college's worth, which is not only inherently inegalitarian but may also not be the truest indicator of a school's value.

As Salon phrases it, “these rankings exhibit a callous disregard for college affordability, prioritizing schools that spend more money on flashy amenities rather than scholarships and grants.” The article goes on to argue that “the magazine glamorizes selectivity, which creates a culture of exclusion that shuns low-income students the hardest.”

This ultimately tells us that, regardless of the quality of U.S. News & World Report's rankings, it is a highly flawed resource if affordability is a prospective student's top priority. This underscores one of the major takeaways from this primer on college rankings, that understanding such biases can help you navigate a densely populated ranking sector.

Consider, for instance, the implications of the Shanghai Ranking, which, even its very best, is limited in scope. From their inception in the early 2000s, the Shanghai Rankings were criticized for employing a methodology and a set of metrics that inherently favored scientific programs over humanities and liberal arts.

A quick glance at the criteria that drive the Shanghai Rankings demonstrates a clear and unmistakable focus on the natural and mathematical sciences. Its consideration of Nobel Prizes and Fields Medals is concentrated solely in the sciences and mathematics. The Shanghai Ranking also relies on scholarly citations concentrated entirely in science and mathematics.

Of course, this is not an accidental bias. In the interest of achieving what it views as an empirical set of metrics, Shanghai has dispensed with indicators that it believes can not be objectively defended.

Shanghai Jiao Tong University recognized the limitations in its own rankings in a 2004 study, conceding that “[m]any well-known institutions specialized in humanities and social sciences are ranked relatively low partly because of the imbalances in the production of articles among various subject fields. The Ranking Group tried hard but was unsuccessful in finding additional indicators that are special for humanities and social sciences.”

This academic bias is hardly insidious, but it reinforces an important theme. The better you understand the criteria, methods, and weighting behind each ranking, the better you'll understand what these rankings mean and whether this meaning is relevant to you and your needs. You'll need to decide what information can be gleaned from these rankings and what should be discarded.

Arbitrariness

This advice underscores another challenge that college rankers collectively face. The ranking business on the whole struggles with the problem of arbitrariness. As the leading industry examples demonstrate, there are various and considerably different ways that one can approach the question of college ranking. That each of the major rankings relies on its own wholly unique formula for measuring colleges against one another is itself quite revealing.

Though each of the major rankers has gone to great editorial lengths to justify its particular formula for ranking, the sheer variation of possibilities conspires to leave unanswered the one question that really matters to students: “Which college is right for me?”

According to a study by the Higher Education Research Institute (HERI), “How students use measures of institutional quality (e.g., SAT scores, student-faculty ratio, degree completion rates, etc.) is up for debate. While some research suggests students tend to factor in graduates' labor market outcomes, other analysis suggests that they rarely consider graduation rates and average student debt in their decision.”

Obviously, getting to the bottom of this debate remains a leading preoccupation for ranking enterprises. But the byproduct of this debate is problematic. Namely, many universities have become fixated not on the question of which school is right for you but on which school is right for this year's rankings. This is a recipe for misplaced priorities, most especially when the metrics and weightings used by rankers diverge from the variables that students consider most important.

Conclusion

Whether you use rankings as a key part of the decision-making process or you are merely curious where your top college choices stack up, you should know exactly what you're looking at. As this discussion should suggest, it is not enough to simply take your school's numerical ranking at face value. Dig deeper, consult methodologies, compare rankings and ultimately, remember that you are seeking the college that suits you best. Global standing aside, you must find the school that rates highest where it matters to you. Rankings are merely the starting point.

Speaking of which, this article is also merely the starting point for an ongoing series in which we feature (and scrutinize) both the major players and new entrants in this highly competitive ranking industry. Check the ever-expanding set of links below for a closer look at the major rankers in the game.

• US News and World Report's Best Colleges
• Academic Ranking of World Universities (ARWU), from Shanghai Jiao Tong University (SJTU) in China
• Quacquarelli Symonds (QS) World University Rankings