Information About the NRC Assessment Report
Questions and Answers about the National Research Council report released on Sept. 28, 2010
1. What is the National Research Council?
The National Research Council is part of the National Academies, a private, nonprofit institution that provides science, technology and health policy advice under a congressional charter signed by President Abraham Lincoln granted in 1863.
2. What is the NRC Assessment?
The NRC Assessment of Research Doctorate Programs is a national study that aims to evaluate the quality of Ph.D. programs across the United States. It was conducted by the National Research Council (NRC). The rankings will be released on September 28, 2010. A total of 4,838 doctoral programs at 212 universities were rated. The data focus on many dimensions of doctoral programs to facilitate comparisons among programs in the same field.
The 2010 report is the third time such an assessment has been conducted. The first two were published in 1982 and 1995. The current project was conducted between 2005 and 2010. The data were collected in 2006-07, based on data provided by each university about students and faculty during the 2005-06 academic year.
3. How are the 2010 NRC rankings different than the 1995 rankings?
There are substantial differences between the 2010 and 1995 rankings, from data collection to statistical analysis to the format of the rankings themselves. These differences are addressed throughout this FAQ; some of the major changes include the following:
- More programs are ranked in 2010.
- Each program will receive five ranges of rankings rather than one ranking.
- Rankings are based on different variables.
- Rankings are determined by a new statistical method.
4. What is UC Riverside’s public position on the rankings?
The report shows not only growth, but increased quality in campus graduate programs. The report evaluates 27 of UCR’s graduate programs, a snapshot in time that represents the campus’ extensive progress in research excellence when compared to rankings issued by the same organization in 1995 on 16 program. While the NRC specifically has moved away from comparison between 1995 and 2005 data, UC Riverside officials have analyzed the report to evaluate campus progress.
5. What does this mean for UC Riverside's future?
"We are very proud of the accomplishments of our faculty, staff and graduate students that showed such marked improvement," said Chancellor Timothy P. White. "UCR's success and momentum is very timely, as it will lead to our ability to meet the goals of UCR's Strategic Plan for 2020."
6. Which UC Riverside programs are ranked?
Listed alphabetically:
- Anthropology
- Biochemistry
- Chemical and Environmental Engineering
- Chemistry
- Comparative Literature
- Computer Science
- Dance History & Theory
- Geology
- Evolution, Ecology and Organismal Biology
- Economics
- Electrical Engineering
- English
- Entomology
- Environmental Toxicology
- Hispanic Studies
- History
- Mathematics
- Mechanical Engineering
- Philosophy
- Physics
- Plant Biology
- Plant Pathology
- Political Science
- Psychology
- Sociology
- Soil and Water Science
- Statistics
The data has been compiled into a series of graphs, which may be downloaded as PDFs. The graphs are separated by college and compare UCR to AAU-accredited schools, schools in the UC system and top schools across the nation.
The graphs indicate a program's numerical ranking range on the S scale. Higher rankings are smaller numbers. The farther to the left on the graph a range is situated, the higher ranking for the beginning of that range.
7. What properties of departments are rated? What are the five different rankings?
The NRC used 20 variables that it considers “indicators of program quality.” Variables include measures of faculty research activity, student support and outcomes, and faculty and student demographics. The indicators come from the extensive data provided by the institutions themselves as well as some data collected by the NRC (e.g., faculty awards, publications, and citations).
Each program will receive five ranges of rankings:
- Overall S-rankings (“Survey”): Based on 20 variables, weighted based on field-specific faculty opinions of the relative importance of the various program factors.
- Overall R-ranking (“Regression”): Based on 20 variables, weighted based on field-specific faculty rankings of actual programs.
- Research Activity subscale: Based on four variables used in the Overall rankings.
- Student Support and Outcomes subscale: Based on five variables: Four used in the Overall rankings, plus “Whether the department collects student outcome/placement data.”
- Diversity of the Academic Environment subscale: Based on five variables used in the Overall rankings.
More information on how the ranges were calculated, how the variables are defined, and data sources are below.
8. What does it mean that these are “ranges of rankings”?
The rankings will be presented in a different form than most other rankings. Rather than receiving a single ranking (e.g., 1st, 5th, 32nd), each program’s five sets of rankings are presented in ranges. The ranges mark 90% confidence intervals.
A program’s range of rankings might be, for example, 2-8 or 4-27 or 13-37. These ranges reflect the inherent uncertainty of ranking a particular program due to differences among raters, statistical uncertainty, and variability in year-to-year data. These ranges of rankings are intended to reflect greater statistical certainty. A range of 2-8 should be read, “It is 90% certain that the program is ranked between 2nd and 8th in this field.”
More information on how the ranges were calculated is below.
9. How did UC Riverside do?
Joseph Childers, UCR’s dean of the graduate division, said that he sees significant progress in the 10 years between the NRC rankings and the current report shows UCR comparing well among national universities.
In 1995, one UCR program, evolution, ecology and organismal biology, was in the top 20 percent of such programs in the nation. Graduate programs in agricultural sciences were not included in that national survey. Many other programs at UCR were too new or too small to be ranked at all.
This time, ranges were provided based on data supplied by universities as well as surveys of faculty. It is difficult to make a direct comparison, but looking at the S ranges, Childers sees eight UCR graduate programs were are in the top quartile of programs in the nation. Those eight programs are, in order:
- Entomology
- Computer Science
- Electrical Engineering
- Mechanical Engineering
- Chemical Engineering
- English
- Plant Biology
- Environmental Toxicology
Six more doctoral programs have S ranges in the top third nationally. They are:
- Evolution, Ecology and Organismal biology
- Psychology
- Plant Pathology
- Physics
- Soil and Water science
- Political Science.
10. Can it be said that UC Riverside has a top graduate program?
This report indicates that our entomology program is among the top two in the nation.
11. How many Ph.D. programs does UCR have?
39
12. What data did the NRC collect?
The NRC collected data on 20 key markers of quality, which included the amount of faculty research activity and awards, student support and outcomes, and faculty and student demographics. The indicators come from the extensive data provided by the universities as well as data collected by the NRC. In addition, faculty members at the nation’s universities were able to indicate what factors make for a top-notch graduate program.
13. Why isn't my Ph.D. program ranked at all?
Some programs were too small or too new, or the field did not fit into a NRC category.
14. How were the ranges of rankings calculated?
The ranges of rankings were produced from a complex statistical analysis. A brief summary follows. The complete methodology is found here.
Overall S- and R- ranges of rankings are derived from the values of the indicators and the field-specific weights for each variable. The S- and R- weights differ by field, recognizing that faculty members in different disciplines value different aspects of doctoral programs. The 20 variables are weighted to produce quantitative estimates of program quality. The field-specific weights are based on two faculty opinion surveys conducted in spring 2007.
- S-weights, based on surveys: The first survey asked all faculty across all fields to rate the importance of 21 variables that influence overall program quality.
- R-weights, based on regressions: The second survey, the “implicit” or “anchoring study,” asked subset of faculty to rate a sample of programs in their field. Regression analysis was then used to determine which quantitative variables at what weights most closely predicted the program rankings in each field.
Ranges of rankings: Rankings from many raters were aggregated and arranged in order to yield ranges of rankings. The NRC study used a “random halves” procedure in which weights are calculated based on the responses of a randomly selected subset of faculty respondents. This is done 500 times, calculating 500 rankings that are ordered each time. The 500 resulting rankings are ordered from best to worst, and the bottom five percent and top five percent are dropped. This results in two scores for each program covering the middle 90% of the 500 rankings.
15. How are these data defined? How can I check whether my department’s data is correct?
The data categories and definitions used by the NRC are often different from those used in most UC Riverside reports. The data may not coincide with numbers in UC Riverside fact books and other university information sources. Therefore, in understanding and checking your department’s data, it is important to understand the details of how each variable is defined, what it measures and how it was calculated. Data definitions are detailed in the NRC methodology guide.
The data generally reflect the academic year 2005-06, though some is cumulative data collected up through the 2005-06 academic year. Many characteristics of UC Riverside’s rated programs have changed substantially over the past five years.
Some of the most important data definitions are:
- Faculty data are based on the number of Core, New, Associated, and Allocated Faculty for each program, as defined by NRC.
- Core” faculty members are generally Academic Senate members with a primary, secondary, or joint appointment in the department.
- “New” faculty members are like Core faculty members, but with an appointment beginning between 2003 and 2006.
- “Associated” faculty members are affiliated with the program through a Courtesy, Acting or other similar appointment, or through dissertation advising.
- Assignment of Core, New and Associated faculty was done by UC Riverside. NRC then determined the number of “Allocated Faculty” using an algorithm based on data about dissertation committee supervision and membership to allocate faculty members on a proportional basis to all departments with which they were affiliated.
- Student data are based on an NRC-defined set of entry cohorts, as well as criteria for continuous enrollment, full-time enrollment, and program status. Student numbers may seem lower than the population generally thought of as associated with a particular department.
- The 18 student activities measures (e.g. “Is there an orientation for graduate students in this department?”) give each program credit for each activity provided by either the university (answered centrally, for all departments) or by the department (answered by each department). Each program received credit for nine activities provided by UC Riverside-wide, plus any others provided by the program.
16. Where did the data come from? Who at UC Riverside vetted the data?
UC Riverside participated in the data collection process by providing data about its programs, faculty and students to the NRC in 2006-07. Some data were also developed directly by the NRC, including data on publications, citations and grants.
UC Riverside data were generated centrally by staff in the offices of the Dean of the Graduate Division, and by the office now called Strategic Academic Research and Analysis. Other data were provided by individual programs.
17. These data don’t reflect the current state of my department. Can we update them?
Across the country, most departments have changed since 2005-06, the time period reflected in the study. These changes may include demographic shifts, policy changes, or departmental reorganizations. This is only a snapshot in time. There is no mechanism to update the report at this time.
18. I filled out the faculty survey, but I don't see most of those questions reflected in the report.
Data from these faculty surveys contributed primarily to developing the variable weights used in the two overall rankings. The general faculty questionnaire also supplied data for the final rankings on how many faculty members in each department are supported by grants. Other data from this questionnaire, like much of the data provided by departments and universities, was ultimately not used. (Institutional and Program Questionnaires are available on pp. 55-121 of the NRC methodology guide.) This is the result of the NRC’s statistical process for identifying a small set of variables (ultimately 20) which they propose as indicators of program quality.
19. How can faculty and departments use the rankings?
These rankings may help programs identify characteristics of their students, faculty, and program features, in comparison to other programs in their field. For example, one can determine whether the program had more or fewer female faculty members, or how their mean time to degree compares. Even so, this should be done with caution.
Be aware that each of these data items is very precisely defined by the NRC, and that the definitions are not necessarily intuitive, or the same as those used in most UC Riverside data reports.
20. How can prospective graduate students use the rankings?
One possible use of these rankings is to allow prospective students considering doctoral studies to compare programs. Every student’s assessment of the best place to pursue graduate studies should be based on his or her own analysis of what the program will have to offer when they plan to undertake pursue their degree. The decision of where to enroll should not be based on a rating or ranking from any organization.
Prospective students could use the information in the study to help them consider and inquire further about different dimensions of a particular program, for example in Research Activity, Student Support, and Diversity, and then place more weight on the those program characteristics that are more important to each individual.
21. How can current graduate students use the rankings?
Current graduate students considering academic careers, much like prospective graduate students, may use these data to compare and inquire further about characteristics of programs at different universities.
Graduate students may also use the data to put their educational experiences in a wider context. For example, graduate students may have a good understanding of the demographics of their particular department, or of the average number of publications by their program’s faculty. They may not, however, know whether these properties are typical of programs in their field. The comparative data provided by the NRC can assist students in contextualizing their experiences.