U S News To Reform Its Disclosure Of Surveyed Employment Data
Updated: 12/15/2010 4:15 pm CST
Thanks to around five months of correspondence between LST and U.S. News, we are happy to report that the organization behind the notorious law school rankings has agreed to make additional information available on its website. While the additional information will not change how U.S. News computes their rankings, it will help make things much clearer for prospectives trying to take apart the employment statistics that U.S. News currently collects. This information is far more detailed than the information required by the ABA, and making it more visible will help prospectives while we continue lobbying the ABA to reform their practices.
In August, LST contacted the U.S. News' ranking guru, Bob Morse, with concerns about the display of employment information included for each law school on its website. Specifically, we were concerned that a number of answers that schools provide to U.S. News in its annual survey were not being disclosed. The effect of the missing answers, we believed, was to limit the usefulness of the employment figures beyond their documented substantive and formal flaws (more on those in our white paper).
After listening to our concerns, U.S. News has agreed to reform how it discloses the results of its annual survey. In doing so, U.S. News will adopt our suggestions and will soon make changes to its website for the Class of 2008 employment information. These changes will remain in place when the new rankings become available (around 3/15/2011), which will include employment information about the Class of 2009.
Thanks for following up. I will try to answer your question clearly. As I said earlier USNEWS is doing a major redesign on the education section of usnews.com website which means Best Grad Schools and Law Schools. Our current plan is that U.S. News will add all the new fields that you suggest to current data for 2008 graduates and those new fields will show up when the redesign goes live in late Winter 2011. Then those fields will remain as part of best law schools web site when [we] launch the next rankings with 2009 graduates data around 3/15/2011. If you have other questions, let me know.
Bob Morse
Update. We received this email a few days ago with a few clarifications from Mr. Morse:
I read your blog post (https://www.lawschooltransparency.com/2010/12/u-s-news-to-reform-its-disclosure-of-surveyed-employment-data/ ) and I wanted to make a few key comments and clarifications from my earlier email.
1. U.S. News is in the midst of doing a major redesign of the education section of usnews.com as I have mentioned earlier.
2. Doing a such a redesign is a very large scale project and the exact timing of when parts will be done is hard to determine.
3. The current plan is that the redesign will be rolled out in phases in late winter 2011.
4. The current plan is that the new law data fields for 2008 graduates will be added as part of one of those phases, but not as part of the first phase.
5. U.S. News is committed to adding these new data fields from our statistical survey, if we are unable to add them as part of the roll out (described above) of the redesign, the current plan is that they will be added for 2009 graduates data when the next Best Graduate Schools law school rankings are launched.If you have questions, let me know.
Bob Morse
We made a number of suggestions by doctoring U.S News' current website. You can view the original here. Our mockup page can be viewed here. Although probably self-explanatory, "A157″ (etc.) represents the answer to question 157 on the 2010 survey (the relevant survey questions are included after the jump). We identified the fields we changed in yellow.
Our suggestions are designed to tie more closely to what the survey questions ask schools. These suggestions include language changes, field position changes, and indented fields whenever a group fits under the previous field. Taken together, we think these changes more aptly tell the story of a school's post-graduation outcomes.
Summary of Suggestions:
Better Language: Changed "Graduates employed at graduation" to "Employed at graduation rate"
Better Language: Changed "Graduates known to be employed nine months after graduation" to "Employed at nine months rate"
This reflects the fact that the percentage does not actually reflect the graduates employed at graduation, but a rate calculated using a proprietary formula. The language must indicate to readers that the percentages that follow below are school-reported rather than U.S. News-derived.
New Section: "Class of 2008 Graduates – Class Breakdown at Nine Months" is a new section, which adds new fields and includes the entire "Areas of Legal Practice (Class of 2008)" section
This clarifies that the percentages are for the nine-month measurement period. The goal with the new fields below is to show more realistic percentages, especially when the percentages of a category add up to 100% but do not reflect 100% of the class.
New Fields: This section includes 7 new fields:
"Graduates whose employment status is unknown"
"Graduates whose employment status is known"
"Graduates known to be enrolled in a full-time degree program"
"Graduates known to be unemployed and seeking work"
"Graduates known to be unemployed and not seeking work"
"Graduates known to be employed"
"Percent employed in a judicial clerkship by an Article III federal judge"
The "Graduates whose employment status is unknown" and "Graduates whose employment status is known" fields, along with the "Graduates known to be employed" field, are the most important new additions. These figures are required for determining how much of the class is actually employed with certain types of employers, particularly those in private practice (law firms + business and industry). The private practice percentages are crucial for determining how many graduates schools used data for in calculating the salary information. These numbers would allow us to relax our assumptions in LST's data clearinghouse.
The other fields round out the full picture of a school's graduating class. While the underlying data are still missing, these nuances help draw attention to the complexity these apparently-simple percentages have.
Changing Location of the Salary Reporting Percentage:
"Percent in the private sector who reported salary information"
One of the most glaring problems with how employment information is presented is the prevalence of the "median private sector salary" statistic, which is in reality only the median salary for the sometimes small percentage of graduates who actually reported salaries. This change puts the percent reporting salary up front, before people see the often exaggerated 25th, median and 75th salary statistics. While moving the location of the percent reporting is a small change, it could help combat the tendency of prospectives to look at the reported median salaries and assume they are actually the median, when in reality they are often significantly higher than the actual (but undisclosed) median salary.
Interactivity:
We also made a technical suggestion regarding using jQuery tooltips:
Finally, and this is for your site developers, I noticed that your website loads jQuery. With this in mind, consider jQuery tooltips, part of the UI library (which I did not check to see if you were loading), instead of html attribute titles where possible. It'll be a little easier to explain how, for example, the employment rates at graduation and 9 months are calculated. If you use just the basic HTML, it will be harder to make it readable because not all browsers recognize line-breaks in the title attribute. Moreover, even if you decide limited cross-browser support is fine, the decision of where to place the line-break is kind of a nuisance.
For more explanations and the U.S. News survey and employment rate formula, read more after the jump.
2010 U.S. News Survey
| Class of 2008 Placement (Employment) Rates | Number at Graduation | Number as of February 15, 2009 | |
|---|---|---|---|
| 157. | Graduates whose employment status is unknown: | ||
| 158. | Graduates whose employment status is known: | ||
| 159. | Graduates known to be employed: | ||
| 160. | Graduates who are enrolled in a full-time degree program: | ||
| 161. | Graduates who are unemployed and seeking work: | ||
| 162. | Graduates who are unemployed and not seeking work: | ||
| 163. | Total graduates from 9/1/2007 – 8/31/2008 | ||
Shockingly, schools' answers to questions #157 through #162 do not appear on U.S. News' website. Instead, U.S. News uses the answers to calculate its "Graduates employed at graduation" and "Graduates known to be employed nine months after graduation" employment rates. Neither employment rate reflects the percentage of the class employed at nine months as reported . It is a figure computed by a formula:
| Employment Rate = |
|
U.S. News uses the federal government's reasoning in configuring this formula. It aims to take those graduates who are not in the labor force out of the equation in calculating its rates. This is sound reasoning on its face, though the ABA was sufficiently concerned a few years ago with schools hiding unemployed graduates in the "not seeking" category, to the point where they opted to include "unemployed—not seeking," "unemployed—seeking," and "unemployed—studying for the bar" all as "unemployed." The ABA has since chosen to split "unemployed" back into the "seeking" and "not seeking" categories.
In closing, we expect that these changes will have a positive impact on the quality of information available to prospective students. We are also pleased that Bob Morse and his team were so open to making the information they collect available for public viewing. U.S. News has a long way to go before law schools will stop accusing them of having a negative impact on legal education, but one thing is certain: U.S. News does force more information about employment prospects than the ABA requires, and absent U.S. News prospectives would likely be even more in the dark. If law schools truly want to diminish the importance of U.S. News, they need to seriously consider the benefits that come with disclosing employment information directly to the consumer. In the meantime, LST will continue working with the ABA to give prospectives access to the information they need in order to make an informed decision.