Transparency Index Methodology

Methodology

From December 18, 2012, through January 9, 2013, the LST team analyzed every ABA-approved law school website. We devised the Transparency Index Questionnaire ("TIQ") with 10 transparency criteria, 8 common Standard 509 problems, and a catch-all column for other Standard 509 problems (together the 19 "TIQ Criteria"). Full explanations of the TIQ Criteria are available below.

We determined the TIQ Criteria by contemplating what consumer information we believe matters to a prospective law student looking to invest three years and a lot of money in a professional degree. How did graduates find jobs? When did they find the jobs? If employed, are they still seeking another job? What compensation did graduates of various categories earn? As to fresh graduates, these are not difficult questions for law schools to answer. Every school collects the data necessary to answer these questions, and making them public requires minimal effort.

To collect data, we navigated school websites actively searching for consumer information. Not only did we explore websites following their layouts as a prospective student might, but we used our experience with these sites to target information that might have been hidden by poor website designs. To get to that information, we used a school's internal search feature and Google with key terms. Our navigation process was particularly thorough when we keyed a school as missing one of the two required Standard 509 charts.

The product of our data collection is an index that sheds light on how well (or poorly) law schools provide access to consumer information, including job outcome data and scholarship retention data. Like the previous year, we maintained a "Live Transparency Index" on our website to document law school disclosure improvements. One benefit of the internet is the fluidity of websites, so schools that recognize disclosure shortcomings have no excuse not to make changes quickly once they are on notice, and we can update our website to recognize positive behavior.

Note that LST's Transparency Index is not a ranking system. It would not be very meaningful to rank a school by the number of criteria met or problems found because different criteria vary in importance, and the severity of different Standard 509 problems fall on a spectrum. Nevertheless, the aggregate statistics for the TIQ Criteria do provide useful metrics for showcasing the challenges prospective students face when they scour the internet for answers to their basic consumer questions.

Finally, once we collected all of the data from website reviews, we contacted the dean, career services office, and admissions office of the 199 ABA-approved law schools we analyzed. Included in the memo were explanations of the Standard 509 requirements, common problems, and what we expect law schools to disclose as a matter of practice. The purpose of putting schools on notice was to help them meeting the Standard 509 requirements and to facilitate transparency.

TIQ Criteria

Minimum ABA Requirements (Standard 509)

To remain an ABA-approved (i.e. accredited) law school, schools must meet every ABA standard, including the consumer protection standard, Standard 509. The ABA updated Standard 509 on August 6, 2012 to clarify the kind of advertising that was prohibited. According to Standard 509(a), "[c]onsumer information ... shall be complete, accurate, and not misleading." The standards also require law schools to publish two charts, one on employment outcomes and the other on conditional scholarship retention.

Required Charts

Schools must publish on their websites an employment outcome chart in the form designated by the Council of the Section of Legal Education. Schools may publish a chart in a modified format provided that it includes all data on the form designated by the Council and provided that readers can easily compare the data to other schools. Schools must publish on their websites a conditional scholarship retention chart in the form designated by the Council of the Section of Legal Education. Schools may publish a chart in a modified format provided that it includes all data on the form designated by the Council and provided that readers can easily compare the data to other schools. As of August 6, 2012, ABA-approved law schools must have these two charts published on their websites.

Misleading, incomplete, and inaccurate consumer information

No Misleading Rate

A school provides a misleading employment rate if it states that x percent are employed without any context, such as other data or a note that the rate includes part-time, short-term, or non-legal jobs. The basic employment rate (i.e. any job had by a graduate, regardless of its characteristics, counts towards the numerator) in isolation is incomplete and likely misleading.

No Undisclosed Methodology

A school that states an employment rate, using a rate other than total employed graduates divided by total graduates, without explicitly defining the methodology or providing the underlying data in close proximity to the rate, provides incomplete and likely misleading information. Example: If a law school includes all employed graduates and all graduates pursuing an additional advanced degree in the numerator and excludes graduates with an unknown employment status and graduates who are unemployed but not seeking work, without making this formula clear.

No Incomplete Salary Information

Certain types of salary information are incomplete, which can often (but not necessarily) be misleading. For example, a mean or median salary without additional context is incomplete because entry-level lawyer salaries follow a bi-modal distribution. Therefore, mean salary must also be accompanied by salary quartiles, just as a median salary must be accompanied by the 25th and 75th percentiles. A salary range is not sufficient context for completeness; if a school says its average salary is $80,000 with a range of $20,000 to $160,000, the information is incomplete despite the added context.

No Missing Response Rates on Advertised Salary Information (Int. 509-3)

Salary information published or distributed “must clearly identify the number of salaries and the percentage of graduates included.” A school must clearly identify the numerator and the denominator to avoid a problem.

No Class of 2011 Employment Data

Separate from the requirement that schools post the Standard 509(d) employment outcome chart, providing outdated information only is an incomplete portrayal of employment outcomes. If the outdated information substantially differs from the most up-to-date information, it may be misleading too.

No Misleading Employer Lists

A list of the employers that hired a schools' graduates must representative based on available data.

No Other Standard 509 Problems

Examples: (i) Using the ABA logo without presenting the required chart, (ii) publishing summer salaries without indicating the response rates or statistical context, (iii) using only blended employment rates from multiple years without showing the disaggregated rates, (iv) using incorrect terminology that the typical reader would not notice, and (v) creating a default rule that all unknown job characteristics must be what the most common job characteristic is.

Transparency Criteria

Law schools are capable of being transparent right now about Class of 2011 job outcomes. If a school claims to value transparency, see if they meet the very basic norms reflected by this section of the Index.

NALP Report

Each summer, NALP provides the NALP Report to each member law school. The report contains valuable granular employment data. A school can meet all ten transparency criteria by making its NALP Report available on its website. However, if the school provided its NALP Report to LST, it still meets the criterion.

Job Characteristics

Credit requires a complete accounting for the relevant category. An accounting is complete if it is clear that the data account for all employed graduates. This requires the exact number (n) or exact percentage of ALL employed graduates. It is not a problem if a school that did not collect these job characteristics data for all employed graduates, but in order to be complete the school must somehow indicate the number of unknowns. For example, a pie chart for job offer timing is not complete if it has just three slices (before graduation, before bar results, and after bar results) with percentages that add up to 100%. The school must somehow indicate the response rate of employed graduates as it pertains to this job characteristic. These data about employed graduates' job characteristics are all available in the school's NALP Report.

Job Source

How did the employed graduate find their job? Some options are on-campus interviews, direct mailing, and returning to a previous employer.

Employed, Still Seeking

Is a graduate still seeking a new job despite the job documented by the law school? The options are yes, no, and unknown.

Job Offer Timing

When did the employed graduate obtain their job offer? The options are before graduation, before bar results, after bar results, and unknown.

Employer Type, Breakdown by Credentials

What kinds of jobs (e.g. Bar Passage Required) did employed graduates find as a subset of employer type (e.g. Business or Law Firm)? For example, how many graduates employed in Business were in positions that required bar passage?

Salary Information

Schools receive credit only if the information is not known to be incomplete, inaccurate, or misleading. Salary information is complete when it includes the 25th, 50th, and 75th percentiles for each category of information provided. It must also indicate the number of salary data points included, as well as the response rate, for each category. These data about how much graduates earn are all available in the school's NALP Report.

All Employed

Salary information for all employed graduates.

Sector

Salary information for both the public and private sectors.

Employer Type

Salary information by the type of employer, e.g. Business or Law Firm.

Law Firm Size

Salary information by the size of the law firm.

Credentials

Salary information by the type of credentials required for the job, e.g. Bar Passage Required or J.D. Advantage.