Executive Summary

This report analyzes how a sample of stakeholders within the pro bono ecosystem and justice community think about and measure pro bono outcomes, and recommends the development of a strong, interoperable outcomes framework that is built on client-centered metrics. The methodology comprised interviews with 32 pro bono stakeholders within law firms and legal services organizations, a review of 22 Legal Services Corporation pro bono client impact stories, and alignment and feedback on existing frameworks from 4 justice measurement experts. Our conclusion is that a high-impact and actionable pro bono outcomes framework must: a) enable an understanding pro bono’s effectiveness and efficiency; b) minimize the burden of outcome reporting for clients; and c) take a client-centric approach to outcome measurement that focuses on client and procedural outcomes.

While these findings are preliminary, we are thrilled to have achieved our goal of soliciting feedback for a workable framework centered on pro bono outcomes on which we can build.

Introduction and Objective

Every year, millions of hours of legal and professional services are donated to serve indigent clients. Data about these pro bono hours — including who donates them, to whom, supported by what infrastructure and funding — are tracked for various purposes by governing bodies, academics, legal services organizations, and law firms. Almost as varied are the approaches and frameworks used to collect these data, with little interoperability between them and little insight into the outcomes of these hours and the impact they have on pro bono clients’ lives and in their communities.

We see this as a missed opportunity for many reasons. Without data interoperability and reporting on client outcomes, we lose the ability to glean a data-informed narrative of pro bono, and assess and articulate its access-to-justice impact more broadly. Is pro bono worthwhile to undertake? Under what circumstances does it best serve clients’ needs? When is it most effective and efficient? How can we compare pro bono work to other interventions in our society? How do existing client impact stories reflect what clients see as important pro bono outcomes, compared to how those providing legal services measure success?

From Paladin’s perspective, as the volume of pro bono work flowing through our platform grows, the natural next step is to measure the real-world impact of these services. In doing so, we’ll contribute to solving the “data problem” in justice related to knowing whether and how individuals in need are able to access justice. Ultimately, we seek to better understand Paladin’s effectiveness in improving access to justice, as well as to help our partners and the ecosystem at large better measure the impact of pro bono legal services.

Thus, we are embarking on a multi-phase Pro Bono Outcomes project to address this gap.

The first phase of this project has the following objective:

To solicit feedback for a workable framework centered on measuring pro bono client outcomes.

In later phases, we will leverage this feedback from the pro bono and justice community into developing a workable framework that is not only useful to the ecosystem, but is also integrated into Paladin’s product to measure outcomes from the pro bono work enabled through the platform.

Why is this important at scale? If the pro bono ecosystem were to implement such a framework, this could pave the way for:

  • Improved Pro Bono Delivery and Outcomes: data around client impact trends could be leveraged to help evolve pro bono delivery models and improve pro bono client outcomes;
  • Value to Pro Bono Stakeholders: access to this data could help legal service organizations (LSOs) and law firms better understand and tailor the resourcing of their programs; and
  • Value to the Justice Ecosystem: this data could help validate the impact of pro bono legal services within the broader justice community and contribute to broader efforts to measure access to justice impact.

We know that a mutually agreed upon framework is an ambitious goal, and that whatever approach on which we collectively land will be only a first step in many ways. However, we believe it is important to start an interdisciplinary dialogue on the value of structuring such a framework, and the criteria to consider, so that a foundation may be laid to support what we believe is the ecosystem’s shared goal: to directly increase access to justice for individuals and communities.

Key Findings

There were four main takeaways from our pro bono outcomes research:

1. General Alignment on the Purpose of Pro Bono and Importance of Measuring Outcomes

Almost all pro bono practitioners interviewed agreed that the primary purpose of pro bono is to increase legal aid’s capacity to serve individuals in need.

However, when asked about the importance of outcome data to measure this access to justice impact, two thirds of the pro bono leader participants ranked having outcome data as “very important,” while the other third ranked it as “somewhat important.”

2. Interest in Measuring the Effectiveness and Efficiency of Pro Bono Legal Services, Specifically

There are currently limited systematic efforts to measure the effectiveness and efficiency of pro bono legal services in solving clients’ legal issues, but there was a significant interest from stakeholders to study those criteria in particular. Most participants responded they are “somewhat confident” or “confident” that pro bono legal services are an effective and efficient mechanism to narrow the justice gap. Legal services organizations were more eager than law firm participants to seek confirmation of value within their own programs.

3. Desire to Collaborate and Data-Share to Minimize the Burden on Clients

It was paramount to pro bono leaders that any efforts to collect outcome data be as low a burden as possible on pro bono clients. To achieve this, participants endorsed data-sharing whenever possible (with the client’s consent), as well as leveraging data collected at naturally convenient points along the client journey, such as at intake and case closure. We see this willingness and eagerness to share as an opportunity to leverage technology in a way that encourages systems (and people) to communicate more easily, while staying mindful and respectful of pro bono clients’ privacy. Participants also noted that for data collection to be effective, efforts should also not impose unrealistic burdens on the attorneys, staff, and administrators who provide pro bono services.

4. Alignment on Prioritizing Immediate Client Outcomes and Procedural Justice Metrics (via a Client-Centered Approach)

Pro bono stakeholders converged on the importance of adopting client-centered metrics, including those that could be gleaned from such prompts as:

  • Did the client resolve his/her/their legal problem?
  • What is the case type?
  • How did the case resolve?
  • Did the client feel more empowered?
  • Did the pro bono lawyer treat the client with respect?

There remain significant questions on precisely the format that these questions and results should take, as well as whether these dimensions of information are sufficiently exhaustive. For example, we acknowledge that these questions do not dive deeply into whether a client’s desired goals were achieved (broader than the question of whether the client’s legal problem was resolved), and how that might compare to a provider’s view of ‘success’ in his/her/their case. As a first step, however, the summit indicated the lens through which these outcomes might be viewed.

Methodology

Because our objective for this phase was to solicit feedback for a workable framework, community and stakeholder involvement was paramount. To that end, we included 32 stakeholders from across the pro bono ecosystem in a three-day Summit. Participants included Pro Bono Counsel/Partners at leading law firms, Executive Directors and Pro Bono Coordinators at LSOs in the United States, as well as global and domestic justice data experts from a range of leading organizations. We also reviewed pro bono client feedback and testimonials from the Legal Services Corporation to inform our research and to structure our Summit activities.

The Summit focused on five areas:

  1. Understanding the Value of Outcomes to Stakeholders
  2. Exploring Existing Frameworks
  3. Adopting Outcomes for Pro Bono
  4. Validating Proposed Framework Questions with Stakeholders
  5. Conducting Supplemental Interviews

Understanding the Value to Stakeholders

Paladin is keenly interested in the objective of the Summit; however, we want to make sure our work is in service to the ecosystem. To that end, we also wanted to validate our hypothesis that collecting outcome data would, in fact, provide value to key stakeholders who work in and across pro bono legal services.

As the first part of our three-day Summit, we convened 17 pro bono and justice leaders to discuss if and why they cared about measuring pro bono outcomes.

Existing Framework Exploration

We asked four experts to present existing frameworks and approaches being used in the broader access to justice or legal services context to understand existing research. Our goals were to learn from these approaches, as well as ultimately co-develop a framework suited to the pro bono ecosystem that would be consistent with and interoperable with other justice data initiatives. The frameworks we examined included:

Adopting Outcomes for Pro Bono

With participants familiarized with the above justice, sustainability, and legal frameworks, we wanted to understand what attributes of those frameworks might be usefully applied to the pro bono ecosystem more specifically. We asked participants: “What are the most important attributes of a pro bono outcomes framework?”

To facilitate the discussion, we consolidated over one hundred pieces of feedback collected from participants during the Summit and asked them to stack-rank various attributes of an outcomes framework. From there, we discussed the highest rated attributes as a group, including why each attribute was important, and what benefits and drawbacks might be associated with each as described in the synthesis section below.

Validating Proposed Framework Questions with Stakeholders

Since the output of session two was directional feedback on desired outcomes framework attributes, the third session aimed to translate that feedback into draft components of an outcomes framework, having those attributes.

To validate the components and inform a next direction, we hosted a series of individual feedback sessions with pro bono stakeholders in which they filled out an online survey about attribute priorities and explained their reasoning aloud to a facilitator.

Conducting Supplemental Interviews and Reviewing Client Impact Stories

Finally, to validate the findings of our Summit, we solicited written feedback from 15 other pro bono leaders at law firms and legal services organizations (LSOs) to whom we asked a standard set of questions. We also reviewed 22 randomized LSC client impact stories to identify central themes that were important to pro bono clients. These supplemental written interviews and stories form part of the below synthesis, but we want to highlight a few pro bono client commonalities that align with the pro bono stakeholders’ feedback:

First, all 22 clients cited their literal case outcome as the end result of their pro bono assistance, seemingly the easiest and most straightforward way for them to measure legal services organizations’ effectiveness. Twelve clients shared how winning their pro bono cases benefitted them monetarily, like Daphne, whose veteran husband Ed was suffering from Alzheimer’s and who was going into debt: “NLS was able to have the couple’s debts discharged, which provided Daphne much needed peace of mind and a fresh start. She has often told NLS that they were her saving grace; without their assistance during those difficult years, she could not have been afforded the precious time she had left with Ed.”

Twelve respondents also shared that a major outcome of their pro bono experience was that they had better mental health and longitudinal outlook after their case(s) closed, like Mary in Hawaii. “I felt like I had hope…. due to Legal Aid’s help, I was able to focus on my children, my work, and I was able to sleep at night,” she wrote. Lastly, three clients noted how pro bono helped achieve procedural justice, like Mrs. Johnson in Cleveland, who explained, “I was getting bullied in that meeting. I knew they would respect an attorney, but we didn’t have money for that.” Overall, the clients were focused on achieving substantive outcomes: solving the legal issue at hand, salvaging what they could of their livelihoods, and easing the stress of their situations.

One last note on methodology: this analysis is designed to launch a dialogue on measuring pro bono outcomes. While we believe our findings to be informative, given the modest sample size of 54 pro bono stakeholders and clients whose feedback we incorporated, they may not reflect the entirety of the pro bono ecosystem. Furthermore, the justice data ecosystem is evolving in real-time, and therefore, we hope to incorporate best practices as they emerge in hopes of contributing to the creation of interoperable data practices for pro bono.

Synthesis

Question 1: Why Measure Pro Bono Outcomes?

Aligning on stakeholders’ desire for knowing pro bono outcomes is an essential step in homing in on how to collectively measure relevant metrics. In the global context, this alignment also helps us understand how we can best achieve Sustainable Development Goal 16’s global commitment to “provide access to justice for all” by 2030.

At the highest level, almost all participants agreed that the primary purpose of pro bono is to support legal aid’s ability to bridge the justice gap faced by millions of low-income people. Serving clients in need was prioritized over other goals such as lawyer development, law firm business cases, or LSO fundraising considerations.

While this may seem obvious, the group was also immediately in agreement that the current practice of collecting pro bono attorney hour and participation data, which comprises the majority of data collected today, does not meaningfully measure whether pro bono is enabling access to justice for low-income individuals. First, the current approach is an attorney-centric and not a pro bono client-centric metric. Second, participants agreed that outputs (like hours and percent of attorney participation) do not necessarily cause or even correlate with positive client outcomes.

When asked about the importance of outcome data, two thirds of participants ranked having outcome data as “very important” to them in their roles, while the other third ranked outcome data as “somewhat important.”

This level of interest was lower than hypothesized going into the Summit based on anecdotal data, but we felt it was still high enough to justify moving forward with the project. While we are interested in diving deeper into specific outcomes data use cases, we realize that there is currently limited incentive for individual organizations to invest in measuring outcomes and, in some cases, there may even be negative incentives. This level of interest may also be related to the fact that to date, there has been little capacity to measure pro bono outcomes and thus, little experience in working with the data that might become available if we can introduce such a framework.

The majority of participants agreed that the following were the most important reasons to leverage impact data in pro bono:

  1. To measure the effectiveness of pro bono legal services in solving the client’s legal issues;
  2. To measure the efficiency of pro bono legal services in solving the client’s legal issues; and
  3. To attract more resources (financial, volunteers, internal support) for their pro bono program or LSO.

Most pro bono stakeholders felt strongly that data on whether the pro bono legal service was ultimately effective in solving the client’s legal issue was the most important to collect. This aligns well with the global justice community’s push toward people-centered data. However, it’s worth noting that effectiveness could encompass a broad range of information ranging from case outcomes (e.g., “contested divorce granted”), to broader societal outcomes to procedural justice and client experience outcomes, to data centered on the achievement of client-defined goals, as discussed further below.

Efficiency was a particularly interesting metric in the pro bono sphere. Most LSO stakeholder participants agreed that legal aid staff attorneys are more efficient in terms of the number of hours spent handling the same matter as a pro bono attorney. But if the question we are trying to answer is instead “does pro bono increase the capacity of legal aid to serve more low-income individuals,” then the definition of efficiency changes. Several participants expressed a desire to measure whether it is more efficient for LSOs to refer matters for pro bono vs handle in-house, thereby increasing their capacity to serve more clients with the same resource inputs.

To dig into the need for such data, we asked the group, “How confident are you that pro bono services are efficient and effective means of narrowing the justice gap?” While we saw a range of responses from “not at all confident” to “very confident,” the vast majority of participants responded, “somewhat confident” or “confident.”

Lastly, the respondents that prioritized attracting resources were less interested in measuring effectiveness and efficiency. More inquiry is needed into whether their underlying assumption is that effectiveness and efficiency are less important, or — in line with the above — whether respondents felt strong doubt or confidence in the efficacy or efficiency of pro bono such that they do not see the need for additional data confirmation.

Question 2: How Might We Adopt Best Practices from Existing Frameworks into a Pro Bono Outcomes Framework?

One of the most pressing challenges in access to justice overall right now is the staggering lack of standardized data. For instance, this recent American Academy of Arts & Science’s report, in calling for substantial expansion of justice data collection, outlines how the lack of data presently prevents knowledge on whether low-income individuals are adequately helped by the legal services they receive. Such reports are combined with growing calls for the development of an interoperable justice data commons, efforts to consolidate tracking outcomes, and an international effort to align progress under proposed SDG indicator 16.3.3.

Without systemic data, it’s impossible to measure progress against our above goals to deliver:

  • Improved Pro Bono Deliver and Outcomes;
  • Value to Pro Bono Stakeholders;
  • Value to the Justice Ecosystem.

With this in mind, we wanted to leverage existing approaches to measuring justice impact, since there is no need to reinvent the wheel for pro bono. We were honored to hear from leading justice experts working with frameworks ranging from catalogues like LSC’s Case Service Reporting to quantitative approaches like Benefit-Cost Ratios, to the World Justice Project’s focus on people-centered justice data, to NCAJ’s Guide to Tracking Outcomes.

Very few of the pro bono stakeholders were familiar with the frameworks and approaches presented, which may point to the often-siloed nature of pro bono work within the broader access to justice ecosystem. At minimum, it indicates a clear opportunity to better link pro bono data collection efforts to current best practices in justice data.

In light of the above, we wanted to develop foundational familiarity with the above frameworks while gathering written feedback concurrently from participants. This feedback was ultimately incorporated into a discussion to answer Question 3, below.

Question 3: How Do Pro Bono Stakeholders Prioritize Framework Attributes and Questions?

Following the above discussion, we distilled twelve attributes discussed by the participants of a potential pro bono outcomes framework for the participants to rank in order of importance. The attributes were as follows:

The bolded top six attributes were ranked as the most important attributes by the pro bono stakeholders. We’ll discuss each in turn.

Far and away the Summit attendees’ most important criterion was that the outcomes collection is not burdensome on clients. To participants, this meant minimizing the time and effort clients spent sharing outcomes with interested parties and lowering the emotional weight of conveying their experiences, while also giving them the space to reflect and air any thoughts or feedback.

Stakeholders felt strongly that any framework should leverage as much information from other sources (e.g., the pro bono attorney) as possible, and encouraged leveraging and sharing data collected at intake or other key client journey moments to minimize duplication of efforts. Participants noted that client questions should be framed in a clear and easy-to-answer way to reduce ambiguity and low or slow response rates, and reflected that immediacy in collecting feedback can also reduce burden of collection (versus attempts to collect long term outcomes).

The second leading attribute, echoed in LSC’s client impact stories, was that pro bono outcome data reflect the immediate client outcome. Within this category, stakeholders noted that there is a strong tension between capturing granular, categorizable client outcome data (e.g., in the form of an outcome catalog), which can quickly become overwhelming, versus higher-order data that is interoperable between case types (e.g., did the client achieve his/her/their desired legal outcome), but is less specific.

Third, stakeholders felt that capturing some form of procedural justice or client experience data was a priority. This was particularly true because several LSO participants mentioned anecdotally that some pro bono volunteers may not have the requisite background to handle certain types of cases (e.g., those involving trauma), so measuring the client experience is paramount. That said, other stakeholders found procedural justice to be too imprecise or vague a requirement to measure in isolation (especially in isolation from the measure of whether the client achieved his/her/their desired legal outcome) and encouraged any procedural justice measurements to be paired with other metrics.

Fourth, participants felt that it was important for outcomes measurements to help assess the value of pro bono legal services as a tool to increase access to justice for individuals and the larger community. This aligns well with the findings under Question 1, as well as the broader global efforts to measure access to justice under SDG 16. From the LSO perspective, such data would also help them assess whether pro bono client outcomes were as effective as staff attorney outcomes, and for those who receive no legal assistance at all, as well as to allow for comparison (taking into account other factors as well) of outcomes across their law firm pro bono partners.

Fifth, participants were cautiously interested in collecting quantifiable data. While several stakeholders expressed concern at reducing client outcomes to a number, especially if such a number was a dollar figure, there was general agreement that legal aid funders and law firm stakeholders would appreciate quantifiable data. Respondents noted that numbers can also be a powerful marketing tool and could be leveraged to bring more resources to access to justice. On the other hand, participants felt that while SROI calculations have an appearance of reliability, data accuracy is a major concern. The overhead required to develop an accurate SROI calculation would be overwhelming because it would likely require cataloging and assigning values to each possible case outcome across a range of communities, geographies, and other variables. The importance of considering any number in the context of other factors that influence outcome– such as the area of law, screening of intake for merit, relative fairness of different decision-makers — was also underlined.

Sixth, the participants rated data accuracy as a top attribute that any pro bono outcomes framework should possess. We agree this is important and acknowledge it may conflict with some of the other attributes such as the desire to collect quantifiable data, or generalizable data that allows for apples-to-apples comparisons.

The group also evaluated the importance of criteria like long-term impact, pro bono causation, and granular outcomes catalogs that include a confluence of hyper local factors, all of which were deemed likely too complex to evaluate and capture at this time. Other metrics, like interoperability with the broader global justice community and contributions to other social services, are valuable but did not rise to the same level of importance for our participants. Lastly, while there is an interest in measuring attorney- and law firm-focused outcomes (i.e., the “business case” for pro bono), those were not shared by the broader LSO and justice stakeholders who favored focusing on clients.

In session three, we solicited feedback from pro bono stakeholders on the relative importance of specific questions that we extrapolated from the above attributes. The primary purpose of this exercise was to further probe the specific types of data that would be valuable to stakeholders in a less abstract format than “attributes.” A secondary purpose was to begin to understand the specific wording and data structure for outcomes components that participants would find useful in practice.

Below are the questions, ranked by participants in order of perceived importance from most important to least:

Overall, participants ranked the client-centric metric of “did the client resolve their legal problem” at the top of the list, which aligns well with approaches taken both at the global and national level to embrace human-centered outcomes.

Second, stakeholders ranked “how did the case resolve,” which we envisioned tracking to the attribute of immediate client outcome, as second most important. Case type followed as the third most important, but stakeholders differed on the level of detail necessary, as well as the case type systems and frameworks that they employ. This would likely make adoption of a unified system or, alternatively, interoperability between systems, a challenge.

Fourth and fifth on the list, respondents ranked procedural justice elements as quite important to measure. There is currently limited procedural justice data being systematically collected in the pro bono sphere, so this is a key area for growth that is well aligned with a client-centered approach.

Sixth, stakeholders were interested in capturing the financial benefits of any pro bono decisions, perhaps as a way to assign a widely understood value to a case outcome without needing to navigate the complexities of an SROI calculation.

Lastly, respondents were interested in client demographic and community-related data to understand how justice might be accessed or applied to varying groups.

While these questions and the exercise of ranking them formed a valuable foundation for the development of a workable framework to measure pro bono client outcomes, we recognize that this analysis is far from exhaustive. More research and development are needed to solidify these questions and insights into a workable framework.

Question 4: What Do Other Stakeholders Think?

In addition to the stakeholders included in the Summit, we reached out to other law firm and LSO pro bono leaders via email to ask them four questions:

  1. How do you currently measure pro bono outcomes, and how did you decide on that framework?
  2. If you could capture anything related to pro bono outcomes, what would it be?
  3. Considering existing frameworks like LSC’s Case Service Reporting, UN SDG 16, SROI, etc., what do you see as most relevant/interesting to pro bono reporting?
  4. How would your program benefit if you could capture outcomes?

Out of 15 responses, only one reported that they currently capture pro bono outcomes. That respondent noted that the law firm has developed its own internal catalogue to capture data around cases ‘wins,’ motions ‘granted,’ and cases ‘settled favorably.’

In addition, only one other firm was familiar with any of the existing frameworks we cited, indicating a significant opportunity to educate and build upon rich existing research.

With respect to desired outcome metrics, the stakeholders responded they ideally, they would want to know (freeform):

  • How the case was resolved (7)
  • Whether the client achieved his/her/their desired outcome (3)
  • Attorney skills built (3)
  • Data around local need/placement trends (2)
  • Lawyer satisfaction and encouragement (2)
  • Longitudinal impact of outcome (2)
  • Client experience (2)
  • Financial reward (1)
  • Whether more resources were allocated for pro bono work (1)

There was consensus that knowing how a case was resolved and whether the client achieved his/her/their desired outcome were the top two requirements for an outcomes framework.

In addition, respondents noted that having this information available would help them: a) strengthen the internal business case for pro bono; b) encourage more lawyers to get involved by highlighting the personal and professional impact of pro bono work; and c) improve the delivery of pro bono services.

While the written respondents highlighted an interest in both client- and lawyer-related metrics to determine pro bono’s success, we continue to focus on the client-centered approach for this project.

Conclusion

We are thrilled to have met our original goal of the Summit to “solicit feedback for a workable framework centered on measuring pro bono client outcomes.” We look forward to leveraging this input from the pro bono and access to justice community into a workable framework that is not only useful to the ecosystem, but also integrated into Paladin’s product to systematically measure the client outcomes of the work that happens through the platform.

As mentioned at the outset, the long-term goal of implementing such a framework will pave the way for:

  • Improved Pro Bono Delivery and Outcomes: data around client impact trends could be leveraged to help evolve pro bono delivery models and improve pro bono client outcomes;
  • Value to Pro Bono Stakeholders: access to this data could help legal service organizations (LSOs) and law firms better understand and tailor the impact of their programs; and
  • Value to the Justice Ecosystem: this data could help validate the justice impact of pro bono legal services within the broader community and contribute to broader efforts to measure access to justice impact.

While we are still far away from achieving this long-term goal, we are delighted to contribute to an interdisciplinary dialogue on the value of structuring such a framework, and the criteria to consider.

As a next step, we will develop a framework that has the desired attributes and meets the identified needs and test ways to gather, implement, and visualize this data in the most impactful way.

Lastly (but very much not least), we are grateful to pro bono leaders at Bradley, Foley Hoag, Katten Muchin, Legal Aid Chicago, Manatt, Mintz Levin, among others, and experts in the space like Katherine Alteneder, Carlos Manjarrez, and Julia Wilson, for their insight and contributions to this report. For any thoughts or additional feedback, don’t hesitate to reach out at [email protected].

Kristen Sonday Pic Headshot

Kristen Sonday

Kristen is the Co-Founder and CEO at Paladin. As a first generation-college Princeton graduate, Kristen first witnessed how complicated our judicial system is to navigate while at the U.S. Department of Justice conducting international criminal work in Mexico and Central America. As one of the few Latinos on the team, she saw the immense value of having an advocate with you throughout the legal process, which inspired her to want to build something to increase access for those in diverse communities. After DOJ, she joined the Founding Team of YC-backed Grouper, where she learned how to build a startup from the ground up. In addition to Paladin, Kristen Co-Chairs the Legal Services Corporation’s Emerging Leaders Council and is a partner at LongJump, investing in overlooked founders in the Chicago area.

More from Kristen
Stacy Zinken Bio

stacy

More from