The following post originally appeared on Forbes | February 18, 2015
Columbia Law School professor, Tim Wu, wrote a great piece in The New Yorker contrasting the Sundance Film Festival success of the documentary What Happened, Miss Simone? —a data-driven selection for production by Netflix—and the failure of the comedy The Bronze—a personal selection for production by Sundance’s director, John Cooper. When you throw this Netflix win into the ring with their numerous others—House of Cards is a personal favorite—it is difficult to dispute the effectiveness of Ted Sarandos’s data-driven selection model: Read the data, do what it says, and reap the rewards.
But not so fast. later in the festival, in a panel discussion entitled “How I Learned to Stop Worrying and Trust the Algorithm,” Sarandos relented: “In practice, it is probably a seventy-thirty mix. … Seventy is the data, and thirty is judgment.” He went on to say, “But the thirty needs to be on top, if that makes sense.”
It does, actually, as I have yet to find the major firm or institution that is solely following a data-driven model. At the same time, data analysis and technology are dramatically changing the way the world works, and certainly the way that law is practiced. Sanrandos’s experience, as it is mapped out in Wu’s article, mirrors, closely, BigLaw’s continued foray into data-driven decisions and the use of e-discovery and technology assisted review (TAR).
We are making huge leaps with technology—getting faster and more accurate—but we still can’t drop the human eye. Data continues to grow. Algorithms continue to get smarter; analysis, more accurate, and equipment continues to get faster. And as these four juggernauts gain critical mass and feed each other, the results compound. Keeping ahead of the game is a challenge.
Looking to better understand the impact of data and technology on the modern law firm and lawyer, today I speak with John Tredennick and Mark Noel of Catalyst Repository Systems, a cloud-based document repository for large-scale discovery and regulatory compliance. See our exchange below:
On What Catalyst Does
Parnell: If you could, talk to me about what your firm does, exactly.
Tredennick: In 2000, I started Catalyst because at the time, I saw that technology was changing the practice of law. Prior to Catalyst, I practiced for 25 years in a big law firm as a trial lawyer. I watched as the digital age came upon us. I watched as the age of the copiers came upon us—faster, slicker, cheaper copiers. I watched the discovery volume increase. It went from red wells to war rooms to data warehouses. Then the clock switched over. The print tools to manage these volumes—the printers and the staffers—weren’t working. Law firms would need tools: computer tools, electronic tools.
I believed in the cloud, even back then, so I sat down with guys to solve the problems in my firm. And in the mid-90s, we built systems that ran over the Internet and we deployed them. We began doing cases for others because they saw advantages. And the next thing you knew, I was a corner-office lawyer filling out tech RFPs and providing services even when we weren’t lawyering. So I convinced the firm to let me spin it off into a separate business, which was founded in 2000.
We were convinced that with the increasing volume of data—and I never could have imagined what we see today—and the fact that things were becoming more global, that law firms would have to reach out beyond their office boundaries and work together with other firms, and find ways to search and manage this data via the cloud so that attorneys can do the same things today that we always did, but do them differently, with technology.
On Law Firms Accepting E-Discovery And Technology Assisted Review
Parnell: Because some of the work you do takes away from the bottom of the leverage pyramid in a law firm, it may be bad news for at least a portion of the partner population. So talk to me about your experiences as they pertain to law firms. What has your experience been, by and large?
Noel: Are you familiar with the phrase, “the one percent problem?” The idea is related to data growth. We’re generating it in such greater volumes every year that even if you could use traditional culling tools to cut a collection down to just one percent of what you grabbed, it would still be too much for humans to review in a reasonable amount of time. That’s the kind of data that we’re up against. So traditional use of things like search terms to cull data down is not going to get it into a viewable realm where a human can put attorney eyes on only about 500 docs a day. You know? You collect 10 million docs; you cull it down to one million; you’re still going to be reviewing for five years if you don’t use some other types of tools or a very large army of reviewing attorneys.
So the advancement of technology assisted review tools is going to be a crucial part of the workflow. It also means that there is still going to be plenty of work for the attorneys to do because the volumes keep increasing. It’s an arms race to try to keep the amount of documents that you put human eyeballs on to a reasonable and doable level.
Many people are realizing that they have to change the way they work. And tools like technology assisted review are changing the way attorneys work. But it’s not going to replace them. TAR tools can quickly analyze millions of documents for subtle patterns, but only humans can decide what’s important to the case, or what stories the documents can tell. So these systems are hybrids: The machines do what they do best, and the humans do what they do best. There will be plenty of work to go around for skilled practitioners who know the tools and have the right skillsets.
On How The Times Have Changed
Parnell: Having worked in industry for decades, can you talk to me about some of the major changes you’ve witnessed over the years?
Tredennick: When I started the company, my whole dream was that someday we’d host a million documents—a million whole documents. At that time, a big case was 30,000 docs.
Fast forward from 2000 to 2015: Now we load, easily, millions of documents every night. We have hundreds of terabytes of hard drive specialized storage devices. And the big cases now start at three million docs. And for one client, it is hundreds of millions of docs. Change happens. And we’re just in that transition around technology assisted review.
Suddenly technology has come where you take a million documents in review—and for any big firm lawyer that’s a big smile on their face because with junior associates reviewing at 500 docs a day, you’ve got your year made—and somebody comes along and says, “You know, with a wave of a wand and a couple training docs, we’re going to cut that million documents down to about 50,000 docs that are probably important.” Maybe 95% of those billable hours go away. That does not make the lawyers smile. That does not make you smile.
But I’ve seen this for 30 years. The innovation comes out; the billable hour suffers; but you always have a few law firms that are not at the top, and then they say, “We don’t have 50,000 associates. Let’s go outsmart them. We’ll lead the way.” And they start taking business away from the big guys. And the corporate entities listen and change happens.
On Challenges Facing The Technology
Parnell: What are some of the challenges that you’re facing with your approach to technology assisted review? How are you overcoming them?
Noel: I think a lot of it is just education. Have you ever heard of the Dunning-Kruger effect? Dunning and Kruger are a couple of social scientists who were doing research on people’s self-assessment of their competency versus their actual competency. You know how everybody thinks that they are above average drivers? By definition, fifty percent are not going to be. But it turns out that there’s a very interesting distribution to these self-assessments. The people who are least competent at something tend to rate themselves better—above average—than people who have started to learn something about it. There actually is a point at which you’ve learned enough to recognize how much you don’t know.
This is something that I think we’ve just been doing naturally in legal education for a long time. Three years of law school doesn’t necessarily prepare anybody for actual practice. But it does make you competent enough in a lot of different areas that you can spot issues. So at least you know when you need to read more or to call in backup.
So I may be an intellectual property guy, but I know enough tax to recognize an issue and say, “Hey, we need to call somebody in to take a look at this.” That’s what’s required, now, of professionals in complex practice areas. But what we’ve found, though, is a lot of times people think that they know more about information or retrieval science and search and data management, and all these other types of e-discovery tasks, than they really do.
We’ve often assumed that we could—if we know the case, we know the data—come up with good search terms. But search terms are far from perfect. Some researchers named Blair and Maron came up with a study in 1985 that showed that experienced attorneys and reviewers thought they had gotten most of the documents out of a known set, when in fact, they had barely gotten 20%. And the last 30 years of research has replicated this result over and over again.
On The Changing Dynamics In Law
Parnell: How do you see the dynamics of today’s legal environment changing for attorneys? What do you think it will it be like for an attorney to practice in the next 10 years or so?
Noel: There used to be this whole idea that the doctor or the surgeon is sort of the “God” of the operating room, and everything revolved around him. Nobody can practice medicine like that anymore because the field is too complex. Now, in addition to a world that has a surgeon, you have to have an anesthesiologist that knows at least as much as the surgeon does. And then you also have to have a surgical nurse who knows at least as much as the other folks do. So you end up having to have a multi-disciplinary team to deal with the complexity of the practice arena. Law is getting there too.
So the best litigator in the world is not also going to be an expert at information retrieval science. You’re going to have to have partners and other people that you work with to help you handle that stuff, as well, which means that not only are you going to have to know enough to know when to bring those people in, but you’re also going to have to develop workflows and processes by which you can integrate these teams and have them work together effectively. And that’s what the practice is going to look like going forward, I think.
On Where Attorneys Should Focus Their Career Efforts
Parnell: If you were still a Big Law attorney, what would you be doing with your efforts? In light of all of this information, what would you do to future-proof yourself? Where would you focus your energies?
Noel: Getting back to the Dunning-Kruger effect, you don’t have to learn everything about every topic. But you have to learn enough to know when you need help. And it helps to line up trusted experts and trusted team members beforehand rather than trying to scramble around at the last minute. This is something that we’ve often recommended in cases, like Chinese language discovery, because every part of the process that happens in mainland China always seems to take longer than anyone expects. Even though nobody feels like they have time to vet vendor capability for on-site collection, in-country reviewing, processing, and tokenization of multiple multi-byte character formats, etc., you are really well served by lining up some trusted partners beforehand. Trying to find and vet people when you’re under the gun is just a recipe for pain. You have to be proactive about this.
On Where Law Firm Management Should Focus Their Efforts
Parnell: Okay. Let’s say you were a managing partner or a chairperson of a major firm. What are some of the things that you’re looking at to futureproof your firm? Where are you directing your strategy? Where are you putting your resources?
Noel: One of the things that I would do for professional development is to make sure my attorneys have at least an introduction to information retrieval science. They should realize that there’s a lot of complexity out there, and a lot of risk. If they go into a meeting or conference totally cold, they could paint themselves into a corner really, really quickly without even knowing it.
The other thing they should know is that there are many different types of tools out there, and that they can do different things. Often when I’m presenting on technology assisted review to people, I start off with a slide of algorithms. There’s support vector machines and k-nearest neighbor and latent semantic indexing—and eyes get wide.
Then I tell them they don’t need to know what all these do. All they need to know is that different flavors exist. Different tools and different algorithms have different strengths and weaknesses. Don’t try to learn it all; just put an expert on your team who can help you match the best tools to your case.
Email: [email protected] Twitter: @davidjparnell
Books: The Failing Law Firm: Symptoms And Remedies; In-House: A Lawyer’s Guide To Getting A Corporate Legal Position