Close
Enter your
text here
Posts By :

Jason Krause

eDiscovery Trends: Lawyers Versus Machines – Who’s “Winning”?
 

As discussed on this blog, mainstream publications including The New York Times and Forbes have noticed the rise of search technology in discovery, particularly predictive coding. The New York Times article, Armies of Expensive Lawyers, Replaced by Cheaper Software, inspired a lot of consternation in the legal community by proposing that technology was replacing human lawyers. Among the first to reply, Ralph Losey wrote a blog post New York Times Discovers eDiscovery, But Gets the Jobs Report  Wrong, arguing that “the supposed job-chilling impact of these new technologies on the legal profession was off the mark. In fact, the contrary is true.”

However, the Times article does point to a real trend – clients demanding that their outside counsel and litigation support teams use technology to work more efficiently. “Just because the “paper of record” says something doesn’t make it so, of course. But it does mean that every GC and Litigation DGC/AGC in America (and likely Canada) now has this trend on their radar,” litigation project management guru Steven Levy wrote on the blog Lexican.

The obvious problem with the New York Times article is that search and review is an iterative process and demands human intervention to make the machines involved function properly.  However, the missing piece of the discussion today is exactly what the relation between human reviewers and computers should be. There is a nascent movement to investigate this topic, finding the line where machine-led review ends and where human intervention is necessary.

Recent research by some of the leaders of the TREC Legal Track research project has begun to explore the interaction between human and machine review. Maura Grossman, a litigator with Wachtell, Lipton, Rosen & Katz and one of the TREC coordinators, and Gordon Cormack, a computer scientist and fellow TREC-er, wrote the research paper Technology Assisted Review in eDiscovery Can be More Effective and Efficient Than Manual Review. As the title indicates, human review cannot match the accuracy of technology-assisted review. However, the paper points out the need for a roadmap detailing the ideal interaction between human lawyers and machine review in litigation. “A technology-assisted review process involves the interplay of humans and computers to identify the documents in a collection that are responsive to a production request, or to identify those documents that should be withheld on the basis of privilege.”

What may be endangered is the existing review process, as it has traditionally been practiced, not human attorneys. Bennett Borden, an attorney with Williams Mullin, argues the linear review processes cannot produce the same results as the skillful use of technology. He has some interesting asides about the ways lawyers can do things computer searches cannot. For example, human reviewers are able to intuitively “come upon a scent” of relevant documents that machines missed. He says that reviewers not only are able to effectively pursue information by following leads initiated by a computer, but they actually enjoyed the process more than straight-ahead manual review.

Clearly, more research is needed in this area, but if lawyers are going to defend their role in litigation, defining the role of lawyers in discovery is an important question. What do you think?  Please share any comments you might have or if you'd like to know more about a particular topic.

eDiscovery Standards: How Does an Industry Get Them?
 

As discussed yesterday, there is a nascent, but growing, movement pushing for industry standards in eDiscovery. That’s something many litigators may chafe at, thinking that standards and industry benchmarks impose checklists or management processes that tell them how to do their job. But industry standards, when implemented well, provide not only a common standard of care, but can help provide a point of comparison to help drive buying decisions.

It’s probably understandable that many of the calls for standards today focus on the search process. Judge Shira Scheindlin wrote in Pension Committee of the University of Montreal Pension Plan v. Banc of America Securities, LLC that a party’s “failure to assess the accuracy and validity of selected search terms” was tantamount to negligence.  As mentioned yesterday, the Text Retrieval Conference TREC Legal Track has been benchmarking different search strategies, even finding ways to optimize the search process. The ultimate goal is to provide baseline standards and guidelines to allow parties to determine if they are being successful in searching electronically stored information in litigation.

Within these technical discussions a new emerging thread is a call for ethical standards and codes of conduct. Jason Baron, National Archives' Director of Litigation and one of the coordinators of the TREC Legal Track, organized the SIRE workshop that concluded last week, focused on information retrieval issues in large data sets. However, even he, who has been working on optimizing search technology, recognizes the need for standards of care and ethics in eDiscovery to manage the human element. In a paper released earlier this year, he noted, “While there are no reported cases discussing the matter of ‘keyword search ethics,’ it is only a matter of time before courts are faced with deciding difficult issues regarding the duty of responding parties and their counsel to make adequate disclosures.”

The leading provider of industry standards is the Electronic Discovery Resource Model (EDRM), which has a number of projects and efforts underway to create common frameworks and standards for managing eDiscovery. Many of the EDRM’s ongoing projects are aimed at creating a framework, and not standards. In addition to the EDRM Framework familiar to many eDiscovery professionals, the group has produced an EDRM Model Code of Conduct Project to issue aspiring eDiscovery ethics guidelines and is working on a model Search Project.

But biggest piece of the discussion is how to create benchmarks and standards for repeatable, defensible, and consistent business processes through the entire eDiscovery process. There are no current quality standards for eDiscovery, but there are several models that could be adopted. For example, the ISO 9000 quality management system defines industry-specific quality standards and could be tailored to eDiscovery. The Capability Maturity Model Integration (CMMI) in software engineering follows a similar model, but unlike ISO, does not require annual updates for certification.

This is still a nascent movement, characterized more by workshops and panel discussions than by actual standards efforts. Recent events include EDRM 2011-2012 Kickoff Meeting, St Paul, MN, May 11-12, ICAIL 2011 DESI IV Workshop, Pittsburgh, PA, June 6, TREC Legal Track, Gaithersburg, MD, November, and the SIRE workshop at the Special Interest Group on Information Retrieval (SIGIR) SIGIR 2011 on July 28.

There seems to be a growing consensus that industry standards are not just useful, but likely necessary in eDiscovery. The Sedona Commentary on Achieving Quality in eDiscovery Principle 3 says, “Implementing a well thought out e-discovery process should seek to enhance the overall quality of the production in the form of: (a) reducing the time from request to response; (b) reducing cost; and (c) improving the accuracy and completeness of responses to requests.”

The question now seems to be, what type of standards need to be in place and who is going to craft them. So, what do you think?  Please share any comments you might have or if you'd like to know more about a particular topic.

Editor's Note: Welcome Jason Krause as a guest author to eDiscovery Daily blog!  Jason is a freelance writer in Madison, Wisconsin. He has written about technology and the law for more than a dozen years, and has been writing about EDD issues since the first Zubulake decisions. Jason began his career in Silicon Valley, writing about technology for The Industry Standard, and later served as the technology reporter for the ABA Journal. He can be reached at jasonkrause@hotmail.com.

eDiscovery Standards: Does the Industry Need Them?
 

eDiscovery Daily recently ran a three part series analyzing eDiscovery cost budgeting. Cost has long been a driving force in eDiscovery decision-making, but it is just one dimension in choosing EDD services. Other industries have well-established standards for quality – think of the automotive or software industries, which have standard measures for defects or bugs. This year there has been a rising call for developing industry standards in eDiscovery to provide quality measures.

There is a belief that eDiscovery is becoming more routine and predictable, which means standards of service can be established. But is eDiscovery really like manufacturing? Can you assess the level of service in EDD in terms of number of defects? Quality is certainly a worthy aim – government agencies have shifted away from cost being the single biggest justification for contract award, more heavily weighting quality of service in such decisions.  The question is how to measure quality in EDD.

Quality standards that offer some type of objective measures could theoretically provide another basis for decision-making in addition to cost. Various attempts have been made at creating industry standards over the years, very little has yet been standardized. The recent DESI (Discovery of Electronically Stored Information) IV workshop at the International Conference on Artificial Intelligence and Law in June investigated possible standards. In the background to the conference, organizers bemoaned that “there is no widely agreed-upon set of standards or best practices for how to conduct a reasonable eDiscovery search for relevant evidence.” 

Detractors say standards are just hoops for vendors to jump through or a checkbox to check that don’t do much to differentiate one company from another. However, proponents believe industry standards could define issues like document defensibility, defining output, or how to go about finding responsive documents in a reasonable way, issues that can explode if not managed properly.

The Sedona Conference, Electronic Discovery Reference Model (EDRM), and Text Retrieval Conference (TREC) Legal Track all have efforts of one kind or another to establish standards for eDiscovery. EDRM provides a model for eDiscovery and standards of production. It has also led an effort to create a standard, generally accepted XML model to allow vendors and systems to more easily share electronically stored information (ESI). However, that applies to software vendors, and really doesn’t help the actual work of eDiscovery.

The Sedona Commentary on Achieving Quality in eDiscovery calls for development of standards and best practices in processing electronic evidence. Some of the standards being considered for broad industry standards are the ISO 9000 standard, which provides industry-specific frameworks for certifying organizations or the Capability Maturity Model Integration (CMMI), centered around improving processes.

The Association for Information Management Professionals (ARMA) is pushing its Generally Accepted Record-keeping Principles (GARP) framework to provide best practices for information management in the eDiscovery context. This article from ARMA is dismissive of information governance efforts such as the EDRM, which it says provides a framework for eDiscovery projects, but “falls short of describing standards or best practices that can be applied to the complex issues surrounding the creation, management, and governance of electronic information.”

Meanwhile, there are efforts underway to standardize pieces of the eDiscovery process. Law.com says that billing code standards are in the works to help clients understand what they are buying when they sign a contract for eDiscovery services.

Perhaps the most interesting and important effort is the TREC Legal Track, which began as government research project into improving search results. The project garnered a fair amount of attention when it discovered that keyword searching was as effective as or better than many advanced concept searches and other technology that was becoming popular in the industry. Since that time, researchers have been trying to develop objective criteria for comparing methods for searching large collections of documents in civil litigation.

As of today, these efforts are largely unrelated, disjointed, or even dismissive of competing efforts. In my next post, I’ll dig into specific efforts to see if any make sense for the industry. So, what do you think? Are standards needed, or is it just a lot of wheel spinning? Please share any comments you might have or if you'd like to know more about a particular topic.

Editor's Note: Welcome Jason Krause as a guest author to eDiscovery Daily blog!  Jason is a freelance writer in Madison, Wisconsin. He has written about technology and the law for more than a dozen years, and has been writing about EDD issues since the first Zubulake decisions. Jason began his career in Silicon Valley, writing about technology for The Industry Standard, and later served as the technology reporter for the ABA Journal. He can be reached at jasonkrause@hotmail.com.

  • 1
  • 2