Close
Enter your
text here

Review

Court Orders Defendants to Sample Disputed Documents to Help Settle Dispute: eDiscovery Case Law

In Updateme Inc. v. Axel Springer SE, No. 17-cv-05054-SI (LB) (N.D. Cal. Oct. 11, 2018), California Magistrate Judge Laurel Beeler ordered the defendants to review a random sample of unreviewed documents in dispute and produce any responsive documents reviewed (along with a privilege log, if applicable) and report on the number of documents and families reviewed and the rate of responsiveness within one week.

Case Background

In this case where the plaintiff, creator of a news-aggregator cell-phone app, claimed that the defendants “stole” their platform and released a copycat app, learned that the defendants used the code name “Ajax” to refer their product.  The defendants determined that there were 5,126 unique documents (including associated family members) within the previously collected ESI that hit on the term “Ajax”, but they had not reviewed those documents for responsiveness.  The plaintiff asked the court to order the defendants to review those documents and produce responsive documents within two weeks.

The defendants claimed that the term “Ajax” is a project name that they created to refer to the plaintiff’s threatened litigation, not the product itself and claimed that “a sampling of the `Ajax’ documents confirms that, in every responsive document, the term `Ajax’ was used to refer to the dispute itself.”  But, the plaintiff cited 93 produced documents generally and two documents in particular (which the defendants were attempting to clawback as privileged) that referred to their product.  However, the defendants also claimed that it would be unduly burdensome and expensive to review the “Ajax” documents at this stage of the litigation and also argued that the term “Ajax” was not included in the ESI Protocol that the parties agreed upon months ago and should not be added at this late stage.

Judge’s Ruling

Judge Beeler observed this: “Whether ‘Ajax’ refers to Updateme or only the defendants’ dispute with Updateme is in some sense a distinction without a difference. Either way, the search term ‘Ajax’ is likely to return documents that are responsive to Updateme’s request for “[a]ll communications . . . concerning Updateme or the updaemi® application[.]” Documents concerning the defendants’ dispute with Updateme are likely documents concerning Updateme.” 

Judge Beeler also noted that “even if ‘Ajax’ refers to the dispute, that does not mean that documents that contain ‘Ajax’ are necessarily more likely to be privileged or protected from disclosure”, using a hypothetical scenario where two non-lawyers might discuss the impact of the “Ajax” dispute on profits.  She concluded her analysis with this statement: “To the extent the defendants are suggesting that if ‘Ajax’ purportedly refers to their dispute with Updateme, ESI containing ‘Ajax’ should remain outside the scope of discovery, the court is not convinced.”

As a result, Judge Beeler ordered the defendants to “randomly select 10% of the unreviewed documents {in dispute}, review them (and their associated family members) for responsiveness, produce responsive documents (and a privilege log for any responsive documents that are withheld), and provide a chart listing the number of documents and families reviewed and the rate of responsiveness” within one week.  Judge Beeler stated that the parties should then meet and confer if they continued to have disputes regarding these documents.

So, what do you think?  Should random sampling be used more to settle proportionality disputes or should it be a last resort?  Please let us know if any comments you might have or if you’d like to know more about a particular topic.

Case opinion link courtesy of eDiscovery Assistant.

Sponsor: This blog is sponsored by CloudNine, which is a data and legal discovery technology company with proven expertise in simplifying and automating the discovery of data for audits, investigations, and litigation. Used by legal and business customers worldwide including more than 50 of the top 250 Am Law firms and many of the world’s leading corporations, CloudNine’s eDiscovery automation software and services help customers gain insight and intelligence on electronic data.

Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by CloudNine. eDiscovery Daily is made available by CloudNine solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Daily should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.

Mike Q Says the Weakest Link in TAR is Humans: eDiscovery Best Practices

We started the week with a post from Tom O’Connor (his final post in his eDiscovery Project Management from Both Sides series).  And, we’re ending the week covering an article from Mike Quartararo on Technology Assisted Review (TAR).  You would think we were inadvertently promoting our webcast next week or something.  :o)

Remember The Weakest Link? That was the early 2000’s game show with the sharp-tongued British hostess (Anne Robinson) telling contestants that were eliminated “You are the weakest link.  Goodbye!”  Anyway, in Above the Law (Are Humans The Weak Link In Technology-Assisted Review?), Mike takes a look at the debate as to which tool is the superior tool for conducting TAR and notes the lack of scientific studies that point to any particular TAR software or algorithm being dramatically better or, more importantly, significantly more accurate, than any other.  So, if it’s not the tool that determines the success or failure of a TAR project, what is it?  Mike says when TAR has problems, it’s because of the people.

Of course, Mike knows quite a bit about TAR.  He’s managed his “share of” of projects, has used “various flavors of TAR” and notes that “none of them are perfect and not all of them exceed all expectations in all circumstances”.  Mike has also been associated with the EDRM TAR project (which we covered earlier this year here) for two years as a team leader, working with others to draft proposed standards.

When it comes to observations about TAR that everyone should be able to agree on, Mike identifies three: 1) that TAR is not Artificial Intelligence, just “machine learning – nothing more, nothing less”, 2) that TAR technology works and “TAR applications effectively analyze, categorize, and rank text-based documents”, and 3) “using a TAR application — any TAR application — saves time and money and results in a reasonable and proportional outcome.”  Seems logical to me.

So, when TAR doesn’t work, “the blame may fairly be placed at the feet (and in the minds) of humans.”  We train the software by categorizing the training documents, we operate the software, we analyze the outcome.  So, it’s our fault.

Last month, we covered this case where the plaintiffs successfully requested additional time for discovery when defendant United Airlines, using TAR to manage its review process, produced 3.5 million documents.  However, sampling by the plaintiffs (and later confirmed by United) found that the production contained only 600,000 documents that were responsive to their requests (about 17% of the total production).  That seems like a far less than ideal TAR result to me.  Was that because of human failure?  Perhaps, when it comes down to it, the success of TAR being dependent on humans points us back to the long-used phrase regarding humans and computers: Garbage In, Garbage Out.

So, what do you think?  Should TAR be considered Artificial Intelligence?  As always, please share any comments you might have or if you’d like to know more about a particular topic.

Image Copyright © British Broadcasting Corporation

Sponsor: This blog is sponsored by CloudNine, which is a data and legal discovery technology company with proven expertise in simplifying and automating the discovery of data for audits, investigations, and litigation. Used by legal and business customers worldwide including more than 50 of the top 250 Am Law firms and many of the world’s leading corporations, CloudNine’s eDiscovery automation software and services help customers gain insight and intelligence on electronic data.

Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by CloudNine. eDiscovery Daily is made available by CloudNine solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Daily should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.

Here’s a Terrific Scorecard for Mobile Evidence Discovery: eDiscovery Best Practices

As we’ve noted before, eDiscovery isn’t just about discovery of emails and office documents anymore.  There are so many sources of data these days that legal professionals have to account for and millions more being transmitted over the internet every minute, much of which is being transmitted and managed via mobile devices.  Now, here’s a terrific new Mobile Evidence Burden and Relevance Scorecard, courtesy of Craig Ball!

Craig has had a lot to say in the past about mobile device preservation and collection, even going as far as to say that failure to advise clients to preserve relevant and unique mobile data when under a preservation duty is committing malpractice.  To help lawyers avoid that fate, Craig has described a simple, scalable approach for custodian-directed preservation of iPhone data.

Craig’s latest post (Mobile to the Mainstream, PDF article here) “looks at simple, low-cost approaches to getting relevant and responsive mobile data into a standard e-discovery review workflow” as only Craig can.  But, Craig also “offers a Mobile Evidence Scorecard designed to start a dialogue leading to a consensus about what forms of mobile content should be routinely collected and reviewed in e-discovery, without the need for digital forensic examination.”

It’s that scorecard – and Craig’s discussion of it – that is really cool.  Craig breaks down various types of mobile data (e.g., Files, Photos, Messages, Phone Call History, Browser History, etc.) in terms of Ease of Collection and Ease of Review (Easy, Moderate or Difficult), Potential Relevance (Frequent, Case Specific or Rare) and whether or not you would Routinely Collect (Yes, No or Maybe).  Believe it or not, Craig states that you would routinely collect almost half (7 out of 16 marked as “Yes”, 2 more marked as “Maybe”) of the file types.  While the examples are specific to the iPhone (which I think is used most by legal professionals), the concepts apply to Android and other mobile devices as well.

I won’t steal Craig’s thunder here; instead, I’ll direct you to his post here so that you can check it out yourself.  This scorecard can serve as a handy guide for what lawyers should expect for mobile device collection in their cases.  Obviously, it depends on the lawyer and the type of case in which they’re involved, but it’s still a good general reference guide.

So, what do you think?  Do you routinely collect data from mobile devices for your cases?  And, as always, please share any comments you might have or if you’d like to know more about a particular topic.

Sponsor: This blog is sponsored by CloudNine, which is a data and legal discovery technology company with proven expertise in simplifying and automating the discovery of data for audits, investigations, and litigation. Used by legal and business customers worldwide including more than 50 of the top 250 Am Law firms and many of the world’s leading corporations, CloudNine’s eDiscovery automation software and services help customers gain insight and intelligence on electronic data.

Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by CloudNine. eDiscovery Daily is made available by CloudNine solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Daily should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.

Plaintiffs Granted Discovery Extension Due to Defendant’s TAR Review Glitch: eDiscovery Case Law

In the case In Re Domestic Airline Travel Antitrust Litigation, MDL Docket No. 2656, Misc. No. 15-1404 (CKK), (D.D.C. Sept. 13, 2018), District of Columbia District Judge Colleen Kollar-Kotelly granted the Plaintiffs’ Motion for an Extension of Fact Discovery Deadlines (over the defendants’ objections) for six months, finding that defendant “United’s production of core documents that varied greatly from the control set in terms of the applicable standards for recall and precision and included a much larger number of non-responsive documents that was anticipated” (United’s core production of 3.5 million documents contained only 600,000 documents that were responsive).

Case Background

In the case involves a multidistrict class action litigation brought by the plaintiffs (purchasers of air passenger transportation for domestic travel) alleging that the defendant airlines willingly conspired to engage in unlawful restraint of trade, the plaintiffs filed an instant Motion for Extension of Time to Complete Discovery, requesting an extension of six months, predicated on an “issue with United’s ‘core’ document production,” asserting that defendant United produced more than 3.5 million [core] documents to the Plaintiffs, but “due to United’s technology assisted review process (‘TAR’), only approximately 17%, or 600,000, of the documents produced are responsive to Plaintiffs’ requests,” and the plaintiffs (despite having staffed their discovery review with 70 attorneys) required additional time to sort through them.

Both defendants (Delta and United) opposed the plaintiffs’ request for an extension, questioning whether the plaintiffs had staffed the document review with 70 attorneys and suggesting the Court review the plaintiffs’ counsel’s monthly time sheets to verify that statement.  Delta also questioned by it would take the plaintiffs so long to review the documents and tried to extrapolate how long it would take to review the entire set of documents based on a review of 3 documents per minute (an analysis that the plaintiffs called “preposterous”).  United indicated that it engaged “over 180 temporary contract attorneys to accomplish its document production and privilege log process within the deadlines” set by the Court, so the plaintiffs should be expected to engage in the same expenditure of resources.  But, the plaintiffs contended that they “could not have foreseen United’s voluminous document production made up [of] predominantly non-responsive documents resulting from its deficient TAR process when they jointly proposed an extension of the fact discovery deadline in February 2018.”

Judge’s Ruling

Judge Kollar-Kotelly noted that “Plaintiffs contend that a showing of diligence involves three factors — (1) whether the moving party diligently assisted the Court in developing a workable scheduling order; (2) that despite the diligence, the moving party cannot comply with the order due to unforeseen or unanticipated matters; and (3) that the party diligently sought an amendment of the schedule once it became apparent that it could not comply without some modification of the schedule.”  She noted that “there is no dispute that the parties diligently assisted the Court in developing workable scheduling orders through their preparation of Joint Status Reports prior to the status conferences in which discovery issues and scheduling were discussed, and in their meetings with the Special Master, who is handling discovery matters in this case.”

Judge Kollar-Kotelly also observed that “United’s core production of 3.5 million documents — containing numerous nonresponsive documents — was unanticipated by Plaintiffs, considering the circumstances leading up to that production” and that “Plaintiffs devoted considerable resources to the review of the United documents prior to filing this motion seeking an extension”.  Finding also that “Plaintiffs’ claim of prejudice in not having the deadlines extended far outweighs any inconvenience that Defendants will experience if the deadlines are extended”, Judge Kollar-Kotelly found “that Plaintiffs have demonstrated good cause to warrant an extension of deadlines in this case based upon Plaintiffs’ demonstration of diligence and a showing of nominal prejudice to the Defendants, if an extension is granted, while Plaintiffs will be greatly prejudiced if the extension is not granted.”  As a result, she granted the motion to request the extension.

So, what do you think?  Was the court right to have granted the extension?  Please let us know if any comments you might have or if you’d like to know more about a particular topic.

Case opinion link courtesy of eDiscovery Assistant.

Also, if you’re going to be in Houston on Thursday, September 27, just a reminder that I will be speaking at the second annual Legal Technology Showcase & Conference, hosted by the Women in eDiscovery (WiE), Houston Chapter, South Texas College of Law and the Association of Certified E-Discovery Specialists (ACEDS).  I’ll be part of the panel discussion AI and TAR for Legal: Use Cases for Discovery and Beyond at 3:00pm and CloudNine is also a Premier Platinum Sponsor for the event (as well as an Exhibitor, so you can come learn about us too).  Click here to register!

Sponsor: This blog is sponsored by CloudNine, which is a data and legal discovery technology company with proven expertise in simplifying and automating the discovery of data for audits, investigations, and litigation. Used by legal and business customers worldwide including more than 50 of the top 250 Am Law firms and many of the world’s leading corporations, CloudNine’s eDiscovery automation software and services help customers gain insight and intelligence on electronic data.

Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by CloudNine. eDiscovery Daily is made available by CloudNine solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Daily should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.

Survey Says! Predictive Coding Technologies and Protocols Survey Results: eDiscovery Trends

Last week, I discussed the predictive coding survey that Rob Robinson was conducting on his Complex Discovery site (along with the overview of key predictive coding related terms.  The results are in and here are some of the findings.

As Rob notes in the results post here, the Predictive Coding Technologies and Protocols Survey was initiated on August 31 and concluded on September 15.  It’s a non-scientific survey designed to help provide a general understanding of the use of predictive coding technologies and protocols from data discovery and legal discovery professionals within the eDiscovery ecosystem.  The survey was designed to provide a general understanding of predictive coding technologies and protocols and had two primary educational objectives:

  • To provide a consolidated listing of potential predictive coding technology and protocol definitions. While not all-inclusive or comprehensive, the listing was vetted with selected industry predictive coding experts for completeness and accuracy, thus it appears to be profitable for use in educational efforts.
  • To ask eDiscovery ecosystem professionals about their usage and preferences of predictive coding platforms, technologies, and protocols.

There were 31 total respondents in the survey.  Here are some of the more notable results:

  • More than 80% of responders (80.64%) shared that they did have a specific primary platform for predictive coding versus just under 20% (19.35%), who indicated they did not.
  • There were 12 different platforms noted as primary predictive platforms by responders, but only three platforms received more than one vote and they accounted for more than 50% of responses (61%).
  • Active Learning was the most used predictive coding technology, with more than 70% of responders (70.96%) reporting that they use it in their predictive coding efforts.
  • Just over two-thirds of responders (67.74%) use more than one predictive coding technology in their predictive coding efforts, while just under one-third (32.25%) use only one.
  • Continuous Active Learning (CAL) was (by far) the most used predictive coding protocol, with more than 87% of responders (87.09%) reporting that they use it in their predictive coding efforts.

Rob has reported several other results and provided graphs for additional details.  To check out all of the results, click here.

So, what do you think?  Do any of the results surprise you?  Please share any comments you might have or if you’d like to know more about a particular topic.

Also, if you’re going to be in Houston on Thursday, September 27, just a reminder that I will be speaking at the second annual Legal Technology Showcase & Conference, hosted by the Women in eDiscovery (WiE), Houston Chapter, South Texas College of Law and the Association of Certified E-Discovery Specialists (ACEDS).  I’ll be part of the panel discussion AI and TAR for Legal: Use Cases for Discovery and Beyond at 3:00pm and CloudNine is also a Premier Platinum Sponsor for the event (as well as an Exhibitor, so you can come learn about us too).  Click here to register!

Image Copyright (C) FremantleMedia North America, Inc.

Sponsor: This blog is sponsored by CloudNine, which is a data and legal discovery technology company with proven expertise in simplifying and automating the discovery of data for audits, investigations, and litigation. Used by legal and business customers worldwide including more than 50 of the top 250 Am Law firms and many of the world’s leading corporations, CloudNine’s eDiscovery automation software and services help customers gain insight and intelligence on electronic data.

Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by CloudNine. eDiscovery Daily is made available by CloudNine solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Daily should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.

If You’re an eDiscovery Professional Interested in Predictive Coding, Here is a Site You May Want to Check Out: eDiscovery Trends

On his Complex Discovery site, Rob Robinson does a great job of analyzing trends in the eDiscovery industry and often uses surveys to gauge sentiment within the industry for things like industry business confidence.  Now, Rob is proving and overview and conducting a survey regarding predictive coding technologies and protocols for representatives of leading eDiscovery providers that should prove interesting.

On his site at Predictive Coding Technologies and Protocols: Overview and Survey, Rob notes that “it is increasingly more important for electronic discovery professionals to have a general understanding of the technologies that may be implemented in electronic discovery platforms to facilitate predictive coding of electronically stored information.”  To help in that, Rob provides working lists of predictive coding technologies and TAR protocols that is worth a review.

You probably know what Active Learning is.  Do you know what Latent Semantic Analysis is? What about Logistic Regression?  Or a Naïve Bayesian Classifier?  If you don’t, Rob discusses definitions for these different types of predictive coding technologies and others.

Then, Rob also provides a list of general TAR protocols that includes Simple Passive Learning (SPL), Simple Active Learning (SAL), Continuous Active Learning (CAL) and Scalable Continuous Active Learning (S-CAL), as well as the Hybrid Multmodal Method used by Ralph Losey.

Rob concludes with a link to a simple three-question survey designed to help electronic discovery professionals identify the specific machine learning technologies and protocols used by eDiscovery providers in delivering the technology-assisted review feature of predictive coding.  It literally take 30 seconds to complete.  To find out the questions, you’ll have to check out the survey.  ;o)

So far, Rob has received 19 responses (mine was one of those).  It will be interesting to see the results when he closes the survey and publishes the results.

So, what do you think?  Are you an expert in predictive coding technologies and protocols?  Please share any comments you might have or if you’d like to know more about a particular topic.

Sponsor: This blog is sponsored by CloudNine, which is a data and legal discovery technology company with proven expertise in simplifying and automating the discovery of data for audits, investigations, and litigation. Used by legal and business customers worldwide including more than 50 of the top 250 Am Law firms and many of the world’s leading corporations, CloudNine’s eDiscovery automation software and services help customers gain insight and intelligence on electronic data.

Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by CloudNine. eDiscovery Daily is made available by CloudNine solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Daily should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.

TAR Rules for the New York Commercial Division: eDiscovery Trends

File this one under stories I missed until yesterday.  We’ve seen plenty of cases where the use of Technology Assisted Review (TAR) has been approved and even one this year where a protocol for TAR was ordered by the court.  But, here is a case of a jurisdiction that has proposed and adopted a rule to encourage use of the most efficient means to review documents, including TAR.

As reported in the New York Law Journal (NY Commercial Division Gives Fuller Embrace to E-Discovery Under New Rule, written by Andrew Denney), the New York Commercial Division has adopted a new rule to support the use of technology-assisted document review in appropriate cases.

As the author notes, plenty of commercial litigants are already using technology to help them breeze through potentially labor-intensive tasks such as weeding out irrelevant documents via predictive coding or threading emails for easier reading.  But unlike the U.S. District Court for the Southern District of New York, which has developed a substantial volume of case law bringing eDiscovery proficiency to the bar (much of it authored by recently retired U.S. Magistrate Judge Andrew Peck), New York state courts have provided little guidance on the topic.

Until now.  The new rule, proposed last December by the Commercial Division Advisory Council and approved last month by Lawrence Marks, the state court system’s chief administrative judge and himself a former Commercial Division jurist, would fill the gap in the rules, said Elizabeth Sacksteder, a Paul, Weiss, Rifkind, Wharton & Garrison partner and member of the advisory council.  That rule, to be incorporated as a subpart of current Rule 11-e of the Rules of the Commercial Division, reads as follows:

The parties are encouraged to use the most efficient means to review documents, including electronically stored information (“ESI”), that is consistent with the parties’ disclosure obligations under Article 31 of the CPLR and proportional to the needs of the case.  Such means may include technology-assisted review, including predictive coding, in appropriate cases.

Muhammad Faridi, a commercial litigator and a partner at Patterson Belknap Webb & Tyler, said that using technology-assisted review is nothing new to most practitioners in the Commercial Division, but it is “revolutionary” for the courts to adopt a rule encouraging its use.  Maybe so!

So, what do you think?  Are you aware of any other rules out there supporting or encouraging the use of TAR?  If so, let us know about them!  And, as always, please share any comments you might have or if you’d like to know more about a particular topic.

Sponsor: This blog is sponsored by CloudNine, which is a data and legal discovery technology company with proven expertise in simplifying and automating the discovery of data for audits, investigations, and litigation. Used by legal and business customers worldwide including more than 50 of the top 250 Am Law firms and many of the world’s leading corporations, CloudNine’s eDiscovery automation software and services help customers gain insight and intelligence on electronic data.

Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by CloudNine. eDiscovery Daily is made available by CloudNine solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Daily should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.

Law Firm Partner Says Hourly Billing Model “Makes No Sense” with AI: eDiscovery Trends

Artificial intelligence (AI) is transforming the practice of law and we’ve covered the topic numerous times (with posts here, here and here, among others).  And, I’m not even including all of the posts about technology assisted review (TAR).  According to one law firm partner at a recent panel discussion, it could even (finally) spell the end of the billable hour.

In Bloomberg Law’s Big Law Business blog (Billable Hour ‘Makes No Sense’ in an AI World, written by Helen Gunnarsson), the author covered a panel discussion at a recent American Bar Association conference, which included Dennis Garcia, an assistant general counsel for Microsoft in Chicago, Kyle Doviken, a lawyer who works for Lex Machina in Austin and Anthony E. Davis, a partner with Hinshaw & Culbertson LLP in New York.  The panel was moderated by Bob Ambrogi, a Massachusetts lawyer and blogger (including the LawSites blog, which we’ve frequently referenced on this blog).

Davis showed the audience a slide quoting Andrew Ng, a computer scientist and professor at Stanford University: “If a typical person can do a mental task with less than one second of thought, we can probably automate it using AI either now or in the near future.” AI can “automate expertise,” Davis said. Because software marketed by information and technology companies is increasingly making it unnecessary to ask a lawyer for information regarding statutes, regulations, and requirements, “clients are not going to pay for time,” he said. Instead, he predicted, they will pay for a lawyer’s “judgment, empathy, creativity, adaptability, and emotional intelligence.”

Davis said AI will result in dramatic changes in law firms’ hiring and billing, among other things. The hourly billing model, he said, “makes no sense in a universe where what clients want is judgment.” Law firms should begin to concern themselves not with the degrees or law schools attended by candidates for employment but with whether they are “capable of developing judgment, have good emotional intelligence, and have a technology background so they can be useful” for long enough to make hiring them worthwhile, he said.

The panelists provided examples of how the use of artificial intelligence can enhance lawyers’ efficiency in areas such as legal research, document review in eDiscovery, drafting and evaluating contracts, evaluating lateral hires and even assessing propensities of federal judges.  Doviken indicated that a partner at a large firm had a “hunch” that a certain judge’s rulings favored alumni of the judge’s law school. After reviewing three years’ worth of data, the firm concluded the hunch was valid, assigned a graduate of that law school to a matter pending before that judge, and started winning its motions.

“The next generation of lawyers is going to have to understand how AI works” as part of the duty of competence, said Davis.  Want one example of how AI works that you are probably already using?  Click here.

So, what do you think?  Do you think that AI could spell the end of the billable hour?  Please let us know if any comments you might have or if you’d like to know more about a particular topic.

Sponsor: This blog is sponsored by CloudNine, which is a data and legal discovery technology company with proven expertise in simplifying and automating the discovery of data for audits, investigations, and litigation. Used by legal and business customers worldwide including more than 50 of the top 250 Am Law firms and many of the world’s leading corporations, CloudNine’s eDiscovery automation software and services help customers gain insight and intelligence on electronic data.

Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by CloudNine. eDiscovery Daily is made available by CloudNine solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Daily should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.

EDRM Needs Your Input on its TAR Guidelines: eDiscovery Best Practices

I’m here in Durham, NC at the annual EDRM Spring Workshop at Duke Law School and, as usual, the Workshop is a terrific opportunity to discuss the creation of standards and guidelines for the legal community, as well as network with like minded people on eDiscovery topics.  I’ll have more to report about this year’s Workshop next week.  But, one part of the Workshop that I will touch on now is the release of the public comment version of EDRM’s Technology Assisted Review (TAR) Guidelines.

Last Friday, EDRM released the preliminary draft of its TAR Guidelines for public comment (you can download it here).  EDRM and the Bolch Judicial Institute at Duke Law are seeking comments from the bench, bar, and public on a preliminary draft of Technology Assisted Review (TAR) Guidelines. Nearly 50 volunteer lawyers, e-discovery experts, software developers, scholars and judges worked on this draft under the auspices of EDRM. A version of the document was presented at the Duke Distinguished Lawyers’ conference on Technology Assisted Review, held Sept. 7-8, 2017. At that event, 15 judges and nearly 100 lawyers and practitioners provided feedback and comments on the draft. The document was further revised based on discussions at that conference, additional review by judges and additional review by EDRM members over the past couple of months (which involved significant changes and a much tighter and briefer guideline document). With the assistance of four law student fellows of the Bolch Judicial Institute, this draft was finalized in May 2018 for public comment.

So, calling this a preliminary draft is a bit of a misnomer as it has already been through several iterations of review and edit.  Now, it’s the public’s turn.

EDRM states that “Comments on this preliminary draft will be carefully considered by the drafting team and an advisory group of judges as they finalize the document for publication. Please send comments on this draft, whether favorable, adverse, or otherwise, as soon as possible, but no later than Monday, July 16, 2018. Comments must be submitted in tracked edits (note: the guidelines are in a Word document for easy ability to track changes) and submitted via email to edrm@law.duke.edu. All comments will be made available to the public.”

That’s all well and good and EDRM will hopefully get a lot of useful feedback on the guideline document.  However, one thing I have observed about public comment periods is that the people who tend to provide comments (i.e., geeks like us who attend EDRM workshops) are people who already understand TAR (and think they know how best to explain it to others).  If the goal of the EDRM TAR guidelines is to help the general bench and bar better understand TAR, then it’s important for the average attorney to review the document and provide comments as to how useful it is.

So, if you’re an attorney or legal technology practitioner who doesn’t understand TAR, I encourage (even challenge) you to review these guidelines and provide feedback.  Point out what you learned from the document and what was confusing and whether or not you feel that you have a better understanding of TAR and the considerations for when to use it and where it can be used.  Ask yourself afterward if you have a better idea of how to get started using TAR and if you understand the difference between TAR approaches.  If these guidelines can help a lot of members of the legal profession better understand TAR, that will be the true measure of its effectiveness.

Oh, and by the way, Europe’s General Data Protection Regulation is now in effect!  Are you ready?  If not, you might want to check out this webcast.

So, what do you think?  Will these guidelines help the average attorney or judge better understand TAR?  Please share any comments you might have or if you’d like to know more about a particular topic.

Sponsor: This blog is sponsored by CloudNine, which is a data and legal discovery technology company with proven expertise in simplifying and automating the discovery of data for audits, investigations, and litigation. Used by legal and business customers worldwide including more than 50 of the top 250 Am Law firms and many of the world’s leading corporations, CloudNine’s eDiscovery automation software and services help customers gain insight and intelligence on electronic data.

Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by CloudNine. eDiscovery Daily is made available by CloudNine solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Daily should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.

A Fresh Comparison of TAR and Keyword Search: eDiscovery Best Practices

Bill Dimm of Hot Neuron (the company that provides the product Clustify that provides document clustering and predictive coding technologies, among others) is one of the smartest men I know about technology assisted review (TAR).  So, I’m always interested to hear what he has to say about TAR, how it can be used and how effective it is when compared to other methods (such as keyword searching).  His latest blog post on the Clustify site talk about an interesting exercise that did exactly that: compared TAR to keyword search in a real classroom scenario.

In TAR vs. Keyword Search Challenge on the Clustify blog, Bill challenged the audience during the NorCal eDiscovery & IG Retreat to create keyword searches that would work better than technology-assisted review (predictive coding) for two topics.  Half of the room was tasked with finding articles about biology (science-oriented articles, excluding medical treatment) and the other half searched for articles about current law (excluding proposed laws or politics).  Bill then ran one of the searches against TAR in Clustify live during the presentation (the others he couldn’t do during the session due to time constraints, but did afterward and covered those on his blog, providing the specific searches to which he compared TAR).

To evaluate the results, Bill measured the recall from the top 3,000 and top 6,000 hits on the search query (3% and 6% of the population respectively) and also included the recall achieved by looking at all docs that matched the search query, just to see what recall the search queries could achieve if you didn’t worry about pulling in a ton of non-relevant docs.  For the TAR results he used TAR 3.0 (which is like Continuous Active Learning, but applied to cluster centers only) trained with (a whopping) two seed documents (one relevant from a keyword search and one random non-relevant document) followed by 20 iterations of 10 top-scoring cluster centers, for a total of 202 training documents.  To compare to the top 3,000 search query matches, the 202 training documents plus 2,798 top-scoring documents were used for TAR, so the total document review (including training) would be the same for TAR and the search query.

The result: TAR beat keyword search across the board for both tasks.  The top 3,000 documents returned by TAR achieved higher recall than the top 6,000 documents for any keyword search.  Based on this exercise, TAR achieved better results (higher recall) with half as much document review compared to any of the keyword searches.  The top 6,000 documents returned by TAR achieved higher recall than all of the documents matching any individual keyword search, even when the keyword search returned 27,000 documents.

Bill acknowledges that the audience had limited time to construct queries, they weren’t familiar with the data set, and they couldn’t do sampling to tune their queries, so the keyword searching wasn’t optimal.  Then again, for many of the attorneys I’ve worked with, that sounds pretty normal.  :o)

One reader commented about email headers and footers cluttering up results and Bill pointed out that “Clustify has the ability to ignore email header data (even if embedded in the middle of the email due to replies) and footers” – which I’ve seen and is actually pretty cool.  Irrespective of the specifics of the technology, Bill’s example is a terrific fresh example of how TAR can outperform keyword search – as Bill notes in his response to the commenter “humans could probably do better if they could test their queries, but they would probably still lose”.  Very interesting.  You’ll want to check out the details of his test via the link here.

So, what do you think?  Do you think this is a valid comparison of TAR and keyword searching?  Why or why not?  Please share any comments you might have or if you’d like to know more about a particular topic.

Sponsor: This blog is sponsored by CloudNine, which is a data and legal discovery technology company with proven expertise in simplifying and automating the discovery of data for audits, investigations, and litigation. Used by legal and business customers worldwide including more than 50 of the top 250 Am Law firms and many of the world’s leading corporations, CloudNine’s eDiscovery automation software and services help customers gain insight and intelligence on electronic data.

Disclaimer: The views represented herein are exclusively the views of the author, and do not necessarily represent the views held by CloudNine. eDiscovery Daily is made available by CloudNine solely for educational purposes to provide general information about general eDiscovery principles and not to provide specific legal advice applicable to any particular circumstance. eDiscovery Daily should not be used as a substitute for competent legal advice from a lawyer you have retained and who has agreed to represent you.