Close
Enter your
text here

eDiscoveryDaily

Working Successfully with eDiscovery and Litigation Support Service Providers: Information to Provide in an RFP
 

Open, two-way communication with a service provider is absolutely critical to a successful project.  It needs to start early, even before a project starts.  For many projects, it starts with the Request for Proposal (RFP).  Your goal with an RFP is to get good information from a vendor: information on pricing, information on schedule, information on approach, and information on deliverables.  To give you complete, accurate information, they need information from you.

Include this information in your RFPs:

  • Information about your Firm/Organization (location, key contacts)
  • Information about the Case (the party you represent, the case schedule)
  • Information about the Proposal Submission Process (contact information for the person who can answer questions about the RFP; contact information for those to whom the proposal should be submitted; the date the proposal is due; in what form the proposal should be delivered; any requirements you have regarding the format of the proposal)
  • Description of the Services you will Require
  • Information about the Scope of the Project  (the size of a document/data collection, types and characteristics of the documents/data)
  • Information on the Deliverable to the Vendor (when documents/data will be available to the vendor; in what form they will be delivered)
  • Description of the Deliverables you Require (formats, media, data elements, etc.)
  • Date by which the Project must be Completed (and include interim milestone dates if that’s appropriate)
  • Description of your Planned Participation in the Project (will you participate in training?  will you be onsite for any portion of the project?)
  • Description of your Preferred Method of Communication with the Service Provider
  • Description of your Requirements regarding Status Reports (how often do you require them? what information should be included?  to whom should they be submitted?)

Later in this blog series, we’ll discuss what questions you should ask in a proposal for several types of eDiscovery services.

What information do you provide to a service provider in an RFP?  Please share any comments you might have and let us know if you’d like to know more about an eDiscovery topic.

eDiscovery Best Practices: Testing Your Search Using Sampling

Friday, we talked about how to determine an appropriate sample size to test your search results as well as the items NOT retrieved by the search, using a site that provides a sample size calculator.  Yesterday, we talked about how to make sure the sample size is randomly selected.

Today, we’ll walk through an example of how you can test and refine a search using sampling.

TEST #1: Let’s say in an oil company we’re looking for documents related to oil rights.  To try to be as inclusive as possible, we will search for “oil” AND “rights”.  Here is the result:

  • Files retrieved with “oil” AND “rights”: 200,000
  • Files NOT retrieved with “oil” AND “rights”: 1,000,000

Using the site to determine an appropriate sample size that we identified before, we determine a sample size of 662 for the retrieved files and 664 for the non-retrieved files to achieve a 99% confidence level with a margin of error of 5%.  We then use this site to generate random numbers and then proceed to review each item in the retrieved and NOT retrieved items sets to determine responsiveness to the case.  Here are the results:

  • Retrieved Items: 662 reviewed, 24 responsive, 3.6% responsive rate.
  • NOT Retrieved Items: 664 reviewed, 661 non-responsive, 99.5% non-responsive rate.

Nearly every item in the NOT retrieved category was non-responsive, which is good.  But, only 3.6% of the retrieved items were responsive, which means our search was WAY over-inclusive.  At that rate, 192,800 out of 200,000 files retrieved will be NOT responsive and will be a waste of time and resource to review.  Why?  Because, as we determined during the review, almost every published and copyrighted document in our oil company has the phrase “All Rights Reserved” in the document and will be retrieved.

TEST #2: Let’s try again.  This time, we’ll conduct a phrase search for “oil rights” (which requires those words as an exact phrase).  Here is the result:

  • Files retrieved with “oil rights”: 1,500
  • Files NOT retrieved with “oil rights”: 1,198,500

This time, we determine a sample size of 461 for the retrieved files and (again) 664 for the NOT retrieved files to achieve a 99% confidence level with a margin of error of 5%.  Even though, we still have a sample size of 664 for the NOT retrieved files, we generate a new list of random numbers to review those items, as well as the 461 randomly selected retrieved items.  Here are the results:

  • Retrieved Items: 461 reviewed, 435 responsive, 94.4% responsive rate.
  • NOT Retrieved Items: 664 reviewed, 523 non-responsive, 78.8% non-responsive rate.

Nearly every item in the retrieved category was responsive, which is good.  But, only 78.8% of the NOT retrieved items were responsive, which means over 20% of the NOT retrieved items were actually responsive to the case (we also failed to retrieve 8 of the items identified as responsive in the first iteration).  So, now what?

TEST #3: If you saw this previous post, you know that proximity searching is a good alternative for finding hits that are close to each other without requiring the exact phrase.  So, this time, we’ll conduct a proximity search for “oil within 5 words of rights”.  Here is the result:

  • Files retrieved with “oil within 5 words of rights”: 5,700
  • Files NOT retrieved with “oil within 5 words of rights”: 1,194,300

This time, we determine a sample size of 595 for the retrieved files and (once again) 664 for the NOT retrieved files, generating a new list of random numbers for both sets of items.  Here are the results:

  • Retrieved Items: 595 reviewed, 542 responsive, 91.1% responsive rate.
  • NOT Retrieved Items: 664 reviewed, 655 non-responsive, 98.6% non-responsive rate.

Over 90% of the items in the retrieved category were responsive AND nearly every item in the NOT retrieved category was non-responsive, which is GREAT.  Also, all but one of the items previously identified as responsive was retrieved.  So, this is a search that appears to maximize recall and precision.

Had we proceeded with the original search, we would have reviewed 200,000 files – 192,800 of which would have been NOT responsive to the case.  By testing and refining, we only had to review 8,815 files –  3,710 sample files reviewed plus the remaining retrieved items from the third search (5,700 – 595 = 5,105) – most of which ARE responsive to the case.  We saved tens of thousands in review costs while still retrieving most of the responsive files, using a defensible approach.

Keep in mind that this is a simple example — we’re not taking into account misspellings and other variations we may want to include in our criteria.

So, what do you think?  Do you use sampling to test your search results?   Please share any comments you might have or if you’d like to know more about a particular topic.

eDiscovery Best Practices: A “Random” Idea on Search Sampling
 

Friday, we talked about how to determine an appropriate sample size to test your search results as well as the items NOT retrieved by the search, using a site that provides a sample size calculator.  Today, we’ll talk about how to make sure the sample size is randomly selected.

A randomly selected sample gives each file an equal chance of being reviewed and eliminates the chance of bias being introduced into the sample which might skew the results.  Merely selecting the first or last x number of items (or any other group) in the set may not reflect the population as a whole – for example, all of those items could come from a single custodian.  To ensure a fair, defensible sample, it needs to be selected randomly.

So, how do you select the numbers randomly?  Once again, the Internet helps us out here.

One site, Random.org, has a random integer generator which will randomly generate whole numbers.  You simply need to supply the number of random integers that you need to be generated, the starting number and ending number of the range within which the randomly generated numbers should fall.  The site will then generate a list of numbers that you can copy and paste into a text file or even a spreadsheet.  The site also provides an Advanced mode, which provides options for the numbers (e.g., decimal, hexadecimal), output format and how the randomization is ‘seeded’ (to generate the numbers).

In the example from Friday, you would provide 660 as the number of random integers to be generated, with a starting number of 1 and an ending number of 100,000 to get a list of random numbers for testing your search that yielded 100,000 files with hits (664, 1 and 1,000,000 respectively to get a list of numbers to test the non-hits).  You could paste the numbers into a spreadsheet, sort them and then retrieve the files by position in the result set based on the random numbers retrieved and review each of them to determine whether they reflect the intent of the search.  You’ll then have a good sense of how effective your search was, based on the random sample.  And, probably more importantly, using that random sample to test your search results will be a highly defensible method to verify your approach in court.

Tomorrow, we'll walk through a sample iteration to show how the sampling will ultimately help us refine our search.

So, what do you think?  Do you use sampling to test your search results?   Please share any comments you might have or if you’d like to know more about a particular topic.

eDiscovery Best Practices: Determining Appropriate Sample Size to Test Your Search
 

We’ve talked about searching best practices quite a bit on this blog.  One part of searching best practices (as part of the “STARR” approach I described in an earlier post) is to test your search results (both the result set and the files not retrieved) to determine whether the search you performed is effective at maximizing both precision and recall to the extent possible, so that you retrieve as many responsive files as possible without having to review too many non-responsive files.  One question I often get is: how many files do you need to review to test the search?

If you remember from statistics class in high school or college, statistical sampling is choosing a percentage of the results population at random for inspection to gather information about the population as a whole.  This saves considerable time, effort and cost over reviewing every item in the results population and enables you to obtain a “confidence level” that the characteristics of the population reflect your sample.  Statistical sampling is a method used for everything from exit polls to predict elections to marketing surveys to poll customers on brand popularity and is a generally accepted method of drawing conclusions for an overall results population.  You can sample a small portion of a large set to obtain a 95% or 99% confidence level in your findings (with a margin of error, of course).

So, does that mean you have to find your old statistics book and dust off your calculator or (gasp!) slide rule?  Thankfully, no.

There are several sites that provide sample size calculators to help you determine an appropriate sample size, including this one.  You’ll simply need to identify a desired confidence level (typically 95% to 99%), an acceptable margin of error (typically 5% or less) and the population size.

So, if you perform a search that retrieves 100,000 files and you want a sample size that provides a 99% confidence level with a margin of error of 5%, you’ll need to review 660 of the retrieved files to achieve that level of confidence in your sample (only 383 files if a 95% confidence level will do).  If 1,000,000 files were not retrieved, you would only need to review 664 of the not retrieved files to achieve that same level of confidence (99%, with a 5% margin of error) in your sample.  As you can see, the sample size doesn’t need to increase much when the population gets really large and you can review a relatively small subset to understand your collection and defend your search methodology to the court.

On Monday, we will talk about how to randomly select the files to review for your sample.  Same bat time, same bat channel!

So, what do you think?  Do you use sampling to test your search results?   Please share any comments you might have or if you’d like to know more about a particular topic.

Working Successfully with eDiscovery and Litigation Support Service Providers: The Evaluation Process
 

Sometimes selecting a service provider for a project will be a quick, easy process.  You may have a small project — similar to others you’ve handled — that you need to get up and running quickly.   If you have a list of good vendors with which you’ve worked, it may be as easy as a phone call or two to check availability and you’ll be all set.  In other cases, your selection process may be more involved.  Perhaps you are looking to build a preferred vendor program or you’ve got a large case involving many stakeholders who are looking for a thorough evaluation.  When a thorough evaluation is needed, here’s a suggested approach:

  1. Make a list of candidates:  Include vendors that have done a good job for you in the past.   Ask peers in the industry for suggestions.  In some cases, stakeholders may ask you to consider vendors with which they have a relationship.
  2. Make initial calls:  Call each vendor to get general information, to ensure they don’t have a conflict of interest, and to gauge their availability and interest in the project.  Revise the list if necessary.
  3. Send out Request for Proposal (RFP) / Request for Information (RFI):  In the next posts in this series, we’ll talk about these documents, so stay tuned.
  4. Review the responses.  Check the responses for completeness.  If there are holes, you can request missing information, or you might consider scratching a vendor from the list if there was blatant disregard for the requirements. 
  5. Follow-up:  You’ll probably have questions about every proposal, and you’ll want to clarify some points with each vendor.  And, there may be some points you’ll want to negotiate.  Even if a proposal is clear and doesn’t require an explanation, it’s useful to verify your understanding of approach and pricing.
  6. Rank each vendor:  List each evaluation point by importance, and rank each vendor for each point.  While this is an important step and a valuable tool, don’t let it replace good judgment.  Sometimes your instincts may tell you something different than the rankings do, and that should not be ignored!
  7. Check references for the vendors of most interest.  Later in this series, we’ll talk about effectively checking references.
  8. Make your selection (or your recommendation to the stakeholders).
  9. Notify the vendor you’ve selected and agree to a contract.
  10. Contact the other vendors and tell them they were not selected.

What has been your experience with evaluating and selecting service providers?  Please share any comments you might have and let us know if you’d like to know more about an eDiscovery topic.

eDiscovery Case Law: Destroy Data, Pay $1 Million, Lose Case

A federal judge in Chicago has levied sanctions against Rosenthal Collins Group LLC and granted a default judgment to the defendant for misconduct in a patent infringement case, also ordering the Chicago-based futures broker’s counsel to pay “the costs and attorneys fees incurred in litigating this motion” where plaintiff’s agent modified metadata related to relevant source code and wiped several relevant disks and devices prior to their production and where the court found counsel participated in “presenting misleading, false information, materially altered evidence and willful non-compliance with the Court’s orders.”

In Rosenthal Collins Group, LLC v. Trading Techs. Int’l, No. 05 C 4088, 2011 WL 722467 (N.D. Ill. Feb. 23, 2011), U.S. District Judge Sharon Johnson Coleman assessed a sanction of $1 million to Rosenthal Collins (RCG) and granted defendant/counter-plaintiff Trading Technologies’ (TT) motion for evidentiary sanctions and default judgment.  Much of the reason was due to the actions of RCG’s agent, Walter Buist.  Here’s why:

  • During Buist’s deposition, he admitted to “turning back the clock” to change the “last modified” date on the previously modified source code to make it appear that the modifications had occurred much earlier.  Despite clear evidence of these facts, RCG continued to deny them, even calling the claims “libelous,” “audacious,” and “Oliver Stone-esque.”
  • Buist also later admitted “wiping” six of seven zip disks that originally contained the relevant source code.  While he did not admit wiping the seventh disk, it was also wiped, and the Court found that it was “impossible to believe that it is merely coincidence that the seventh disk happened to be wiped on May 2, 2006, which just happened to be the same day that TT was scheduled to inspect it.”
  • The Court found that here was evidence that “virtually every piece of media ordered produced by the Court in May 2007 and July 2008 was wiped, altered, or destroyed.”
  • Despite RCG’s (and its counsel’s) attempts to distance itself from “its own agent, employed for the purposes of pursuing this litigation” and disavowing any “actual knowledge” of wrongdoing, Buist was RCG’s agent and, therefore, RCG was bound by Buist’s behavior and actions.
  • Even if RCG and its counsel had no knowledge of the destruction of the evidence, the destruction might have been avoided if RCG had complied with the Court’s orders in a timely manner to produce the materials and/or preserved the evidence by taking custody of it.

So, what do you think?  Should parties and their counsel be liable for the actions of an agent on their behalf?  Please share any comments you might have or if you’d like to know more about a particular topic.

Working Successfully with eDiscovery and Litigation Support Service Providers: Other Evaluation Criteria
 

In the last posts in this blog series, we talked about evaluating service provider pricing, quality, scalability and flexibility.  There are a few other things you may wish to look at as well, that may be especially significant for large, long-term projects or relationships.  Those things are:

  1. Litigation Experience:  Select a service provider that has litigation experience versus general business experience.   A non-litigation service provider that does scanning — for example — may be able to technically meet your requirements.  They are probably not, however, accustomed to the inflexible schedules and changing priorities that are commonplace in litigation work.
  2. Corporate Profile and Tenure:  For a large project, be sure to select a service provider that’s been around for a while and has a proven track record.  You want to be confident that the service provider that starts your project will be around to finish your project.
  3. Security and Confidentiality:  You want to ensure that your documents, data, and information are secure and kept confidential.  This means that you require a secure physical facility, secure systems, and appropriate confidentiality guidelines and agreements.
  4. SaaS Service Providers: For them, you need to evaluate the technology functionality and ensure that it includes the features you require, that those features are easy to access and to use, and that access, system reliability, system speed, and system security meet your requirements.
  5. Facility Location and Accessibility:  For many projects and many types of services, it won’t be necessary to spend time on the project site.   For other projects, that might not be the case.  For example, if a service provide is staffing a large document review project at its facility, the litigation team may need to spend time at the facility overseeing work and doing quality control reviews.  In such a case, the geographic location and the facility’s access to airports and hotels may be a consideration.

A lot goes into selecting the right service provider for a project, and it’s worth the time and effort to do a careful, thorough evaluation.  In the next posts in this series, we’ll discuss the vendor evaluation and selection process.

What has been your experience with evaluating and selecting service providers?  What evaluation criteria have you found to be most important?  Please share any comments you might have and let us know if you’d like to know more about an eDiscovery topic.

eDiscovery Trends: Forbes on the Rise of Predictive Coding
 

First the New York Times with an article about eDiscovery, now Forbes.  Who’s next, The Wall Street Journal?  😉

Forbes published a blog post entitled E-Discovery And the Rise of Predictive Coding a few days ago.  Written by Ben Kerschberg, Founder of Consero Group LLC, it gets into some legal issues and considerations regarding predictive coding that are interesting.  For some background on predictive coding, check out our December blog posts, here and here.

First, the author provides a very brief history of document review, starting with bankers boxes and WordPerfect and “[a]fter an interim phase best characterized by simple keyword searches and optical character recognition”, it evolved to predictive coding.  OK, that’s like saying that Gone with the Wind started with various suitors courting Scarlett O’Hara and after an interim phase best characterized by the Civil War, marriage and heartache, Rhett says to Scarlett, “Frankly, my dear, I don’t give a damn.”  A bit oversimplification of how review has evolved.

Nonetheless, the article gets into a couple of important legal issues raised by predictive coding.  They are:

  • Satisfying Reasonable Search Requirements: Whether counsel can utilize the benefits of predictive coding and still meet legal obligations to conduct a reasonable search for responsive documents under the federal rules.  The question is, what constitutes a reasonable search under Federal Rule 26(g)(1)(A), which requires that the responding attorney attest by signature that “with respect to a disclosure, it is complete and correct as of the time it is made”?
  • Protecting Privilege: Whether counsel can protect attorney-client privilege for their client when a privileged document is inadvertently disclosed.  Fed. Rule of. Evidence 502 provides that a court may order that a privilege or protection is not waived by disclosure if the disclosure was inadvertent and the holder of the privilege took reasonable steps to prevent disclosure.  Again, what’s reasonable?

The author concludes that the use of predictive coding is reasonable, because it a) makes document review more efficient by providing only those documents to the reviewer that have been selected by the algorithm; b) makes it more likely that responsive documents will be produced, saving time and resources; and c) refines relevant subsets for review, which can then be validated statistically.

So, what do you think?  Does predictive coding enable attorneys to satisfy these legal issues?   Is it reasonable?  Please share any comments you might have or if you’d like to know more about a particular topic.

eDiscovery Best Practices: Does Size Matter?
 

I admit it, with a title like “Does Size Matter?”, I’m looking for a few extra page views….  😉

I frequently get asked how big does an ESI collection need to be to benefit from eDiscovery technology.  In a recent case with one of my clients, the client had a fairly small collection – only about 4 GB.  But, when a judge ruled that they had to start conducting depositions in a week, they needed to review that data in a weekend.  Without FirstPass™, powered by Venio FPR™ to cull the data and OnDemand® to manage the linear review, they would not have been able to make that deadline.  So, they clearly benefited from the use of eDiscovery technology in that case.

But, if you’re not facing a tight deadline, how large does your collection need to be for the use of eDiscovery technology to provide benefits?

I recently conducted a webinar regarding the benefits of First Pass Review – aka Early Case Assessment, or a more accurate term (as George Socha points out regularly), Early Data Assessment.  One of the topics discussed in that webinar was the cost of review for each gigabyte (GB).  Extrapolated from an analysis conducted by Anne Kershaw a few years ago (and published in the Gartner report E-Discovery: Project Planning and Budgeting 2008-2011), here is a breakdown:

Estimated Cost to Review All Documents in a GB:

  • Pages per GB:                75,000
  • Pages per Document:      4
  • Documents Per GB:        18,750
  • Review Rate:                 50 documents per hour
  • Total Review Hours:       375
  • Reviewer Billing Rate:     $50 per hour

Total Cost to Review Each GB:      $18,750

Notes: The number of pages per GB can vary widely.  Page per GB estimates tend to range from 50,000 to 100,000 pages per GB, so 75,000 pages (18,750 documents) seems an appropriate average.  50 documents reviewed per hour is considered to be a fast review rate and $50 per hour is considered to be a bargain price.  eDiscovery Daily provided an earlier estimate of $16,650 per GB based on assumptions of 20,000 documents per GB and 60 documents reviewed per hour – the assumptions may change somewhat, but, either way, the cost for attorney review of each GB could be expected to range from at least $16,000 to $18,000, possibly more.

Advanced culling and searching capabilities of First Pass Review tools like FirstPass can enable you to cull out 70-80% of most collections as clearly non-responsive without having to conduct attorney review on those files.  If you have merely a 2 GB collection and assume the lowest review cost above of $16,000 per GB, the use of a First Pass Review tool to cull out 70% of the collection can save $22,400 in attorney review costs.  Is that worth it?

So, what do you think?  Do you use eDiscovery technology for only the really large cases or ALL cases?   Please share any comments you might have or if you’d like to know more about a particular topic.

eDiscovery Best Practices: Is Disclosure of Search Terms Required?
 

I read a terrific article a couple of days ago from the New York Law Journal via Law Technology News entitled Search Terms Are More Than Mere Words, that had some interesting takes about the disclosure of search terms in eDiscovery.  The article was written by David J. Kessler, Robert D. Owen, and Emily Johnston of Fulbright & Jaworski.  The primary emphasis of the article was with regard to the forced disclosure of search terms by courts.

In the age of “meet and confer”, it has become much more common for parties to agree to exchange search terms in a case to limit costs and increase transparency.  However, as the authors correctly note, search terms reflect counsel’s strategy for the case and, therefore, work product.  Their position is that courts should not force disclosure of search terms and that disclosure of terms is “not appropriate under the Federal Rules of Civil Procedure”.  The article provides a compelling argument as to why forced disclosure is not appropriate and provides some good case cites where courts have accepted or rejected requests to compel provision of search terms.  I won’t try to recap them all here – check out the article for more information.

So, should disclosure of search terms be generally required?  If not, what does that mean in terms of utilizing a defensible approach to searching?

Personally, I agree with the authors that forced disclosure of search terms is generally not appropriate, as it does reflect strategy and work product.  However, there is an obligation for each party to preserve, collect, review and produce all relevant materials to the best of their ability (that are not privileged, of course).  Searching is an integral part of that process.  And, the article does note that “chosen terms may come under scrutiny if there is a defect in the production”, though “[m]ere speculation or unfounded accusations” should not lead to a requirement to disclose search terms.

With that said, the biggest component of most eDiscovery collections today is email, and that email often reflects discussions between parties in the case.  In these cases, it’s much easier for opposing counsel to identify legitimate defects in the production because they have some of the same correspondence and documents and can often easily spot discrepancies in the production set.  If they identify legitimate omissions from the production, those omissions could cause the court to call into question your search procedures.  Therefore, it’s important to conduct a defensible approach to searching (such as the “STARR” approach I described in an earlier post) to be able to defend yourself if those questions arise.  Demonstrating a defensible approach to searching will offer the best chance to preserve your rights to protect your work product of search terms that reflect your case strategy.

So, what do you think?  Do you think that forced disclosure of search terms is appropriate?   Please share any comments you might have or if you’d like to know more about a particular topic.