Close
Enter your
text here

Review

Managing an eDiscovery Contract Review Team: First Steps in Drafting Criteria
 

In theory, responsive documents are described in the other side’s request for production.  In practice, those requests are often open to interpretation.  Your goal in drafting responsive criteria is to distill those requests and create a clear set of objective rules that leave little room for interpretation – a set of rules that can be applied correctly and consistently to the document collection.  This step is important for a couple of reasons:

  • It is difficult to get consistent results from a group of people doing the same task.  No two people will make exactly the same decision about every document – not even attorneys.  Even an individual attorney will not always make the same decision about duplicates of the same document.  Thorough, clear, detailed and objective criteria will minimize inconsistencies.
  • If discovery disputes arise, it may be necessary to demonstrate a good-faith effort.  Thorough, detailed criteria will help.  Judges understand the human error factor.  They are less tolerant of work that was approached casually or sloppily.  Clear, detailed criteria will demonstrate a carefully thought-out approach.

Where do you start?  First, do a little preparation.  There are some basic materials and information that you’ll need:

  • The complaint.
  • The request for production
  • Knowledge of the document collection (in the last blog in this series, we talked about sampling the collection).
  • Knowledge of the strategy for defending or prosecuting the case.

Once you’ve read the complaint and the document request and you’ve sampled the collection, you’ll have a feel for the materials that reviewers are likely to see and how those documents relate to the facts and legal issues in the case.  If a strategy for defending or prosecuting the case has been developed, make sure you understand that strategy.  It is likely that an understanding of the allegations and the strategy will broaden your view of what is responsive and important.

After these preparation steps, you’ll be ready to develop a first draft of the criteria.  In the next issue, we’ll talk about how to structure and write effective criteria.

Have you drafted criteria for a document review of a large collection?  How did you approach it and how well did it work?  Please share any comments you might have and let us know if you’d like to know more about an eDiscovery topic.

Managing an eDiscovery Contract Review Team: Get a Handle on the Document Collection
 

Once you’ve defined the objectives of the review, you need to move forward with other preparation steps: You need to draft review criteria, you need to identify the type of people that are appropriate for the review (do you need a staff of attorneys?  lay people?  staff with expertise in a specific subject matter?), and you need to pull that team together.  

Before moving forward with these steps, you need a bit more information.  You need to know what’s in the document collection.  You need to know what types of documents are in the collection and you need to know what type of content is in the documents.  Once you’ve got a handle on the collection, you’ll be in a better position to make decisions on subsequent steps.

Start by interviewing custodians.  You don’t need to talk to every custodian, but talk to a representative sample.  For example, if you are collecting documents from a corporate client, speak to at least one person from each department from which you’ve collected documents.  The person you speak to should probably be a manager or someone who has a good handle on the overall operation of the department.  Find out about the department’s operations and determine its role in the events that are at issue in the case.  Ask about the types of documents that are generated and retained.  Information that you glean here will help in the next step:  sampling the collection.

After you’ve collected information from the custodians, take a look at the documents.  Review a representative sample.  Look at documents from each custodian.  Take notes on what you are finding and make copies of documents that can be used as examples to illustrate the criteria you’ll be drafting and to be used in training.

Your ultimate goal is to develop a set of objective rules that a well-trained staff can apply effectively and consistently to the collection during the review.  The more you learn about the documents in advance, the better you’ll be able to do that.  So spend the time up front learning what you can about what’s in your document collection.

Do you typically sample an eDiscovery document collection before a review?  How did you approach it?  Please share any comments you might have and let us know if you’d like to know more about an eDiscovery topic.

Managing an eDiscovery Contract Review Team: Clearly Define Objectives
 

Yesterday, we introduced the blog series to discuss Managing an eDiscovery Contract Review Team.  Now, it’s time to get started!  The first step in preparing for a document review is to very clearly define the objectives of the review.  It’s an easy step, but it’s very important.  It will drive several subsequent decisions that you’ll make regarding management of the project. 

Here are some likely objectives you may choose:

  • Identify responsive documents
  • Identify privileged documents
  • Identify documents to be reviewed by an expert
  • Identify significant helpful and harmful documents

The choices you make here will affect the type of people you’ll assign to the review, the amount of time the review will take, the type of criteria you’ll need to draft, and the level of training you’ll need to do.

How do you make these decisions?  There are a few factors that should affect your choices:

  • The nature of the case and the nature of the document collection:  What type of case are you handling and what types of documents are in the collection?  If the case involves highly technical or scientific subject matter, you may need to train the review staff to segregate those documents that require review by an expert.
  • Where are you on the case and what do you know so far?  If you don’t know much yet about the case and what will be important, you won’t be in a position to ask reviewers to recognize significant materials.
  • What’s the pool of available reviewers?  Can you easily pull together a team that’s qualified to identify potentially privileged or significant documents?   If you need a very large team, you might be better off working with a team that can more easily focus on objective criteria, and use a smaller group of attorney staff to work with a smaller collection after the initial review.

Determine the objectives that will work best for your case and that can be accomplished with the available resources.  Make sure that the objectives are clearly defined and that everyone on the litigation team understands the objectives and has the same expectations.

What do you look to accomplish with an eDiscovery document review?  Have you had objectives in addition to those listed above?  Please share any comments you might have and let us know if you’d like to know more about an eDiscovery topic.

Managing an eDiscovery Contract Review Team: Introduction
 

In a perfect world, attorneys responsible for a case would review an entire document collection for responsive materials.  On large cases with huge collections, that’s just not practical or possible.  In those situations, your only choice may be to pull together a team of contract reviewers to identify responsive materials.  

How well does this work?  A review done by a contract review team will certainly cost less than one done by a team of law firm attorneys.  More likely than not, it will be done more efficiently.  And if there’s good preparation and management, the quality will be just as good (in fact, it may be better because a contract staff is more likely to stay better focused on the inevitable, more mundane aspects of the work).

I’ve managed many successful review projects done by teams of contract employees.  Sometimes those teams were made up of attorneys, but more often they included mostly paralegals and college-educated lay personnel with good reading and comprehension skills.  These projects were successful because they were structured and managed in a way where decision-making responsibility was in the hands of the attorneys, but there were effective mechanisms in place for disseminating those decisions to the team.  In this blog series, I’m going to walk through how to do this.  Specifically, we’ll be covering:

  • Clearly defining the objectives of the document review
  • Getting a handle on the document collection
  • Determining the right mix of people for the project
  • Creating effective document review criteria
  • Effectively training the review team
  • Managing the project
  • Disseminating updated project information
  • Implementing effective quality control procedures

What has been your experience with contract review teams for large projects?  Do you have good or bad experiences you can tell us about?  Are there any specific problems you’ve had with review teams?  Please share any comments you might have and let us know if you’d like to know more about an eDiscovery topic.

eDiscovery Trends: 2011 Predictions — By The Numbers
 

Comedian Nick Bakay”>Nick Bakay always ends his Tale of the Tape skits where he compares everything from Married vs. Single to Divas vs. Hot Dogs with the phrase “It's all so simple when you break things down scientifically.”

The late December/early January time frame is always when various people in eDiscovery make their annual predictions as to what trends to expect in the coming year.  We’ll have some of our own in the next few days (hey, the longer we wait, the more likely we are to be right!).  However, before stating those predictions, I thought we would take a look at other predictions and see if we can spot some common trends among those, “googling” for 2011 eDiscovery predictions, and organized the predictions into common themes.  I found serious predictions here, here, here, here and here.  Oh, also here and here.

A couple of quick comments: 1) I had NO IDEA how many times that predictions are re-posted by other sites, so it took some work to isolate each unique set of predictions.  I even found two sets of predictions from ZL Technologies, one with twelve predictions and another with seven, so I had to pick one set and I chose the one with seven (sorry, eWEEK!). If I have failed to accurately attribute the original source for a set of predictions, please feel free to comment.  2) This is probably not an exhaustive list of predictions (I have other duties in my “day job”, so I couldn’t search forever), so I apologize if I’ve left anybody’s published predictions out.  Again, feel free to comment if you’re aware of other predictions.

Here are some of the common themes:

  • Cloud and SaaS Computing: Six out of seven “prognosticators” indicated that adoption of Software as a Service (SaaS) “cloud” solutions will continue to increase, which will become increasingly relevant in eDiscovery.  No surprise here, given last year’s IDC forecast for SaaS growth and many articles addressing the subject, including a few posts right here on this blog.
  • Collaboration/Integration: Six out of seven “augurs” also had predictions related to various themes associated with collaboration (more collaboration tools, greater legal/IT coordination, etc.) and integration (greater focus by software vendors on data exchange with other systems, etc.).  Two people specifically noted an expectation of greater eDiscovery integration within organization governance, risk management and compliance (GRC) processes.
  • In-House Discovery: Five “pundits” forecasted eDiscovery functions and software will continue to be brought in-house, especially on the “left-side of the EDRM model” (Information Management).
  • Diverse Data Sources: Three “soothsayers” presaged that sources of data will continue to be more diverse, which shouldn’t be a surprise to anyone, given the popularity of gadgets and the rise of social media.
  • Social Media: Speaking of social media, three “prophets” (yes, I’ve been consulting my thesaurus!) expect social media to continue to be a big area to be addressed for eDiscovery.
  • End to End Discovery: Three “psychics” also predicted that there will continue to be more single-source end-to-end eDiscovery offerings in the marketplace.

The “others receiving votes” category (two predicting each of these) included maturing and acceptance of automated review (including predictive coding), early case assessment moving toward the Information Management stage, consolidation within the eDiscovery industry, more focus on proportionality, maturing of global eDiscovery and predictive/disruptive pricing.

Predictive/disruptive pricing (via Kriss Wilson of Superior Document Services and Charles Skamser of eDiscovery Solutions Group respective blogs) is a particularly intriguing prediction to me because data volumes are continuing to grow at an astronomical rate, so greater volumes lead to greater costs.  Creativity will be key in how companies deal with the larger volumes effectively, and pressures will become greater for providers (even, dare I say, review attorneys) to price their services more creatively.

Another interesting prediction (via ZL Technologies) is that “Discovery of Databases and other Structured Data will Increase”, which is something I’ve expected to see for some time.  I hope this is finally the year for that.

Finally, I said that I found serious predictions and analyzed them; however, there are a couple of not-so-serious sets of predictions here and here.  My favorite prediction is from The Posse List, as follows: “LegalTech…renames itself “EDiscoveryTech” after Law.com survey reveals that of the 422 vendors present, 419 do e-discovery, and the other 3 are Hyundai HotWheels, Speedway Racers and Convert-A-Van who thought they were at the Javits Auto Show.”

So, what do you think?  Care to offer your own “hunches” from your crystal ball?  Please share any comments you might have or if you’d like to know more about a particular topic.

eDiscovery Trends: Myth of SaaS Insecurity Finally Busted

Eleven years ago, when I first began talking to attorneys about hosting document collections online to manage the review and production process for discovery, the typical response that I got was “I would never consider putting my client’s documents online – it’s just not secure”.  Let’s face it – lawyers are not exactly early adopters of technology… 😉

These days, few folks seem to have that concern any more when it comes to putting sensitive data and documents online.  Many people bank online, buy items from Amazon and other “etailers”, share pictures and other personal information on Facebook, etc.  As for business data, SalesForce.com has become the top customer relationship management (CRM) application and many business users are using Google Docs to share documents with colleagues, as just two examples.

What do all of these applications have in common?  They are Software as a Service (SaaS) applications, delivering data and functionality via an online application.  As noted previously on this blog, a new IDC study forecasts the SaaS market to reach $40.5 billion by 2014, an annual growth rate of 25.3%.  Also by 2014, about 34% of all new business software purchases will be via SaaS applications, according to IDC.

SaaS review applications have also become increasingly popular in eDiscovery with several eDiscovery SaaS applications available that provide benefits including: no software to install, intuitive browser-based interfaces and ability to share the collection with your client, experts, and co-counsel without distributing anything more than a login.

As for security concerns, most litigators have come to accept that these systems are secure.  But, do they realize just how secure they are?

As an example, at Trial Solutions, the servers hosting data for our OnDemand® and FirstPass™ (powered by Venio FPR™) platforms are housed in a Tier 4 data center in Houston (which is where our headquarters is).  The security at this data center is military grade: 24 x 7 x 365 onsite security guards (I feel sorry for the folks who have to work this Saturday!), video surveillance, biometric and card key security required just to get into the building.  Not to mention a building that features concrete bollards, steel lined walls, bulletproof glass, and barbed wire fencing.  And, if you’re even able to get into the building, you then have to find the right server (in the right locked room) and break into the server security.  It’s like the movie Mission Impossible where Tom Cruise has to break into the CIA, except for the laser beams over the air vent (anyone who watches movies knows those can be easily thwarted by putting mirrors over them).  To replicate that level of security infrastructure would be cost prohibitive for even most large companies.

From the outside, SaaS applications secure data with login authentication and Secured Sockets Layer (SSL) encryption.  SSL encryption is like taking a piece of paper with text on it, scrambling the letters on that piece of paper and then tearing it up into many pieces and throwing the scraps into the wind.  To intercept a communication (one request to the server), you have to intercept all of the packets of a communication, then unscramble each packet individually and then reassemble them in the correct order.

Conversely, desktop review application data could be one stolen laptop away from being compromised.  No wonder why nobody talks about security concerns anymore with SaaS applications.

So, what do you think?  How secure is your document collection?  Please share any comments you might have or if you’d like to know more about a particular topic.

Happy Holidays from all of us at Trial Solutions and eDiscovery Daily!

eDiscovery Trends: Predictive Coding Strategy and Survey Results

Yesterday, we introduced the Virtual LegalTech online educational session Frontiers of E-Discovery: What Lawyers Need to Know About “Predictive Coding” and defined predictive coding while also noting the two “learning” methods that most predictive coding mechanisms use to predict document classifications.  To get background information regarding the session, including information about the speakers (Jason Baron, Maura Grossman and Bennett Borden), click here.

The session also focused on strategies for using predictive coding and results of the TREC 2010 Legal Track Learning Task on the effectiveness of “Predictive Coding” technologies.  Strategies discussed by Bennett Borden include:

  • Understanding the technology used by a particular provider:  Not only will supervised and active learning mechanisms often yield different results, but there are differing technologies within each of these learning mechanisms.
  • Understand the state of the law regarding predictive coding technology: So far, there is no case law available regarding use of this technology and, while it may eventually be the future of document review, that has yet to be established.
  • Obtain buy-in by the requesting party to use predictive coding technology: It’s much easier when the requesting party has agreed to your proposed approach and that agreement is included in an order of the court which covers the approach and also includes a FRE 502 “clawback” agreement and order.  To have a chance to obtain that buy-in and agreement, you’ll need a diligent approach that includes “tiering” of the collection by probable responsiveness and appropriate sampling of each tier level.

Maura Grossman then described TREC 2010 Legal Track Learning Task on the effectiveness of “Predictive Coding” technologies.  The team took the EDRM Enron Version 2 Dataset of 1.3 million public domain files, deduped it down to 685,000+ unique files and 5.5 GB of uncompressed data.  The team also identified eight different hypothetical eDiscovery requests for the test.

Participating predictive coding technologies were then given a “seed set” of roughly 1,000 documents that had previously been identified by TREC as responsive or non-responsive to each of the requests. Using this information, participants were required to rank the documents in the larger collection from most likely to least likely to be responsive, and estimate the likelihood of responsiveness as a probability for each document.  The study ranked the participants on recall rate accuracy based on 30% of the collection retrieved (200,000 files) and also on the predicted recall to determine a prediction accuracy.

The results?  Actual recall rates for all eight discovery requests ranged widely among the tools from 85.1% actual recall down to 38.2% (on individual requests, the range was even wider – as much as 82% different between the high and the low).  The prediction accuracy rates for the tools also ranged somewhat widely, from a high of 95% to a low of 42%.

Based on this study, it is clear that these technologies can differ significantly on how effective and efficient they are at correctly ranking and categorizing remaining documents in the collection based on the exemplar “seed set” of documents.  So, it’s always important to conduct sampling of both machine coded and human coded documents for quality control in any project, with or without predictive coding (we sometimes forget that human coded documents can just as often be incorrectly coded!).

For more about the TREC 2010 Legal Track study, click here.  As noted yesterday, you can also check out a replay of the session or download the slides for the presentation at the Virtual LegalTech site.

Full Disclosure: Trial Solutions provides predictive coding services using Hot Neuron LLC’s Clustify™, which categorizes documents by looking for similar documents in the exemplar set that satisfy a user-specified criteria, such as a minimum conceptual similarity or near-duplicate percentage.

So, what do you think?  Have you used predictive coding on a case?  Please share any comments you might have or if you’d like to know more about a particular topic.

eDiscovery Trends: What the Heck is “Predictive Coding”?
 

Yesterday, ALM hosted another Virtual LegalTech online "live" day online.  Every quarter, theVirtual LegalTech site has a “live” day with educational sessions from 9 AM to 5 PM ET, most of which provide CLE credit in certain states (New York, California, Florida, and Illinois).

One of yesterday’s sessions was Frontiers of E-Discovery: What Lawyers Need to Know About “Predictive Coding”.  The speakers for this session were:

Jason Baron: Director of Litigation for the National Archives and Records Administration, a founding co-coordinator of the National Institute of Standards and Technology’s Text Retrieval Conference (“TREC”) legal track and co-chair and editor-in-chief for various working groups for The Sedona Conference®;

Maura Grossman: Counsel at Wachtell, Lipton, Rosen & Katz, co-chair of the eDiscovery Working Group advising the New York State Unified Court System and coordinator of the 2010 TREC legal track; and

Bennett Borden: co-chair of the e-Discovery and Information Governance Section at Williams Mullen and member of Working Group I of The Sedona Conference on Electronic Document Retention and Production, as well as the Cloud Computing Drafting Group.

This highly qualified panel discussed a number of topics related to predictive coding, including practical applications of predictive coding technologies and results of the TREC 2010 Legal Track Learning Task on the effectiveness of “Predictive Coding” technologies.

Before discussing the strategies for using predictive coding technologies and the results of the TREC study, it’s important to understand what predictive coding is.  The panel gave the best descriptive definition that I’ve seen yet for predictive coding, as follows:

“The use of machine learning technologies to categorize an entire collection of documents as responsive or non-responsive, based on human review of only a subset of the document collection. These technologies typically rank the documents from most to least likely to be responsive to a specific information request. This ranking can then be used to “cut” or partition the documents into one or more categories, such as potentially responsive or not, in need of further review or not, etc.”

The panel used an analogy for predictive coding by relating it to spam filters that review and classify email and learn based on previous classifications which emails can be considered “spam”.  Just as no spam filter perfectly classifies all emails as spam or legitimate, predictive coding does not perfectly identify all relevant documents.  However, they can “learn” to identify most of the relevant documents based on one of two “learning” methods:

  • Supervised Learning: a human chooses a set of “exemplar” documents that feed the system and enable it to rank the remaining documents in the collection based on their similarity to the exemplars (e.g., “more like this”);
  • Active Learning: the system chooses the exemplars on which human reviewers make relevancy determinations, then the system learns from those classifications to apply to the remaining documents in the collection.

Tomorrow, I “predict” we will get into the strategies and the results of the TREC study.  You can check out a replay of the session at theVirtual LegalTech site. You’ll need to register – it’s free – then login and go to the CLE Center Auditorium upon entering the site (which is up all year, not just on "live days").  Scroll down until you see this session and then click on “Attend Now” to view the replay presentation.  You can also go to the Resource Center at the site and download the slides for the presentation.

So, what do you think?  Do you have experience with predictive coding?  Please share any comments you might have or if you’d like to know more about a particular topic.

eDiscovery Trends: Some SaaS Benefits for eDiscovery

I found an interesting article on Ezine Articles by Sharon Gonzalez, a freelance technical writer with 15 years experience writing on various technical subjects, especially in the areas of cloud computing, Software as a Service (SaaS), and Internet technologies.  The article entitled EDiscovery on SaaS, discusses some of the benefits of SaaS solutions for eDiscovery.

Gonzalez notes that “use of [the] eDiscovery SaaS model which has brought down the costs of many organizations” because the “model is a vendor hosted infrastructure that is highly secured and the customers can run the applications from their own machines”.  Advantages noted by Gonzalez include:

  • Easy Manageable Services: Legal teams are able to process, analyze and review data files using the eDiscovery tools from the SaaS provider via their own browser and control and secure information within those tools.  No software to install.
  • No Problem for Storage Space: The SaaS model “eliminates all requirements of added infrastructure for…increasing storage space”.  While many eDiscovery SaaS models charge a monthly fee based on data stored, that fee is eliminated once the data is no longer needed.
  • Cost-Effective Solutions Provided: Gonzalez notes “Since…the SaaS architecture is maintained by vendors, IT departments are free from the burden of maintaining it. It is also a cost-effective method as it cuts down expenditure on hiring additional IT professionals and other physical components. The companies have to pay a charge to the vendors which work out far cheaper than investing large sums themselves”.
  • Built-In Disaster Recovery: Redundant storage, backup systems, backup power supplies, etc. are expensive to implement, but those mechanisms are a must for SaaS providers to provide their clients with the peace of mind that their data will be secure and accessible.  Because the SaaS provider is able to allocate the cost for those mechanisms across all of its clients, costs for each client are considerably less to provide that secure environment.

There are SaaS applications for eDiscovery throughout the EDRM life cycle from Information Management thru Presentation.

Full disclosure: Trial Solutions is the leader in self service, on demand SaaS litigation document review solutions, offering FirstPass™, powered by Venio FPR™, for early case assessment and first pass review as well as OnDemand™ for linear review and production.

So, what do you think?  Have you used any SaaS hosted solutions for eDiscovery?  Please share any comments you might have or if you’d like to know more about a particular topic.

eDiscovery Project Management: Review the Work

Yesterday, we talked about resolving questions quickly and keeping team members informed about changes to procedures to minimize the chance for significant rework.  However, even with the best staff, mistakes happen — especially on projects that require a team of people.  There are two general types of errors you can expect:

  • Errors that are made because someone doesn’t properly understand the task.  They have misunderstood the procedures or misinterpreted a subjective component of the work.
  • Errors that are made simply because it is inevitable.  People have bad days.  They get tired.  Knowing how to do something doesn’t mean you will do it right every time.

The first type of error is easy to identify and fix.  Check initial work quickly and give immediate feedback.  I always distribute small batches of initial work at the beginning of a task – work that an individual can finish quickly.  Then I have that first initial batch for each person checked thoroughly and right away.  Misinterpretations of the procedures or the criteria are evident and can be dealt with right away, before a lot of work has been done.

Catching the second type of error is a little more difficult unless your schedule and budget permit you to check 100% of the work.  With a good staff, that’s probably overkill.  But, it’s important that work is spot-checked throughout the life of a project, and that an intelligent approach is used to isolate problems.  For example, if you find a few careless errors made by a staff member, see if you can isolate all of the work that person did on that day and check it completely.  Or perhaps you’ll identify a particular type of document or situation that caused problems, and you can take steps to isolate just those documents or situations.  Very often you can apply a systematic approach to finding and fixing errors.

What do you think?  Have you worked on projects where quality control reviews were absent or inadequate and work quality suffered?  Please share your comments or let us know if you would like more information on a topic.