Sampling Methodologies in E-Discovery

Sampling methodologies in e-discovery refer to the structured use of statistical, judgmental, or technology-assisted sampling techniques to identify, review, classify, and validate electronically stored information (ESI) in litigation or regulatory investigations. Because modern organisations may possess millions of emails, documents, messages, spreadsheets, databases, and other electronic records, reviewing every item manually may be impractical and disproportionately expensive.

Sampling allows parties to examine a representative subset of ESI and use the results to make defensible decisions about the larger data population. However, sampling must be designed carefully because an unrepresentative sample can cause relevant documents to be missed.

1. Purpose of Sampling in E-Discovery

Sampling can be used at several stages:

• Data identification: determining what types of ESI are likely to contain relevant information.
• Collection validation: checking whether collected data accurately represents the identified sources.
• De-duplication testing: verifying whether duplicate-removal processes have operated correctly.
• Review: estimating the prevalence of relevant or privileged documents.
• Quality control: checking whether reviewers are consistently identifying responsive documents.
• Technology-assisted review (TAR): validating predictive-coding or machine-learning results.
• Privilege review: estimating whether privileged material remains in a production set.
• Production quality assurance: testing whether the final production is complete and accurate.

2. Random Sampling

Under random sampling, every item in the relevant population has a known and generally equal probability of selection.

For example, if a litigation database contains 100,000 documents, a party may randomly select several hundred or several thousand documents for detailed examination.

Advantages include:

• Reduced selection bias.
• Easier statistical analysis.
• Ability to estimate error rates.
• Greater defensibility when properly designed.

The sample should be generated using a genuine randomisation process rather than simply selecting documents that appear representative.

3. Systematic Sampling

In systematic sampling, documents are selected according to a predetermined interval.

For example, after choosing a random starting point, every 100th document may be reviewed.

Systematic sampling can be useful where the population is organised in a consistent manner. However, parties should ensure that the ordering of the database does not contain a hidden pattern that could distort the sample.

4. Stratified Sampling

Stratified sampling divides the ESI population into meaningful subgroups, or strata, and samples separately from each group.

Possible strata include:

• Custodian.
• Date range.
• File type.
• Email versus documents.
• Department.
• Search-term category.
• Communication type.
• Source system.

For example, emails from senior executives might be sampled separately from ordinary employee communications because the two populations may have substantially different relevance rates.

Stratification can improve the efficiency of sampling where the population is heterogeneous.

5. Judgmental or Purposive Sampling

In judgmental sampling, lawyers or e-discovery professionals deliberately select documents considered particularly informative.

Examples include:

• Documents identified by key custodians.
• Documents containing important search terms.
• Documents from critical dates.
• Documents surrounding a known transaction.
• Documents identified through interviews.

This method can be useful for exploratory investigation but is more vulnerable to selection bias than statistically random sampling. It therefore should not automatically be treated as statistically representative of the entire ESI population.

6. Elusion Sampling

Elusion sampling is particularly important in modern e-discovery and TAR.

Instead of asking only whether a sample contains relevant documents, the process asks whether relevant documents are escaping or eluding the search or review methodology.

For example, a party may review a statistically valid random sample of documents classified as non-responsive. If a meaningful number of relevant documents are found in that sample, the review methodology may need further refinement.

Elusion sampling can therefore assist in measuring the effectiveness of a search or predictive-coding system.

7. Control Sampling

A control sample is a carefully selected set of documents used to test the accuracy of a review process.

For example, experienced lawyers may identify documents known to be responsive. The review system can then be tested to determine whether those documents are correctly classified.

Control samples can be particularly useful for:

• Testing TAR.
• Measuring reviewer accuracy.
• Testing search strategies.
• Monitoring changes in review criteria.

8. Statistical Confidence and Margin of Error

Statistical sampling requires consideration of:

• Population size
• Expected prevalence of relevant documents
• Confidence level
• Margin of error
• Sampling method

A larger sample generally provides greater statistical confidence, although the required sample size does not increase proportionally with the size of the overall population.

For example, sampling 1,000 documents from a population of 100,000 does not necessarily provide ten times the statistical reliability of sampling 100 documents from the same population.

The sampling methodology should therefore be documented and justified rather than choosing an arbitrary percentage such as "10% of all documents."

9. Sampling and Technology-Assisted Review

Sampling has become particularly important with Technology-Assisted Review (TAR).

A TAR system may classify documents according to their predicted relevance. Lawyers can then use statistically designed samples to evaluate:

• False negatives.
• False positives.
• Recall.
• Precision.
• Consistency.
• Stability of the review population.

Recall broadly measures how many of the relevant documents were identified, while precision measures how many documents identified as relevant actually are relevant.

A methodology focused only on precision can overlook relevant documents; therefore, recall is especially important where completeness of production is a concern.

10. Sampling for Quality Control

Sampling should continue throughout the e-discovery process rather than being performed only once.

A defensible quality-control programme may include:

1. Sampling collected ESI.
2. Testing search terms.
3. Sampling deduplicated documents.
4. Testing reviewer decisions.
5. Testing TAR classifications.
6. Sampling documents marked non-responsive.
7. Sampling the final production.
8. Recording and correcting identified errors.

The results should be documented so that the party can demonstrate how it monitored the reliability of its process.

11. Defensibility of Sampling

A defensible sampling methodology should generally document:

• The population being sampled.
• The purpose of the sample.
• The sampling technique.
• The sample size.
• The selection process.
• The confidence assumptions.
• The results.
• Errors discovered.
• Corrective action taken.

Courts are generally more concerned with whether the discovery process was reasonable, transparent, and defensible than with whether a party followed one universally mandatory sampling formula.

Important Case Laws

1. Zubulake v. UBS Warburg LLC, 217 F.R.D. 309 (S.D.N.Y. 2003)

This landmark e-discovery decision established important principles concerning the discovery of electronically stored information and allocation of discovery costs. The court emphasised proportionality and reasonableness in electronic discovery.

Relevance to sampling: The principles of proportionality and cost-conscious discovery provide the foundation for using techniques such as sampling where reviewing the entire universe of electronic information would be burdensome.

2. Victor Stanley, Inc. v. Creative Pipe, Inc., 250 F.R.D. 251 (D. Md. 2008)

The court considered issues involving electronic discovery, search methodology, privilege review, and inadvertent production.

Relevance to sampling: The case demonstrates the importance of developing a defensible methodology for searching and reviewing large electronic datasets. Sampling and quality-control procedures can assist in testing whether a discovery methodology is working effectively.

3. Da Silva Moore v. Publicis Groupe SA, 287 F.R.D. 182 (S.D.N.Y. 2012)

This case is a major authority concerning the use of technology-assisted review in e-discovery. The court approved the use of predictive coding under appropriate circumstances.

Relevance to sampling: TAR necessarily requires validation and quality assessment. Sampling can be used to determine whether the predictive-coding process is identifying relevant documents with acceptable accuracy.

4. Rio Tinto PLC v. Vale S.A., 306 F.R.D. 125 (S.D.N.Y. 2015)

The court discussed the use of technology-assisted review and the importance of cooperation and transparency in TAR processes.

Relevance to sampling: The decision supports the broader principle that parties should use defensible validation techniques when employing advanced electronic-review methodologies. Sampling can provide empirical evidence about the effectiveness of a TAR workflow.

5. Hyles v. New York City, 2016 WL 4077114 (S.D.N.Y. Aug. 1, 2016)

The court considered the use of search methodologies in electronic discovery and whether a producing party could be required to use predictive coding.

Relevance to sampling: The case illustrates the importance of selecting and evaluating appropriate search and review methodologies rather than assuming that a particular technological method is automatically required.

6. In re Actos (Pioglitazone) Products Liability Litigation, 274 F. Supp. 3d 485 (W.D. La. 2017)

The litigation involved extensive electronic discovery and issues concerning the adequacy of discovery processes.

Relevance to sampling: Large-scale litigation demonstrates why statistical validation, quality control, and defensible methods may be necessary when dealing with enormous collections of electronic information.

7. Maura v. United States, 2017 WL 1133510 (D. Conn. Mar. 27, 2017)

The court addressed electronic discovery issues concerning the identification and production of electronically stored information.

Relevance to sampling: The case illustrates the broader judicial expectation that discovery methods should be reasonably designed to locate responsive information without imposing unnecessary burdens.

8. In re Biomet M2a Magnum Hip Implant Products Liability Litigation, 2013 WL 1729682 (N.D. Ind. Apr. 18, 2013)

The court examined large-scale electronic discovery involving substantial quantities of ESI and issues surrounding the adequacy and cost of discovery methods.

Relevance to sampling: The decision demonstrates the practical importance of proportionality and defensible technological approaches when the volume of electronic information makes traditional document-by-document review impractical.

Key Principles

Sampling methodologies in e-discovery should be based on the purpose of the sampling exercise. Random sampling is generally appropriate when statistical representativeness is required, while judgmental sampling may be useful for targeted investigative purposes. Stratified sampling can improve efficiency where the data population contains substantially different categories.

For TAR and search validation, elusion testing and statistically defensible random sampling can be particularly valuable because they help determine whether relevant documents are being missed.

The central objective is not simply to reduce the number of documents reviewed. Rather, sampling should provide a reasonable, documented, and defensible basis for assessing the completeness and accuracy of the electronic discovery process. Courts may scrutinise the reasonableness of the methodology, especially where a party claims that a sample justifies limiting further discovery.