Pass@k: A Practical Metric for Evaluating AI-Generated Code

Measuring how well generative AI models really perform is a tough job, and it’s still far from solved. These models can spit out all kinds of results, so people have come up with all sorts of ways to judge how good or useful the outputs are, depending on the situation.

When it comes to AI-made code, figuring out if the code actually works just adds another headache to the mix. One of the newer solutions in this area is something called the pass@k metric. It’s basically become a go-to way for checking if code that comes from large language models actually runs like it’s supposed to.

I don't know why its writing in reverse honestly.

In this blog, I’m going to dig into what pass@k really means, why it’s such a big deal for generating code, what problems it helps sort out, and how it stacks up against older ways of rating AI code.

TL; DR

Pass@k is a metric that checks whether an AI model can produce at least one correct code solution (out of k tries) for a given problem. Unlike text-matching metrics that can penalize perfectly valid but differently written solutions, Pass@k uses functional correctness (unit tests) to evaluate code. This better reflects real-world coding, where multiple attempts may be tried before a working solution is found. The latest research showed that earlier approaches to measuring Pass@k were biased, and introduced an unbiased estimator, which Hugging Face implements. Overall, Pass@k captures a model’s practical value: the probability of generating a correct solution in k shots, focusing on whether the code actually works rather than how closely it matches a reference.

What is pass@k? (A Simple Explanation)

Pass@k is a way to measure how often a code generation system can produce at least one correct solution (out of several tries) to a given coding problem. Instead of checking whether just one generated sample is correct, Pass@k allows multiple attempts and considers the problem solved if any of those attempts passes the testing criteria (for example, a set of unit tests). This approach can better capture how a developer might interact with an AI coding assistant by generating and reviewing several candidate solutions until finding one that works.

Formally, Pass@k was introduced in the context of functional correctness. Given a model that generates code, you sample k different solutions for each programming prompt. If any one of those k samples runs successfully (i.e., passes the tests associated with that problem), then the model is considered to have “passed” that prompt. The fraction of prompts on which the model passes is reported as Pass@k.

This metric acknowledges the reality that code generation often involves trying multiple solutions, especially when solving challenging tasks. By reflecting the chance of generating at least one working sample among k tries, Pass@k gives a clearer sense of a model’s practical usefulness than only checking whether the first try is correct.

• Suppose there are 10 coding problems in an evaluation, and a model generates k=5 solutions for each problem. If 7 of those 10 problems have at least one solution that passes all unit tests, then Pass@5 is 70%.
• If a single solution per problem were tested (k=1) and the model got 3 correct out of 10 problems, Pass@1 is 30%. If, by using five tries (k=5), the model succeeds on 7 total problems, that higher score (70%) indicates the real benefit of generating multiple samples.

When n total code samples are actually generated (with n ≥ k) and c of those n samples are correct, an unbiased way to estimate Pass@k uses the formula:

In words:

• n is the total number of generated samples for a given problem.
• c is the number of correct (passing) samples out of those n.
• “(n choose k)” is the number of ways to choose k distinct samples out of n.
• “(n−c choose k)” counts how many k-combinations contain only incorrect solutions.

Hence, ((n−c) choose k / (n choose k)) is the fraction of ways to pick k samples that are all wrong. Subtracting that fraction from 1 estimates the probability that at least one correct sample appears in any draw of k solutions. Because it directly captures the chance of getting a correct solution among k tries without double-counting or bias this formula is especially suitable for benchmarking code generation models.

Calculating Pass@k via Code

One practical way to compute Pass@k is to implement the formula using combinations. Below is a snippet adapted from Hugging Face’s “evaluate” library, illustrating the procedure:

def estimate_pass_at_k(num_samples, num_correct, k):
    """Estimates pass@k of each problem and returns them in an array."""

    def estimator(n: int, c: int, k: int) -> float:
        """Calculates 1 - comb(n - c, k) / comb(n, k)."""
        if n - c < k:
            return 1.0
        return 1.0 - np.prod(1.0 - k / np.arange(n - c + 1, n + 1))

    if isinstance(num_samples, int):
        num_samples_it = itertools.repeat(num_samples, len(num_correct))
    else:
        assert len(num_samples) == len(num_correct)
        num_samples_it = iter(num_samples)

    return np.array([estimator(int(n), int(c), k) for n, c in zip(num_samples_it, num_correct)])

The function iterates over each problem, which has two important numbers:

n (num_samples): total code samples generated for that problem.

c(num_correct): how many of those samples pass the tests.

• For each problem, it calls estimator (n, c, k), which in turn uses the formula “1 − ((n−c choose k) / (n choose k))” but written with a product term (np.prod) to avoid some numerical instability when directly dealing with combinations.
• If n−c < k, the function returns 1.0, meaning that because there aren’t enough incorrect samples to fill a k-combination of failures, it’s certain (100% chance) that at least one of the k picks would be correct.
• The final outcome is an array of floating-point numbers, each representing the pass@k result for one problem in the dataset. That array can then be averaged or otherwise aggregated to measure performance across all problems.

Why Pass@k Is Needed, and How Things Were Done Before?

Before Pass@k gained traction for evaluating code generation, a common approach was to use string-based or semantic matching metrics originally designed for natural language tasks (for example, BLEU or ROUGE). These metrics rely heavily on the similarity between a generated snippet of code and a single reference solution, treating the problem like a translation or text-generation task. However, these methods miss a crucial point about code: two programs can look very different yet still produce exactly the same correct results (i.e., be functionally equivalent).

Shortcoming of Text-Based Metrics

• Exact Match: Early attempts sometimes used exact match (i.e., does the generated code match a reference solution word for word?). This is too strict for code because many correct solutions might differ in variable names, code style, or function structure.
• Semantic or Fuzzy Matching: More sophisticated text-based metrics, such as BLEU, try to measure approximate similarity to a reference. Yet code can be correct with little to no textual overlap. For instance, the same functionality can be implemented with different algorithms or libraries.

Why Functional Equivalence Matters

• Multiple Correct Solutions: In programming, one problem may have numerous correct implementations. A text-matching metric may penalize a perfectly valid solution if it diverges from the reference in style or approach.
• Real-World Code Testing: In practice, developers care whether a function or script works correctly for all required inputs — exactly the question that functional evaluations (like passing unit tests) answer. This insight led researchers toward measuring correctness via tests rather than text comparisons.

Enter Pass@k: Measuring Functional Correctness

• Reflecting Actual Coding Workflows: People naturally try multiple solutions or variants of a single solution when coding. Pass@k aligns with this reality by letting a model generate several attempts (k samples) and deeming the problem solved if at least one of them passes the tests.
• Reduced False Negatives: Unlike text-based checks that might dismiss a valid but differently written program, Pass@k directly evaluates whether code solves the problem’s logic. This reduces the risk of missing good solutions that fail only on superficial textual criteria.
• A More Direct Metric: By focusing on unit tests (or more extensive test suites), Pass@k transcends the pitfalls of semantic matches. If the code runs and meets the specification’s requirements, it is correct — regardless of how similar or dissimilar it looks to a reference solution.

In a nutshell, Pass@k emerged as the preferred alternative to purely text-based or semantic matching metrics because it measures what really matters in a coding context: getting the job done correctly. By counting how often the model is able to produce a correct solution in k tries, Pass@k offers a clearer picture of a code-generation system’s practical usefulness for developers.

What Is a Biased vs. Unbiased Estimator?

Going a little off topic but stay with me. (ㅅ´ ˘ `)

An estimator is simply a rule or formula that uses sample data to approximate some underlying “true” quantity (often called a parameter). In statistics, an estimator can either be:

• Biased: On average, it systematically overestimates or underestimates the true parameter we want.
• Unbiased: On average, it gets the true parameter right — meaning the expected value of the estimator equals the actual parameter over many theoretical repeated samples.

Simple Explanation

• Imagine you want to guess the average height of all the people in your city. You measure 50 people randomly and compute the mean height among them. If done properly, that sample mean is an unbiased estimator of the city’s true average height: on average, you neither overshoot nor undershoot.
• If, however, you only measured very tall people — maybe because you only sampled from a basketball team — then your sample mean would likely overestimate the city’s average height. That would be a biased estimator because it systematically produces a bigger number than the true population mean.

Examples

• Sample Mean for Estimating the True Mean: If you pick a random sample from a population and compute the average, that sample mean is an unbiased estimator of the true population mean.
• Sample Variance for Estimating Population Variance: Using the formula with “n − 1” in the denominator (rather than “n”) is known to be an unbiased estimator of the population variance. This small correction ensures that, on average, you neither overestimate nor underestimate how spread out the data really is.

Why It Matters

• Fairness and Accuracy: Unbiased estimators help ensure that, in the long run, your guesses will be right on average. This is particularly important in scientific studies or any context where you need trustworthy estimates.
• Comparing Methods: If you have two methods for estimating, say, the average height or the probability of passing a coding test, it often matters whether those methods systematically drift high or low, or whether they’re unbiased.

In essence, an unbiased estimator aligns best with the true value over many hypothetical repetitions. Meanwhile, a biased estimator consistently skews results up or down. In practice, even biased estimators can sometimes be useful if you understand and can correct for the bias, but unbiased estimators are often preferred for clearer interpretation and reliability.

Why Pass@k Can Be Biased, and How It’s Corrected

Earlier work (for example, Kulal et al.) introduced a straightforward way to estimate Pass@k, but it could be biased. One common “short-cut” is to estimate Pass@k by 1 − (1 − p̂) ᵏ, where p̂ is simply the empirical probability of getting a correct solution from a single try (Pass@1). Unfortunately, this shortcut assumes all samples are drawn independently with replacement, which leads to systematic underestimates or overestimates when you only have a limited number of samples n. In short, the naive approach often does not accurately reflect the true likelihood of finding at least one correct sample among k tries.

The Naive (Biased) Approach

• Often expressed as 1 − (1 − Pass@1) ᵏ.
• Assumes sampling with replacement and can either underestimate or distort results when n is small.
• Can yield misleadingly good (or bad) results if one strictly relies on the fraction of correct single samples (Pass@1) rather than the actual combinations of correct and incorrect code generated out of n attempts.

Unbiased Estimator from the Codex Paper

• The paper “Evaluating Large Language Models Trained on Code” (Chen et al.) provides an approach that measures the fraction of ways to pick k failures (incorrect samples) from the total draws, ensuring no incorrect-sample set is double-counted or over-counted.
• In formula form: Pass@k = 1 − ((n−c choose k) / (n choose k)), where n is total generated samples, c is how many of them are correct, and “(n choose k)” is the number of ways to choose k distinct samples out of n.

Why the Hugging Face Implementation Is Unbiased

• The Hugging Face code snippet (estimate_pass_at_k) explicitly uses 1 − comb (n — c, k) / comb (n, k) (in practice, they implement it with a product term to avoid large-number instability).
• That means it is sampling without replacement, reflecting the real scenario: we draw up to k unique tries from the n generated samples.
• Hence, it directly matches the unbiased formula described by Chen et al. in the Codex paper.

Why Correcting Bias Is Important

• Fair Comparisons: In research or model benchmarking, a biased estimate can make one system appear better (or worse) simply due to how Pass@k was computed. Having an unbiased metric ensures apples-to-apples comparisons across different models and sample counts.
• Proper Evaluation: Code-generation systems can produce a large variety of partially correct or incorrect samples. Counting exactly how often at least one valid solution emerges (out of k attempts) is the metric that best reflects practical coding scenarios — so it must be an accurate reflection.

In summary, using a naive estimate of Pass@k (e.g., 1 − (1 − Pass@1) ᵏ) risks bias due to independence assumptions and sampling with replacement. The corrected formula from the Chen et al. Codex paper also implemented in Hugging Face’s evaluate library — avoids this bias by calculating the chance of selecting only failures without replacement, making the final measurement truly unbiased.

Thank you for reading!