Cohen’s Kappa Statistic: Definition & Example

Cohen’s Kappa Statistic is used to measure the level of agreement between two raters or judges who each classify items into mutually exclusive categories.

The formula for Cohen’s kappa is calculated as:

k = (p_o – p_e) / (1 – p_e)

where:

• p_o: Relative observed agreement among raters
• p_e: Hypothetical probability of chance agreement

Rather than just calculating the percentage of items that the raters agree on, Cohen’s Kappa attempts to account for the fact that the raters may happen to agree on some items purely by chance.

How to Interpret Cohen’s Kappa

Cohen’s Kappa always ranges between 0 and 1, with 0 indicating no agreement between the two raters and 1 indicating perfect agreement between the two raters.

The following table summarizes how to interpret different values for Cohen’s Kappa:

The following step-by-step example shows how to calculate Cohen’s Kappa by hand.

Calculating Cohen’s Kappa: Step-by-Step Example

Suppose two museum curators are asked to rate 70 paintings on whether they’re good enough to be hung in a new exhibit.

The following 2×2 table shows the results of the ratings:

Step 1: Calculate relative agreement (p_o) between raters.

First, we’ll calculate the relative agreement between the raters. This is simply the proportion of total ratings that the raters both said “Yes” or both said “No” on. 

We can calculate this as:

• p_o = (Both said Yes + Both said No) / (Total Ratings)
• p_o = (25 + 20) / (70) = 0.6429

Step 2: Calculate the hypothetical probability of chance agreement (p_e) between raters.

Next, we’ll calculate the probability that the raters could have agreed purely by chance.

This is calculated as the total number of times that Rater 1 said “Yes” divided by the total number of responses, multiplied by the total number of times that Rater 2 said “Yes” divided by the total number of responses, added to the total number of times that Rater 1 said “No” multiplied by the total number of times that Rater 2 said “No.”

For our example, this is calculated as:

• P(“Yes”) = ((25+10)/70) * ((25+15)/70) = 0.285714
• P(“No”) = ((15+20)/70) * ((10+20)/70) = 0.214285
• p_e = 0.285714 + 0.214285 = 0.5

Step 3: Calculate Cohen’s Kappa

Lastly, we’ll use p_o and p_e to calculate Cohen’s Kappa:

• k = (p_o – p_e) / (1 – p_e)
• k = (0.6429 – 0.5) / (1 – 0.5)
• k = 0.2857

Cohen’s Kappa turns out to be 0.2857. Based on the table from earlier, we would say that the two raters only had a “fair” level of agreement.

Additional Resources

You can use this Cohen’s Kappa Calculator to automatically calculate Cohen’s Kappa for any two raters.