ICC and kappa calculator for interrater reliability
Paste ratings from Excel – one row per subject, one column per rater. The tile computes the intraclass correlation for measurements or kappa for categories, with confidence intervals, a figure and a methods text. Free; data stay in your browser.
More about this tool
What you can do
- Measurements: intraclass correlation coefficient (ICC) in all six forms after Shrout and Fleiss – one-way, two-way random (absolute agreement) and two-way mixed (consistency), each for a single rating and for the mean of the raters – with confidence interval and F test.
- Two raters, categories: Cohen's kappa, with ordered categories also linearly or quadratically weighted kappa, plus observed and chance-expected agreement, PABAK and Gwet's AC1; with two categories also positive and negative agreement, prevalence and bias index.
- Three or more raters: Fleiss' kappa with bootstrap confidence interval, kappa per category, Light's kappa and Gwet's AC1.
- For the ICC: standard error of measurement (SEM) and minimal detectable change (MDC95).
- Interpretation according to Koo and Li (ICC) and Landis and Koch (kappa), with notes on pitfalls such as the kappa paradox.
How it works
Paste your data or load a CSV or TXT file: the first row holds the rater names, then one row per subject, separated by tab, semicolon or comma. Up to 20,000 rows and 40 columns are possible. Rows with missing values are excluded. The tile detects automatically whether to compute an ICC or kappa; you can override this, select the rater columns, choose the ICC model, the weighting, the order of categories and a 90%, 95% or 99% confidence level. Three examples with made-up data let you try it out.
Output and export
Results table to copy into Word or Excel or as CSV, a figure as SVG, vector PDF or PNG at 300 or 600 dpi in nine languages, and a suggested methods text. Projects can be saved and loaded again.
Methods
ICC from an analysis of variance after Shrout and Fleiss (1979), confidence intervals after McGraw and Wong (1996); Cohen's kappa with asymptotic standard error after Fleiss, Cohen and Everitt (1969); Fleiss' kappa after Fleiss (1971) with percentile bootstrap.
Data and privacy
All data stay in your browser and are not transmitted; please do not enter names or dates of birth. The tile does not replace statistical advice: check results for a publication with a second software.
Frequently asked questions
Which ICC form should I report?
Usually ICC(2,1), two-way random with absolute agreement, when raters stand for other raters and values should agree. ICC(3,1) applies when only these raters matter and systematic differences are irrelevant; ICC(1,1) when each subject was rated by different raters.
Why is kappa low although raters almost always agree?
With a very rare or very common category, kappa drops (kappa paradox). Report the observed agreement and check PABAK and Gwet's AC1.
Cohen's or Fleiss' kappa?
Cohen's kappa for two raters, Fleiss' kappa for three or more. For ordered ratings by several raters the ICC is suitable.
pubmed.info sets no tracking cookies and builds no user profiles. Visits are counted anonymously (cookieless, shortened IP addresses). Ads from Google appear only according to your consent choice.