Building better AI benchmarks: How many raters are enough?

In the paper 'Forest vs Tree: The (N, K) Trade-off in Reproducible ML Evaluation,' Google Research studies the trade-off in AI evaluation between the number of annotations per item and the number of items annotated, and open-sources a simulator based on real datasets.