Ranking pull request reviewers by expertise, workload, and expected response time

Abstract

Relevance. Reviewer rankings are usually judged by whether the suggested person later reviewed the pull request, which says little about how widely recommendations spread or how fast that person responds. Purpose of the article. We ask how accurately reviewers can be ranked when a pull request is created, and what accounting for concentration and expected response time costs in accuracy. Objectives. We build candidate pools from prior reviews, compare a learned ranker with simple rules, test it on unseen repositories, and compare two ways of balancing other criteria. Methods. We use merged pull requests from twelve open-source repositories. Chronological splits limit each recommendation to earlier information. We train a gradient-boosted ranker, measure whether an actual reviewer appears among the first three candidates, and calculate bootstrap confidence intervals. Scientific novelty. We compare a weighted penalty with a rule that reorders only candidates near the top relevance score, reporting accuracy, the spread of first choices, and estimated response time together. Practical significance. A model trained elsewhere can serve a new project once it has its own review history, and a tolerance parameter bounds departures from the relevance-only ranking. Results. On the test set, the ranker's first three choices included an actual reviewer for about two-thirds of pull requests, against under half for the most-active rule; ranking the author's past reviewers already captures most of that gain. In leave-one-repository-out tests, accuracy fell by under one percentage point. The tolerance rule put a third more people in first place or nearly halved the first choice's median estimated response time, each for under one percentage point of accuracy. No nonzero weighted penalty met the validation accuracy threshold. Conclusions. Balancing workload or expected response time costs little within a tolerance of the best relevance score and much when applied to every candidate. Whether this shortens actual waiting time remains to be tested in a live pilot.