Continuous software bug prediction is the study of how defect prediction models should be built, updated, and evaluated when a project evolves continuously and releases many versions in chronological order — instead of the randomly selected, widely spaced snapshots that populate the standard benchmark datasets. The concept comes from the ESEM 2021 paper Continuous Software Bug Prediction by Song Wang (York University, Toronto), Junjie Wang (Chinese Academy of Sciences, Beijing), Jaechang Nam (Handong Global University, Pohang) and Nachiappan Nagappan (Facebook, Seattle), published by ACM (12 pages, ESEM 2021 held 11–15 October 2021 in Bari).
The paper starts from pilot studies showing that existing benchmark datasets (PROMISE, NASA, AEEEM, SOFTLAB, ReLink, YatishData) mostly contain random release versions, and that the choice of training version can shift AUC by up to 0.359 (average 0.221) on Camel — so evaluating models on such data introduces non-trivial performance bias. It also confirms that release-based, file-level models suffer from data drift (Camel's AUC falls from 0.775 at v2 to 0.428 at v20), but that the loss is not sharp, which makes model reuse viable.
To close the gap with real-world practice, the authors propose three complementary techniques — ConBuild (training-version selection), ConUpdate (model reuse vs. update), and ConEA (effort-aware evaluation based on the change in a file's buggy probability between versions) — and validate them on 120 continuously released versions from six large-scale open-source Java systems. Bug prediction has been applied in industry by Samsung Electronics, AT&T, Microsoft Research, Google, and Cisco.
Key Points
Problem. Most existing bug-prediction benchmark datasets consist of randomly selected history versions, which does not match real projects that evolve continuously; how to predict bugs in real-world continuous development is not well studied.
Pilot Study I (benchmark bias). AEEEM and ReLink were excluded (one version per project) and SOFTLAB excluded (no version numbers). In half of the PROMISE projects and in all YatishData projects the selected versions are discontinuous; two selected ActiveMQ versions in YatishData are four years apart and skip 11 intervening releases.
Bias measurement. On Camel and Lucene, 20 continuous versions each, ADTree models, latest five versions (v16–v20) as tests, 2 projects × 85 training–test pairs = 170 experiments. AUC differences across training datasets were significant (Wilcoxon signed-rank, p < 0.05): Camel up to 0.359 (mean 0.221), Lucene up to 0.227 (mean 0.151).
Pilot Study II (performance evolution). Models trained on v1 of Camel and Lucene decline over later versions (Camel AUC 0.775 → 0.428; Lucene 0.671 → 0.508) but do not collapse — Camel is stable at 0.63 ± 0.014 across v4–v7 and Lucene similarly around v11–v13, enabling reuse to save model-building cost.
Similarity predicts performance. Using standard-deviation vectors and cosine similarity, the Spearman rank correlation between training/test distributional similarity (vs. v1) and test-version AUC was 0.84 on Lucene and 0.91 on Camel.
Three approaches. ConBuild selects training versions by distributional similarity; ConUpdate decides Reuse vs. Update; ConEA re-ranks files for inspection by the difference in buggy probability between consecutive versions.
Result. ConBuild finds an applicable model in 78.3% (47/60) of runs across the 60 test versions (v11–v20 of six projects), significantly outperforming FSSBagging and RandomBuild; failures cost only 5.5%–14.5% AUC (average 8.6%).
Bug prediction pipeline. Files touched by bug-fixing changes are labeled buggy; all others clean. Defect features are collected for those files, and instances (features + labels) train machine-learning classifiers that predict files in a later version as buggy or clean.
Training version / test version. The release version used to build a model is the training version; the release version used to evaluate it is the test version.
Best / applicable / inapplicable models. The best model achieves the best AUC on a given test version; applicable models are the best model plus any model within 5% deviation of it (5% being an acceptable deviation in statistics); all others are inapplicable.
Distributional characteristics. The authors describe a dataset's distribution by 11 statistical indicators (mode, median, mean, minimum, maximum, first quartile, third quartile, variance, standard deviation, skewness, kurtosis) computed over its bug prediction metrics; combined with 54 metrics these form a 594-dimensional vector (54 × 11). Cosine similarity measures distance between such vectors (following prior work and the pilot study).
Data drift. As a project evolves, the distribution of software artifacts (bug types, code changes, functionalities) and of features across release versions changes, degrading models trained on earlier versions.
Effort-aware evaluation. Because developers can inspect only a limited number of lines of code (LOC), predictions are ranked and only the top portion is inspected. Existing approaches rank files by buggy probability within one version; ConEA ranks by the evolution of fault proneness between versions.
Metrics. AUC measures model performance; Accuracy measures the correctness of ConBuild/ConUpdate recommendations; PofB20 measures the percentage of bugs a developer can find by inspecting the top 20% of LOC (higher is better).
Dataset terminology — a noted ambiguity. In the paper's related work, "CPBP" appears as cross-project bug prediction (in the description of FSSBagging), while elsewhere it denotes continuous bug prediction; the two senses appear in the same document.
Details
Motivation and pilot studies
Software bug prediction plays a fundamental role in quality assurance, guiding bug detection and reducing development cost, but the benchmark datasets used to validate it were built by heuristic collection — identifying post-release defects via keywords or issue IDs within a post-release window (e.g., six months). The popular benchmarks are: PROMISE (71 release versions, 33 open-source projects), NASA (12 versions, 4 projects), AEEEM (5 versions, 5 projects), SOFTLAB (3 versions, 1 project), ReLink (3 versions, 3 projects), and YatishData (32 versions, 9 projects; uses affected releases recorded in issue trackers to collect defect data more accurately per bug). YatishData is the most continuous-aware, but still discontinuous per project.
Pilot Study I measured whether the selected versions reflect continuous development and how much the arbitrariness costs. All YatishData projects and half of the PROMISE projects have discontinuous version selections. The 170 training/test experiments on Camel and Lucene showed statistically significant performance differences depending on the training dataset, with AUC gaps up to 0.359/0.227 as described above — meaning randomness in data collection can introduce non-trivial performance bias.
Pilot Study II tested whether file-level, release-based models degrade over time as reported for buggy-commit models and monthly bug-frequency models. Training on v1 of Camel and Lucene and testing on v2–v20 confirmed decline (Camel 0.775 → 0.428; Lucene 0.671 → 0.508) due to data drift, but with plateau periods where performance stayed stable (Camel 0.63 ± 0.014 on v4–v7; Lucene around v11–v13). One likely cause is differing distribution of features among release versions. The pilot then tested the assumption that training versions whose distribution resembles the test version yield better performance: with v1 trained models on Camel and Lucene, the Spearman rank correlation between model AUC on each test version and the cosine similarity of standard-deviation vectors (v1 vs. test version) was 0.91 (Camel) and 0.84 (Lucene). These results motivate all three proposed techniques.
The three approaches
Rendering diagram…
ConBuild (model building, §3.1). Given a test version vn, ConBuild computes the distributional characteristics of vn and of every history version v1…vn−1 (54 bug prediction metrics × 11 indicators from Table 1), uses cosine similarity to rank the history versions, and recommends the top-ranked version as the training version. Algorithm 1: input vn and Vtrainings; for each v extract indicators, compute similarity s to vn, store (v, s) in map S; sort S by s; return the top one. The 11 indicators are mode, median, mean, minimum, maximum, first quartile, third quartile, variance, standard deviation, skewness, and kurtosis.
ConUpdate (model reuse/update, §3.2). For a newly released version vn, ConUpdate recommends either Reuse of the model already applied to vn−1 or Update (relabel, collect features, retune, and rebuild using ConBuild). It ranks all previous versions by similarity to vn; if both vn−1 and the training version of vn−1's applied model fall in the top k (1 < k < n), it recommends Reuse; otherwise Update. Algorithm 2: inputs vn, vn−1, Appliedn−1 (training version of vn−1's applied model), Vall, threshold k; build and sort map S by similarity; return Reuse if vn−1 and Appliedn−1 are in the top k, else Update.
ConEA (effort-aware evaluation, §3.3). Existing effort-aware approaches rank files within a version by buggy probability and ignore the evolution of fault proneness. ConEA assumes a file whose fault proneness changes dramatically between versions is a potential risk and should be inspected earlier. It computes, for the predicted buggy files of vn, the probability difference relative to vn−1, ranks files by that difference, and sets inspection order accordingly. Files that are new in vn (e.g., from new features) and absent from earlier versions use their vn buggy probability as the difference and are ranked together with the rest. The motivating example: f1 goes from 0.0 (v1) to 0.51 (v2) while f2 goes from 0.52 to 0.521; existing approaches rank f2 first because 0.521 > 0.51, but ConEA ranks f1 first because its difference (0.51) far exceeds f2's (0.01). Algorithm 3: inputs predicted results Rn and Rn−1; for each file f in Rn, if f was in Rn−1 and truly buggy there, store (f, diff_f = probability difference); if f is not in Rn−1 and is predicted buggy in Rn, store (f, probability in Rn); sort the map by diff and return it.
Experiment setup
Data were collected following the paper's own pipeline (heuristic bug-fixing commit detection with regular expressions over messages containing fix(e[ds])?, bugs?, defects? plus issue/report IDs). Six Java projects with the 20 most recent continuous release versions before 2018-12-31 were used — 120 versions total. Two projects come from PROMISE (Ant, Poi), two from YatishData (ActiveMQ, HBase), and two are shared by both (Lucene, Camel):
Project
Description
Files
KLOC
Bug rate
Releases
Ant
Java build framework
0.71K–0.86K
94–107
9.7%–16.4%
v1.7.1–v1.9.13
Poi
Java library for MS Office docs
1.8K–2.3K
199–244
7.2%–13.1%
v3.10.1–v4.0.1
ActiveMQ
Java message broker
19K–36K
142–299
6.2%–15.5%
v5.12.3–v5.15.8
HBase
Distributed scalable data store
1.5K–1.8K
364–525
6.1%–20.5%
v1.1.10–v2.0.3
Lucene
Text search engine library
3.9K–4.5K
497–643
4.1%–6.2%
v5.5.4–v7.6.0
Camel
Enterprise integration framework
4.6K–7.7K
133–362
3.3%–9.6%
v2.17.7–v2.23.0
Metrics (§4.2). The goal is not to find the best metrics, so widely used ones are adopted: lines of code, code complexity (e.g., McCabe Cyclomatic), and object-oriented metrics (e.g., coupling between object classes), collected with SciTools' Understand, which documents each metric. Method-level metrics are aggregated to file level with three schemes (min, max, average), following YatishData. Table 3 lists 37 file metrics (AvgCyclomatic, AvgCyclomaticModified, AvgCyclomaticStrict, AvgEssential, AvgLine variants, AvgLineBlank/Code/Comment, CountDeclClass, CountDeclClassMethod, CountDeclClassVariable, CountDeclFunction, CountDeclInstanceMethod, CountDeclInstanceVariable, CountDeclMethod plus Default/Private/Protected/Public variants, CountLine variants, CountSemicolon, CountStmt variants, MaxCyclomatic variants, RatioCommentToCode, SumCyclomatic variants, SumEssential), 5 class metrics (CountClassBase, CountClassCoupled, CountClassDerived, MaxInheritanceTree, PercentLackOfCohesion), and 12 method metrics (CountInput, CountOutput, CountPath, MaxNesting, each with Min/Mean/Max).
Baselines (§4.4).FSSBagging — the state-of-the-art training-version selection approach for cross-project bug prediction — uses a different similarity measure: it builds a synthetic dataset from a candidate training version plus the test version, trains logistic regression on it, uses that classifier's accuracy as the similarity, picks the top-N training versions from different projects, filters unstable features, and bags the resulting models. RandomBuild picks a random prior release version with no selection strategy. EA (the effort-aware baseline) sorts test files by predicted buggy probability, accumulates LOC until 20% of the test set's LOC is inspected, and reports the percentage of buggy files found as PofB20; ConEA computes PofB20 the same way but sorts files by the between-version probability difference instead.
Research questions (§4.5). RQ1: To what degree can ConBuild find the appropriate training version for model building in continuous development? RQ2: How effective is ConUpdate in guiding model reuse and update? RQ3: How much can ConEA improve existing effort-aware bug prediction approaches? RQ1 covers Algorithm 1, RQ2 Algorithm 2, RQ3 Algorithm 3.
Results
RQ1 / ConBuild (§5.1). The latest ten versions (v11–v20) of each project serve as test datasets; for each test version, earlier versions with bug information available at that time were labeled, features collected, and ADTree models built and tuned to establish best/applicable/inapplicable ground truth. Across the 60 experimental test versions, ConBuild found an applicable model in 78.3% (47/60) and failed in 21.7% (13/60); failed recommendations still stayed within 5.5%–14.5% of the best model (average 8.6%). ConBuild outperformed both FSSBagging and RandomBuild on all six projects (Figure 4), and a Wilcoxon signed-rank test showed the improvement to be significant. FSSBagging used its recommended parameters with ADTree; RandomBuild was repeated 100 times per test version to remove bias. A proposed reason for FSSBagging's weakness in continuous settings is that its core assumption — that combining models built from different projects beats a model built from insufficient same-project data — does not hold when a project has ample historical training data.
Per-project outcomes on v11–v20 (Table 4; √ = applicable model from the recommended training version, × = unsuccessful with AUC gap vs. best in parentheses):
RQ2 / ConUpdate. ConUpdate (Algorithm 2) was evaluated with the same configuration as RQ1: the latest ten release versions (v11–v20) as test datasets, with each test dataset's earlier release versions labeled using bug information available at that release.
RQ3 / ConEA. ConEA is evaluated with PofB20, differing from the EA baseline only in that files are ranked by the between-version difference of predicted buggy probability rather than by the probability itself.
Related work and positioning
Bug prediction research uses features manually extracted from labeled historical defect data to train ML classifiers; features span static code features (Halstead, McCabe, MOOD), process features, semantic features, and context metrics. Most such studies were evaluated on PROMISE, NASA, AEEEM, SOFTLAB or ReLink — datasets with randomly selected discrete versions, so their evaluation can carry non-trivial bias. Prior cross-version work includes Kastro et al. (change information to predict the number of bugs in a new version), Krishna et al. (time-series forecasting of next-month bug/enhancement reports from the last four months), and Liu et al. (differences of code metrics between two versions, evaluated on PROMISE). Unlike these, this paper targets file-level bug prediction across building, updating, and evaluation in continuous development.
On effort: Mende et al. first introduced effort into a bug prediction model, and many effort-aware models followed, with file LOC as the de facto effort measure and files ranked by buggy probability; other work estimates the time required to fix bugs. ConEA instead leverages the evolution of buggy probability between sequential versions. Prediction granularity divides related work into file-level, change-level (just-in-time), and method-level; as the first study of continuous bug prediction, this paper deliberately focuses on file level only.
Threats to validity. Internal: only code metrics produced by SciTools' Understand were used, which — although large and diverse — does not include every metric reported in the literature. External: all subjects are Java open-source projects from the Apache Software Foundation; findings may not generalize to commercial projects or other ecosystems. Construct/conclusion: results rely on a single defect-prediction scenario (within-project file-level prediction), so conclusions may differ for change-level or method-level prediction.
Conclusion and impact. The paper revisits bug prediction for continuous development: ConBuild redefines training-data selection via distributional characteristics, ConUpdate guides reuse/update via the evolution of those characteristics, and ConEA redefines effort-aware evaluation via the evolution of file buggy probabilities. Experiments on 120 continuous releases across six large-scale open-source systems demonstrate practical value, and the work opens the door to rethinking other software analytics models — effort estimation, reviewer recommendation, and bug triage — in continuous development. The work was supported by Canada's NSERC and China's NSFC (grant No. 62072442).
Reference clusters (selected)
The bibliography spans classic defect prediction and bug/change linking (Śliwerski et al. FSE'05; Wu et al. "Relink"), cross-company and cross-project prediction (Turhan et al. EMSE'09; Zhang et al. MSR'04/ICSE'16; Zimmermann et al. FSE'09), effort-aware and just-in-time prediction (Yang et al. ASE'16 and FSE'16), deep/semantic learning for defect prediction (Wang et al. TSE'18 and ICSE'16; Yang et al. QRS'15), parameter optimization and practice pitfalls (Tantithamthavorn et al. ICSE'16, ICSE-SEIP'18), release-aware data selection (Yatish et al. ICSE'19; Zhang et al. "CDS" IEEE Access 2020), network/dependency-graph and security defect prediction (Zimmermann et al. ICSE'08, ICST'10), fault localization (Zou et al. TSE'19), crowdsourced testing (Wang et al. ASE'16, TSE'19), and the Witten et al. data-mining textbook (Morgan Kaufmann).