CHID is a change impact analysis (CIA) approach and supporting tool that estimates the impact of changes introduced by a pull request (PR) and surfaces the result inside the code review workflow. Its distinguishing feature is that it combines mining software repositories (MSR) with dependency analysis using call graphs at the PR level, targeting modern code review. Unlike DIVER (Cai and Santelices, 2014) and CPCIA (Dai et al., 2022), which work at method or statement granularity, CHID analyzes the entire PR including its associated files and changesets; MSR tools such as ROSE target long-term change-propagation trends rather than real-time review, and FaultTracer and JRipples perform static/dynamic analyses for fault localization and dependency tracking without using MSR data. Combining MSR with dependency analysis is intended to give developers real-time insight during review, aligning with CI practices. CHID was assessed through a focus group study, a post-demo survey, and computational feasibility experiments on three open-source projects.
Key Points
CHID forecasts a PR's potential impact by mapping changed methods onto the project call graph and highlighting their neighbors as potentially affected.
Impact is defined at method level: methods invoking a changed method are potentially affected, and the effect ripples further if those methods are themselves called elsewhere.
Impact propagation is bounded to up-to-third-degree neighbors for visual intelligibility, because GitHub comments accept only static images and large graphs become hard to read.
An impact graph showing changed plus potentially impacted methods is posted to the PR, and an impact size metric quantifies the reach of the change using PageRank over the project call graph.
An overall risk score for the PR is a weighted sum of metric categories, intended to capture risks introduced by the proposed changes (similar to post-release defect risk).
The call graph is kept current by an incremental update algorithm run after a PR is merged, avoiding full regeneration of the graph for large projects.
Implementation comprises three parts: a call graph server (Java, Spring Boot, JavaParser, currently Java 8 projects), a GitHub bot, and a single-page web application (React/JavaScript front end; Node.js/Express.js REST API; MongoDB storage).
Evaluated in a 150-minute focus group with seven practitioners plus a 22-question post-demo survey; perceived benefits averaged 4/5, facilitation 3.6/5, acceleration 3.7/5, self-correction 3.4/5, and senior-engineer action 3.7/5.
Identified limitations: the demo used an open-source project while participants mostly worked on commercial projects, and the sample of seven participants was small.
Planned future work: additional languages (starting with Python and Kotlin), richer metrics drawn from focus group feedback, case studies on open-source and commercial projects, and multiple granularity levels incorporating dynamic analysis.
Concepts
Change impact analysis (CIA). Determining which parts of a program are affected by a change. CHID performs this at PR level on a call graph rather than at method/statement level.
Impact graph. A visualization containing the PR's changed methods and the potentially impacted methods (neighbors up to the third degree), included in the GitHub comment so reviewers can inspect and drill into affected methods.
Impact size. A metric measuring the size of the impact graph — the broader effect of a PR's changed methods within the project. It is computed with the PageRank link-based algorithm (Brin and Page 1998), in which a node with a higher score has greater centrality significance; PageRank considers incoming edges and, recursively, incoming edges of neighboring nodes.
Risk score. A weighted sum of metrics, where metric categories (A to E) are mapped to numeric values (1 to 5), multiplied by weights, and combined. Weights are user-modifiable. The risk concept resembles post-release defect risk (Krutauz et al. 2020; Thongtanunam et al. 2015) rather than risk of knowledge loss (Hajari et al. 2024; Kazemi et al. 2022).
Mining software repositories (MSR). Extracting change-propagation information from version histories; combined with call-graph dependency analysis in CHID.
Goal-Question-Metric (GQM). Framework (Caldiera and Rombach 1994) used to formulate the study goal, research questions, and metrics.
Details
Prior CIA and change-prediction techniques
The paper surveys earlier techniques that CHID is compared against:
DistIA (Cai and Thain, 2016) — dynamic analysis that forecasts proliferated impacts inside and outside process boundaries by partially sorting distributed method-execution events. On six Java programs against real impact sets it achieved 71.2% precision and 100% recall for any query.
Tochal (Alimadadi et al., 2015) — DOM-sensitive CIA for JavaScript merging static and dynamic call graphs; ranks entity importance in the impact set using a dependency graph it builds. In a controlled experiment with 10 participants on predetermined tasks, the Tochal group finished 78% faster and 223% more accurately than controls.
Chianti (Ren et al., 2004) — detects interdependent small changes across program versions to build call graphs for test suites, then determines potentially impacted methods and relevant affecting changes using affected tests.
JRipples (Buckner et al., 2005) — an Eclipse plug-in using static information to analyze dependencies between entities, helping developers locate the impact set by tracking visited elements and the elements dependent on them.
FaultTracer (Zhang et al., 2012) — determines atomic changes from ASTs of two program versions, finds dependencies by tracking references of each called method, runs selected tests to emphasize failure-inducing changes, and ranks them with spectrum-based fault localization (Java programs).
ROSE (Zimmermann et al., 2005) — finds coupling between program components (files, functions, variables), extracts coupling information, and predicts future changes by mining version histories; because it uses histories it can also find coupling between non-code items.
Code Change Sniffer (CCS) (Ufuktepe and Tuglular, 2021) — predicts future code changes using static call information, forward slicing, and method change information, combined into Markov chains.
Tools that enhance the code review experience
Research improving code review spans recommending reviewers (Yu et al. 2016; Hajari et al. 2024; Kazemi et al. 2022; Asthana et al. 2019), understanding and improving reviewer behavior (Chen et al. 2022; Egelman et al. 2020; Ebert et al. 2019; Mukhtarov et al. 2023), and assessing code quality (Chen et al. 2022; Chatley and Jones 2018; Balcı et al. 2021; Tuna et al. 2024). Known reviewer challenges include finding relevant code sections, deciding how closely to examine code, and handling multiple reviewers.
Commercial static analysis tools SonarQube, Code Climate, and PMD report rule violations, potential security threats, and refactoring opportunities, and can compute metrics such as code coverage and technical debt. Codium PR-Agent uses LLMs and GPTs to analyze PR diffs — summarizing PRs, suggesting code modifications, checking test coverage, identifying security vulnerabilities, and estimating effort.
Academic tools:
ChangeViz (Gasparini et al., 2021) — helps developers understand and assess PR changesets; integrated into GitHub, letting reviewers navigate method calls and definitions without leaving the PR screen.
DERT (Balcı et al., 2021) — shows change structure in a UML-like relationship diagram of changed classes, methods, interfaces, etc., colored by added/removed/modified status; also shows how much a class changes after the proposed modifications, plus an artifact map of relations between commits, issues, and code files to detect points of interest.
iReview (Hijazi et al., 2021) — uses biometric data (heart rate variability, eye movement dynamics) collected during review to detect poorly reviewed code and generates an evaluation report.
CLUSTERCHANGES (Barnett et al., 2015) — a static analysis method (description truncated in the source notes).
Call graph maintenance and impact detection
Because every accepted change makes the call graph outdated, and regenerating it from scratch is time-consuming for large projects, CHID uses an algorithm that updates the existing project call graph after a PR is merged. The update extracts added and removed function calls and definitions by generating two call graphs — one for the old and one for the new version of the changed files. Because these graphs come only from the changed files, the process is significantly faster than processing the entire project. The two graphs are compared to find newly added or removed calls and definitions. Algorithm 1 gives the pseudocode, Fig. 5 depicts the process steps, and Fig. 4 shows call graph generation.
Once up to date, the call graph drives impact detection: changed methods are extracted from the changeset by generating two sets of ASTs (old and new versions of changed files) and applying the tree differencing algorithm of Fluri et al. (2007). The up-to-third-degree neighbors of the changed methods are then found in the call graph, marked as potentially affected, and presented as an impact graph containing both changed and potentially impacted methods. Algorithm 2 outlines the impact procedure and Fig. 7 illustrates the general impact calculation. Fig. 6 shows an example impact graph from an Arduino project PR (https://github.com/arduino/Arduino/pull/11794).
Impact size and risk score
Impact size quantifies potential impact at method level by running PageRank on the whole project call graph:
where N_i is the set of all methods in the project when PR_i is opened, C_i is the set of changed methods in PR_i, and PageRank(v_i) is the PageRank score of method v_i in the project call graph at that time. Table 7 gives the category thresholds for impact size.
The overall risk score is a weighted sum of metrics converted from categories A–E into values 1–5 and multiplied by the weights in Fig. 8; users can modify the weights to fit their project. Rationale for the coefficients: impact size receives the highest contribution because a ripple effect requires the change to propagate — if it does not, the other metric values matter less. The author's PR merge rate receives the lowest contribution, since a PR with issues is often merged after revisions and reviewers, and developers on the same project tend over time to have similar merge rates; it is not excluded entirely because a newcomer or open-source contributor may have a lower acceptance rate at a given time. The other three metrics contribute relatively similarly.
Implementation
The CIA implementation consists of three parts:
Call graph server — a microservice for call-graph-related tasks (creation/update, impact calculation). Implemented in Java with Spring Boot; currently analyzes projects using Java 8; JavaParser is used for AST generation in the call graph generator algorithm.
Web application — a single-page app (Fig. 11) displaying all analyses in detail. Like SonarQube, users see the category value of each metric and quality gate results, configurable to their selections; reasons for quality gate failures appear beneath the corresponding metrics so users can act, and this information can be added to a pipeline to enforce quality gate constraints under DevOps practices. Metrics and thresholds are configurable (Fig. 12). The frontend uses React and JavaScript; the server side uses JavaScript, Node.js, and Express.js for a REST API, and data is fetched from MongoDB rather than doing heavy calculation or calling the GitHub API.
A mermaid sketch of the pipeline, based strictly on the described steps:
Rendering diagram…
Focus group study
Research objectives and questions. A focus group is a research method that collects data through group discussion concentrated on a research topic (Kontio et al. 2008). The authors ran one to gather practitioners' perspectives on CHID's effectiveness; Fig. 13 summarizes the workflow. Using the GQM framework, the objective was formulated as: "Analyze proposed CIA approach for the purpose of evaluation with respect to the effectiveness, and applicability of the proposed CIA approach; and the effectiveness of the proposed risk score approach from the point of view of software practitioners."
Research questions:
RQ1. How effective is analyzing change impact on the PR level for enhancing the code review experience? (Perceived effectiveness and computational feasibility of CHID at PR level from different aspects.)
RQ2. How effective is the introduced risk score approach for indicating the change impact of a PR?
RQ2.1 Does the introduced risk formula correctly represent the change impact of a PR?
RQ2.2 Does the risk score representation thoroughly reflect the change impact of a PR?
Design. Preparatory work included live demo sessions with four software engineers in one-on-one settings to demonstrate the tool, gather feedback, and address concerns; these insights plus the authors' reflections shaped the focus group outline, discussion questions, and post-demo survey. Preparation also produced a presentation introducing the tool, a survey aligned with the research questions, and a tool video sent with the study invitations. For practical context they forked the Arduino project and selected an open PR for analysis using default metric category and threshold values.
Participant selection used a hybrid of purposeful sampling (Palinkas et al. 2013) and convenience sampling (Stratton 2021), seeking diversity in years of experience and company industries and eager participants. Sessions were 150-minute face-to-face meetings, chosen to maximize coverage and overcome connectivity issues. Twelve practitioners from the authors' network were invited and seven accepted, with roles including developers, architects, and project managers (demographics in Table 8). Co-moderators (the first and second author) were present and presented material interchangeably; participants discussed CHID's potential effects on code review, the risk score formulation, and possible improvements. Each session was recorded in audio and video with consent (two voice recorders plus Zoom for video) and began with an explanation of the case study goal, background, and a data usage disclaimer.
The background presentation briefly explained the survey questions so participants could focus on key points across the 150 minutes, then highlighted the metrics' definition, importance, and calculation method, followed by a live demo on the selected PR explaining analysis results and the tool's mechanism. Participants could ask questions throughout. The post-demo survey had 22 questions (open-ended, Likert-scale, and multiple-choice) on demographics, CHID's effect on code reviews, and the risk score mechanism, completable in about 20 minutes under observation. The focus group discussion outline followed Krueger and Casey (2014) guidelines: an opening question on participants' companies' code review and CIA processes, a transition question on general impressions of the tool, and key questions on CIA at the PR level with elaboration on survey answers.
Qualitative data analysis. Audio recordings were transcribed and translated into English by hand; video was not used because participants could be identified from audio. Open coding from grounded theory (Stol et al. 2016) was applied. In the first iteration the first and second authors independently coded the transcripts, identifying 32 and 20 codes respectively. Thirteen instances of differing labels for the same concept were found (e.g., the first author used authenticity for originality of the approach while the second used novelty), and three codes — speed, static analysis, and dynamic analysis — were identical. Twenty-one codes differed because one author did not code a concept or used a broader code. In the second iteration the two authors merged codes and agreed on a preliminary set of 35. In the third iteration all three authors reviewed and revised codes, addressing unmarked or incorrectly coded sections, with the third author facilitating dispute resolution.
Results (RQ1)
Perceived effectiveness was assessed through computational feasibility experiments, post-demo survey questions, and focus group discussion (Figs. 14 and 15); Likert answers were mapped to a 1–5 numeric scale (Fig. 16). Fig. 15's axes each represent a different aspect of perceived effectiveness: (1) Benefits — potential to improve the code review experience; (2) Facilitate and (3) Accelerate — the tool's facilitation and acceleration effect; (4) Self-correct — potential to help developers find their own mistakes; (5) Trigger Senior Engineer Action — potential to guide other team members to act on results.
Benefits — Asked "Do you think the Change Impact Detector is beneficial for the code review process?", participants averaged 4 out of 5. P1: "There will be an increase in overall effectiveness and acceleration when this tool is used." P2 dissented: "The results need a senior's review. Let's say an architectural decision was taken because of a requirement that can create potential problems. I think it does not make my review process 100% efficient because the tool is unaware of the underlying architectural problem."
Facilitation — Asked to agree/disagree that the review process is easier with CHID, participants averaged 3.6 out of 5. P2: "The tool adds something to the system but does not automate my code review process. I still need to review it manually. It does not 100% measure that the changes in the opened PR cause anything bad." Some implied a learning curve; P1 noted, "The job would become more complex as the amount of things I have to look at has increased... Even after I learn the tool, there will still be more things I need to do."
Acceleration — Asked whether the review process is faster with CHID, participants averaged 3.7 out of 5. P5: "It is not possible for me to reduce the amount of time I need to complete the review using only your tool." P1 was neutral ("I could not say the tool could made the review process faster because the number of things to look at increased") and P2 agreed: "There is more information to look at directly when using this tool. Therefore, it creates manual work instead of speeding it up." Some found impact graphs hard to interpret, especially for unfamiliar developers; P3: "For example, looking at the graph takes time. I think I will be comfortable over time. Some of the presented data are just numeric, which is easy to understand, but the visual part takes time."
Finding mistakes — Asked whether developers can find their mistakes using CHID, participants averaged 3.4 out of 5, with contrasting opinions. P3: "If the analysis tells the PR's risk score is high, more tests can be added." P4: "I do not know if I can get anything out by just interpreting the graph. I did not totally get the proposed value with this feature."
Senior engineer action — Asked whether a team lead/senior engineer can act on CHID's analysis at PR level, participants averaged 3.7 out of 5. They thought the impact graph might help find overlooked methods. P6: "Who will find a method that calls another method which calls a deleted method? You will have to trust the tests for this instance." Other use cases emerged in discussion. P7: "Sometimes the product manager wants a feature but the developer wants to refactor. You have to give him a reason. If they see it in the risk factor, it will be a use case for developers," and "I can use impact graph to learn the system or say to my lead that I will delete a method by showing its relations from impact graph."
Granularity level. Participants had varying views on the optimum granularity. P1 said the tool should respond to different development approaches such as trunk-based development (Forsgren et al. 2017). P2 said CIA would be more applicable if focused on commits, since their company reviews each commit separately. P3: "Since we work on a pull request basis, it would be suitable for us." P4 argued the approach should focus on project requirements instead of solely on PRs.
Limitations and conclusion
Limitations: the tool was introduced using an open-source project, while focus group participants primarily worked with commercial projects, so the results may not fully capture commercial dynamics or challenges (code quality, maintenance practices, collaboration may differ); expanding to commercial projects would give more insight. The study also involved only seven participants, a relatively small sample; a larger, more diverse set of practitioners would help evaluate applicability across organizational contexts and development environments.
The study introduces a novel CIA approach and supporting tool that uses call graphs and several metrics to estimate the impact of changes in a PR. Practitioners' perspectives were gathered via a preliminary feature survey, the focus group study, and the post-demo survey, while computational feasibility experiments assessed practicality in real-world scenarios. Focus group participants concluded CHID eases and accelerates code review and that results could help a team lead act at PR level; however, some companies use atomic units other than PRs, so CHID should comprehend these granularity levels to be fully effective. The risk score was judged effective in representing risk, but its presentation, metrics, coefficients, and thresholds require further study. Across discussions CIA was seen as applicable to various code review use cases, but practitioners may not be motivated toward an analysis requiring effort to interpret, and may behave in ways that mislead the analysis — a problem for CHID's effectiveness, pointing to a need for guidelines merging core review and CIA practices. Planned enhancements: support for additional languages starting with Python and Kotlin; diversifying metrics using focus group and post-demo survey data; case studies on open-source and commercial projects; and multiple granularity levels incorporating dynamic analysis.
Related work on CIA for code review
Unlike the background techniques, these studies aim to use CIA to improve code review; feature comparisons appear in Table 13 (academic studies) and Table 14 (commercial tools).
SEMCIA (Hanam et al., 2019) — determines semantic change impact relations from JavaScript to reduce CIA noise; cuts false positives by up to 53% and considerably shrinks change impact sets. A preliminary study with 11 participants doing three commit review tasks each found the noise reduction helped developers review more quickly and accurately.
BLIMP Tracer (Wen et al., 2018) — uses a Build Dependency Graph (BDG) created from build logs, processing each file to create intermediate and final deliverables, and analyzes the BDG to determine the set of impacted product deliverables per patch. Integrated with the DELL EMC code review platform, with results shown in its interface.
Taint Impact (Lüscher, 2021) — uses dynamic taint analysis tracking data flow across a program to determine lines impacted by a change. Evaluated with three artificial program examples and a real-world bug; results show it highlights impacted code parts that developers find hard to determine.
CRITICS (Zhang et al., 2015) — an interactive code review approach that creates a change template from related data and models/controls the flow of change as an AST; reviewers can alter the template, and the final template is compared with the codebase to locate possibly omitted edits. In a controlled experiment with 12 participants using CRITICS and Eclipse diff, participants answered systematic questions 47% faster and 31% more precisely with CRITICS diff.
Diggit (Chatley and Jones, 2018) — guides developers based on expected changes and historical repository information by generating code review comments; it determines co-changed files historically and alerts the user when a co-changed file is missing from the PR changeset, and warns about highly churned files. It performs no CIA but resembles CHID in posting analysis results directly as PR comments.
Two main industrial competitors were identified:
Softagram — the closest competitor; provides visual PR reviews by analyzing change impacts with program dependency graphs and machine learning. Like CHID it offers impact visualization, alerts about missing co-changed files, and customizable analysis; unlike CHID it does not compute a final metric representing the PR's potential effects.
CodeCov — a code coverage reporting solution that analyzes PRs to determine coverage rate and summarizes findings in the comments section. It performs impact analysis to list the most user-facing parts of a code change (such as HTTP endpoints) and to identify changed files containing code frequently used in production.