Improve jit-analyze performance and memory usage - #441
Open
MihaZupan wants to merge 18 commits into
Open
Conversation
…ggregate Cache the join, unique-method lists and filtered deltas. Add dependency-free parser and CLI regression coverage, including optional baseline comparison. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Enumerate paths without parsing, then parse and compare each pair for all requested metrics before releasing unchanged method data. Preserve result order while bounding concurrent work. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Avoid allocating strings and LINQ records for instruction lines that contain no metrics. Preserve StreamReader encoding and newline handling, including long records split across buffers. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Use shared generated regexes and span captures, and accumulate directly into per-name method records instead of retaining and repeatedly traversing groups of line records. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Keep doubles in a single array per collection rather than allocating a polymorphic object per metric. Materialize metric display objects only for reports and reuse parsed values when forming deltas. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Bound git workers, compare equal-size inputs with pooled buffers, cache text counts across metrics, pass paths without shell quoting, and surface git failures. Match absolute count keys when reporting text-only changes, fixing the previously missing section. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Recognize byte-identical inputs before parsing the second side, and avoid building joins or method deltas for them. Preserve absent-metric relative delta semantics. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Walk matched directories without following directory links, include dangling links, and cover link cycles and quoted paths. Compare JSON and Markdown against the original analyzer as well as stdout and TSV. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Record the issue 2148 artifact workload, seven incremental optimizations, repeated end-to-end timings, peak RSS, and benchmark commands and caveats. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…port Use git diff --quiet for other files while preserving their contribution to the changed-file count. Avoid unused numstat work, including streaming object hashes for very large files, and retain full counts for callers of DiffInText. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Record per-file profiling, three complete workload timings, and the reduced peak RSS. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ce50c2b-b096-4210-9633-948434c43ee4
Remove the application-specific eight-worker cap from both phases. Record fresh interleaved full-workload benchmarks: capped median 13.93s and 0.86 GiB peak RSS, default median 14.34s and 1.12 GiB. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ce50c2b-b096-4210-9633-948434c43ee4
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ce50c2b-b096-4210-9633-948434c43ee4
Replace the pooled buffer with a 32K-character array and grow it with Array.Resize for long lines. Remove pool cleanup and update the reader documentation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ce50c2b-b096-4210-9633-948434c43ee4
Add a ProcessStartInfo overload so ArgumentList preserves literal paths, while retaining the existing overload's pre-quoted argument behavior. Share output capture and process disposal instead of duplicating them in jit-analyze. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ce50c2b-b096-4210-9633-948434c43ee4
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ce50c2b-b096-4210-9633-948434c43ee4
Retain equality detection for sharing parsed methods, but let Git check text pairs directly. Update the description of the text-analysis pipeline. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ce50c2b-b096-4210-9633-948434c43ee4
Keep the cumulative changes focused on the analyzer optimizations. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6ce50c2b-b096-4210-9633-948434c43ee4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reduce allocations and repeated work, reuse analysis for identical inputs, and parallelize independent comparisons.
Also fix the text-only summary: diff counts were stored under absolute paths but looked up using relative filenames, causing files with textual changes but unchanged aggregate metrics to be omitted. The benchmark input contains 106 such files, which are now reported.
Performance
Using the artifacts from MihuBot/runtime-utils#2148—760 assemblies per side, totaling 25.5 GB of disassembly:
Approximately 15× faster with 91% lower peak RSS.
Measurements are medians of three runs on a 16-logical-processor Linux machine using a Release build with .NET 10, with textual comparisons enabled.
Incremental improvements
27ea778b973865bc47d72bcb13f8c882924bb910ada816f75e80f9a4These are historical single-run measurements of successive optimization snapshots, starting at 229.39 s. Reductions are relative to the preceding measured snapshot. The final medians above include subsequent simplifications, including removal of the redundant pre-Git equality scan.
(copilot generated description)