A guide to what automated analysis of a codebase can and cannot see. Written to accompany the Code Futures report, but the subject is the judgement rather than the format: tools change, and the question of what a tool's output is evidence of does not.
Developed for use in Open Source with SLU experiential learning programs, and applicable to any team trying to work out how much to believe a dashboard.
- Guidebook — What automated analysis establishes and what it only appears to establish; how to read each finding; how to choose one thing to change; what making a check blocking costs
- Worked example report — A fabricated Code Futures report, for a repository that does not exist. Composite by design, so every kind of outcome appears somewhere
- Tech Leads who have received a Code Futures report and want to act on it
- Developers trying to understand what a green check actually proves
- Mentors and advisors reading reports alongside a team
- Faculty running a program like this one at another institution
- Anyone who has looked at a dashboard and wondered how much to believe it
Read §2, What Automated Analysis Can and Cannot See, before §3, Reading Your Report. The order is deliberate — read the other way round, the findings look more authoritative than they are.
Keep the worked example open alongside §3. It is the report §3 describes.
This guidebook is the companion to the Codebase Technical Assessment Guidebook and Rubric, not a replacement for it or a competitor to it.
The rubric is a human instrument: a person reading a codebase and exercising judgement across seventeen dimensions. This guidebook is about automated signal — broad, cheap, repeatable, and blind to everything requiring judgement. Automation finds every repository without branch protection in under a minute and will never tell you the domain model is wrong.
The rubric remains the assessment instrument. A Code Futures report produces evidence to bring to it.
This guidebook describes reports carrying the schema code-futures/v1. Every report states its schema in the header. If the two disagree, trust the report and open an issue — a mismatch is a bug on our side.
Corrections and suggestions are welcome by pull request.
Feedback on anything that assumes knowledge you do not have is particularly wanted. That is the failure mode hardest for the authors to see and the one most worth hearing about, and it does not need to arrive as a pull request — an issue saying "I did not understand this paragraph" is a useful contribution.
This work © 2026 by Open Source with SLU, the open source program office of Saint Louis University, is licensed under CC BY-NC-SA 4.0.