Skip to content

Enhancement | The Snyk PR check tests nothing; 51 open Dependabot alerts on poetry.lock #358

Closed

Description

@diego-ojeda-binbash

What?

The security/snyk (Binbashar) check is always green, but it never tests anything. Meanwhile Dependabot reports 51 open alerts on the same poetry.lock, and nothing opens PRs to fix them.

Evidence

  1. Every PR gets the same description: "No manifest changes detected in 1 project". That includes Enhancement | Lint with ruff in CI and pre-commit, drop pylint #353 and Enhancement | Type check leverage/ with mypy (gradual) in CI #355, which did change pyproject.toml and poetry.lock. If Snyk were watching poetry.lock, it would have run its test there.

  2. Likely cause: the repo moved to Poetry in [157] Refactor of build processes and dependency management using Poetry #261 (April 2024), which deleted Pipfile, Pipfile.lock, dev-requirements.txt, setup.py and setup.cfg. The "1 project" Snyk has imported is most likely one of those. To be confirmed in app.snyk.io (org binbashar). The Snyk webhook on the repo is still active.

  3. Even when set up right, Snyk's PR check only tests when a manifest changes. It is not a required check either.

  4. Dependabot alerts (open): 1 critical, 20 high, 24 medium, 6 low.

    • Runtime dependencies: certifi (critical), urllib3, requests, idna, cryptography, jinja2, pygments
    • Dev dependencies: black, virtualenv, filelock, pytest

    There is no .github/dependabot.yml, so no update PRs get opened.

Proposal

  1. Decide on Snyk. Either re-import the project in Snyk pointing at pyproject.toml/poetry.lock, or turn off its PR check and rely on Dependabot, which is already on and needs no external account.
  2. Add .github/dependabot.yml for pip (Poetry) and github-actions, weekly, with grouped updates to keep the PR count low.
  3. Update the lock to close the open alerts, starting with the runtime ones. Test it on Python 3.9–3.13: some fixed versions may no longer support 3.9 (see Discussion | Review the supported Python versions (3.9/3.10 end of life, 3.14 installs an old release) #341).

Found while reviewing the CI checks for #346.

🤖 Generated with Claude Code

Activity

  1. diego-ojeda-binbash commented on Nov 15, 2026

    @diego-ojeda-binbash
    CollaboratorAuthor

    Update

    Dependabot: done.

    Snyk: root cause found, fix in progress.

    • Cause: the 2 projects Snyk had imported for this repo pointed at files that no longer exist:
    • Done: the repo was re-imported. Snyk created a pyproject.toml project (dependencies) and a "Snyk Code" project (source code analysis, which also covers PRs that don't touch a manifest). The PR check works now: on Enhancement | Support Python 3.14 and 3.15, no cap on the minor version #370 it said "1 security test has passed".
    • Pending (this week):
      • Confirm the import finished
      • Delete the 2 old projects (Dockerfile, dev-requirements.txt) if they're still listed
      • Review the Snyk Code findings, if any

    Close this issue once those are done.

    🤖 Generated with Claude Code

  2. diego-ojeda-binbash commented on Nov 15, 2026

    @diego-ojeda-binbash
    CollaboratorAuthor

    One more item for the Snyk review: since the re-import Snyk tests for real, and on #365 its check failed with "You have used your limit of private tests" (https://app.snyk.io/org/binbashar/pr-checks/16202e6e-33c8-427d-9337-66db4244eea9). The repo is public, but Snyk counts these as private tests: check how the project was imported (visibility) and the org's plan limits. The check isn't required, so it doesn't block merges.

  3. diego-ojeda-binbash commented on Nov 15, 2026

    @diego-ojeda-binbash
    CollaboratorAuthor

    Closing summary

    Snyk is out of this repo; #380 closes this issue.

    • Why: the org's Snyk is on the free plan, and the private-test quota is used up on the first day of each billing period (this period: 1166/200 manifest tests, 4322/300 IaC; previous one: 200/200). Since the re-import, every PR check failed with "You have used your limit of private tests". It's an org-wide issue, to be handled separately from this repo.
    • What leverage used from Snyk: the PR check (it tested nothing from April 2024 until the re-import) and fix PRs (14 between 2022 and 2024, none since April 2024). No .snyk policy, IaC or license checks.
    • Now:
    • Trade-off: Snyk's own vulnerability database and priority scores are no longer used here; GitHub's Advisory Database covers the dependencies instead.

    🤖 Generated with Claude Code

  4. diego-ojeda-binbash commented on Nov 15, 2026

    @diego-ojeda-binbash
    CollaboratorAuthor

    Addendum on the trade-off: Snyk's API reports the binbashar org's entitlements as fullVulnDB: false, licenses: false, reports: false, and API access isn't included in its plan. On the free plan the org wasn't using Snyk's full vulnerability database, so removing Snyk here loses nothing in practice: Dependabot (GitHub Advisory Database) and CodeQL cover what it did.

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions