Repository navigation
Make INSEE authentication resilient to intermittent refusals - #477
Merged
Merged
Conversation
skelz0r
force-pushed
the
feature/insee-auth-resilience
branch
2 times, most recently
from
October 6, 2026 13:09
8b3de6c to
5ec8db2
Compare
GoodJob 4.19.3 references ActiveModel::Model while loading its configuration validator without requiring it, so the site smoke script, which loads the gems one by one instead of booting Rails, crashed before its first scenario.
INSEE's OAuth endpoint is now a Keycloak realm that grants a new access token on every exchange, valid for 300 seconds and without a refresh token. The fixtures still carried the week long lifetime of the WSO2 gateway recorded in 2018, which hid how often siade really authenticates: at least once every five minutes per instance.
Sentry scrubs provider_error_description on every INSEE authentication
alert: Keycloak's "Invalid user credentials" holds a word its data
scrubber filters. The refusals of September and October therefore
reached Sentry as a bare 400 invalid_grant, with nothing to say why the
realm refused a password known to be right.
The description is now also mapped to a refusal_reason drawn from a
fixed vocabulary the scrubber leaves alone. Anything unexpected comes
out as unknown rather than being echoed.
The HTTP code matters as much: in Keycloak 26 a wrong password and an
account locked by brute force detection both answer 401 "Invalid user
credentials", while 400 is reserved to a disabled account and to an
accepted password whose account has pending required actions ("Account
is not fully set up").
INSEE tokens live five minutes and were only renewed once expired or rejected, with every Sirene request of an instance waiting on that one OAuth exchange. A refusal at that moment, which the Keycloak realm answers intermittently to a valid password, left the instance without any token. The token is now renewed from 90 seconds before its expiry. The request that wins the existing lock renews while the others keep the current token, which stays published until a new one is granted: a refused or failed renewal costs nothing as long as the current token lives. Each attempt postpones the next one by 30 seconds whatever its outcome, otherwise an OAuth endpoint answering 503 would get one call per Sirene request until the token expires; once no retry fits, the window closes. An attempt may chain up to three exchanges, current password, previous one, then current again, each taking up to 20 seconds with its connect and read timeouts. No attempt starts unless all of them can complete before the cached token expires, so a renewal does not outlive the token it replaces and make expired requests wait on the lock for nothing. A five minute token is thus renewed at 3:30, then 4:00 if needed before the derivation starts, and at 3:30 only afterwards. The renewal window is only published once the token itself is: a window written for a token that never reached the cache would postpone the renewal of the one still served until it expires. A token living less than 90 seconds is kept until its expiry, as before. Each exchange grants a brand new token, so renewing early does not revoke the current one.
A refused authentication used to hold back every other one for thirty minutes. Tokens live five minutes, so a single intermittent refusal from the Keycloak realm took Sirene down on that instance for half an hour, with no alert after the first one. The pattern of October 1st, a failure every thirty minutes from 16:01 to 17:31, is that guard's rhythm. The hold now starts at thirty seconds and doubles up to five minutes, and the first granted token resets it. Together with the early renewal, an isolated refusal now costs nothing while the current token lives. Thirty minutes only remain for a refusal of the OAuth exchange itself, such as a revoked client secret, where retrying cannot help. Waiting longer would not spare Keycloak's lockout budget anyway: five consecutive refused passwords lock the account and only a success resets that count. Nor can a lockout be detected to stop earlier: Keycloak 26 answers it exactly like a wrong password. The September and October refusals were 400s, which in that version follow an accepted password and do not spend the budget, so a dedicated long hold on account related descriptions would only bring the thirty minutes back. Refusals are counted per episode, in a single entry beside the hold whose one hour expiry slides with each refusal. The first refusal raises the error, the following ones stay silent, and the first granted token reports the recovery with the number of refusals and the length of the outage. Only the process that deletes the episode reports it, so two boots sharing a Redis during a deployment do not both announce it. With a single candidate, before the derivation starts, a refusal cannot mean a desynchronization: there is no other password to fall back on, and the alert now says so. The hold and the episode move to INSEE::AuthenticationBackoff to keep the interactor about the exchange.
The smoke scripts still simulated an OAuth server granting one hour tokens and refusing only wrong passwords, and expected siade to hold back for thirty minutes after any refusal. The simulated server now behaves like INSEE's Keycloak realm: five minute tokens, 401 for a wrong password, 400 refusals of a valid password on demand and a disabled account. siade scenarios check the renewal ahead of expiry, including refused, unavailable and concurrent ones, and the backoff steps with their single alert and recovery report. Two scenarios pin known limits rather than desired behaviour: after November an intermittent refusal makes the walk spend the previous password, and two instances without a common Redis reach six refusals in 91 seconds, beyond Keycloak's lockout threshold of five.
skelz0r
force-pushed
the
feature/insee-auth-resilience
branch
from
October 6, 2026 15:04
5ec8db2 to
adc1643
Compare
Un3x
approved these changes
Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
INSEE's OAuth endpoint is a Keycloak realm granting a new five minute token on every exchange (checked in production: expires_in 300, no refresh token), not the week long token our fixtures assumed. A single intermittent invalid_grant on a valid password then held back every authentication for thirty minutes, which is the rhythm of Sentry issues 304319, 303016 and 302776: one failure every thirty minutes, without any trace in between.
In Keycloak 26 a wrong password and a brute force lockout both answer 401 "Invalid user credentials", so a lockout cannot be detected and the backoff applies to every invalid_grant. The production refusals were 400s, which in that version only come from a disabled account or from an accepted password with pending required actions; refusal_reason will tell which on the next one.
Known limits, documented in docs/rotation_mot_de_passe_insee.md and pinned by smoke scenarios: each siade server has its own Redis, so two instances refused with 401s reach Keycloak's lockout threshold of five in 91 seconds; and from November an intermittent refusal makes the candidate walk spend the previous password, which needs its own fix before November 1st.
Checked in production on October 6th: four password grants in a row return four distinct tokens that Sirene all accepts, so renewing early never revokes the token still served. Sirene also accepts a token until two seconds before its exp, so the ten seconds cache margin holds. The same day, a refusal at 15:01:46 was over by 15:18:33 at the latest, while the thirty minutes guard kept Sirene down until the guards were cleared by hand.