Repository navigation
Close the filtering gaps: keys by their words, HMAC hashes, card numbers, URL secrets, request IDs and SQL binds - #24
Merged
Conversation
- Sensitive key names are matched by their words, whatever their case or
separators: apiKey, X-Api-Key, HTTP_AUTHORIZATION, Set-Cookie and plurals
like passwords are filtered by default, and so are cookie, session, jwt,
bearer, otp and passphrase. Whole words keep author, keyboard and
signature_field_id. filter_keys entries match by their words too.
- Rails' dotted filter_parameters ("card.number", /\Acard\.number\z/) match
nested keys by their path, and three-argument filter blocks get the
parameters around the key, as they do in Rails. New path_matchers setting.
- Hashed values (emails, filter_keys_with_hashes) are an HMAC-SHA256 keyed
with hash_salt, or by default with a key derived from the app's
secret_key_base instead of a salt shared by every install, and are 16
characters long. A filtered String no longer logs its length.
- The filter configuration example adds to the default lists instead of
replacing them.
- Card numbers of 13 to 19 digits are scrubbed, written together, in fours with one kind of separator, or in American Express and Diners Club groups. Numbers that fail the Luhn check are left alone. - URL passwords: the user part ends at its first colon (user:pass:word kept "user:pass:" before), an empty user is matched (redis://:pw@host), and lowercase %3a/%40 in encoded URLs too. - Sensitive query params in URLs are filtered (?token=, ?apiKey=, user[password]=, OAuth's ?code=): the names are checked like keys, with the filter keys, the default pattern and Rails' filter_parameters. New url_query_params setting, on by default. - Hash keys that aren't plain identifiers are scrubbed (an email address as a key), and so are a filtered Hash's _keys, Symbol values and GlobalIDs.
- Request, blocked host, CSRF and IP spoofing lines, and the request context sent to error reporters, have the Referer without its query and fragment, which can carry password reset, confirmation and magic link tokens or OAuth codes. - ActiveStorage url lines leave out the signed URL (S3 signatures, Disk tokens): anyone who has it can read the file until it expires. Rails logs it at debug only. - Error reporters (Sentry, Bugsnag, Rollbar, Honeybadger and custom handlers) get the context filtered and scrubbed like a log line, instead of raw parameters, referers and mail recipients.
The request ID and the running Sidekiq job's context came from SemanticLogger's named tags after the line had been filtered, by parsing the line's JSON and writing it again. An X-Request-Id header can carry anything (a card number was logged as it was sent), and filtering req_id had no effect. LogStruct::Formatter#call now takes the request ID and job context, and puts them and the program name through the same filters and scrubbers as the rest of the line. A line's own req_id wins over the request's. The SemanticLogger formatter no longer parses and regenerates each line's JSON.
With sql_log_bind_params on, bind values were filtered only when they looked like tokens or hashes, so a password bind was logged as it was. Binds are now paired with their column names and filtered like request params (the filter keys, the default pattern and Rails' filter_parameters), as Rails' own SQL lines do. Binary values are logged as their size, and type_casted_binds passed as a lambda (Rails 6.0 and later) are read instead of raising. The integration's comment said SQL is always parameterized: it isn't (string conditions, or prepared_statements off), so it now says the SQL is scrubbed but not filtered.
The colour formatter (opt-in, for development) wrote plain messages, their payloads, exceptions and tags as they were logged, and loggers added with broadcast_to got String messages and block results unscrubbed. Both now go through the string scrubbers, like the strings in a JSON line.
The filtering page said email keys are filtered with a hash by default (they are scrubbed to [EMAIL:hash] unless the key is filtered too), that "Password: secret123" is scrubbed to [PASSWORD] (only URL passwords are), and that Rails' dotted filters and three-argument blocks don't apply. It now covers word-based key matching and the default pattern, Rails' paths and blocks, HMAC hashes and their key (pseudonyms, not anonymous data), query params, card numbers and the Luhn check, scrubbed keys, and what the integrations leave out: Referer queries, ActiveStorage URLs, delivery job arguments, SQL binds, error report context. The configuration example is shown once. The SOC 2 page listed a _size field that doesn't exist and a payment_data key that isn't filtered, and promised retention "without privacy concerns". The docs' sample log generator gave filtered strings a length, arrays a _length and password keys "[PASSWORD]", and filtered keys LogStruct doesn't. The README mentions key filtering, Rails' dotted filters and the Referer without its query.
|
Warning Review limit reached
This review includes 40 billable files and costs up to $10.00.
Reviews can continue after your included limit without a manual trigger. An admin must approve usage-based billing. Or wait 40 minutes for your next included review. View limit details
Comment |
This was referenced Oct 17, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ninth of twelve stacked PRs for 0.2.0. These are the fixes from a security audit of the filtering.
Keys
apiKey,X-Api-Key,HTTP_AUTHORIZATION,Set-Cookieand plurals likepasswordsare filtered by default, and so arecookie,session,jwt,bearer,otpandpassphrase. Matching whole words leavesauthor,keyboardandsignature_field_idalone.filter_keysentries match by their words too.filter_parameters("card.number",/\Acard\.number\z/) match nested keys by their path, andthree-argument filter blocks get the parameters around the key, as in Rails. New
filters.path_matcherssetting.filter_keys_with_hashes) are an HMAC-SHA256, 16 characters long, keyed withhash_saltor, by default, a key derived from the app'ssecret_key_base. The default was a published salt(
"l0g5t0p"), so anyone with a list of addresses could match them to the hashes. A filtered String no longer logs its length.Strings
Express and Diners Club groups. Numbers that fail the Luhn check are left alone.
user:pass:wordkeptuser:pass:before), an empty user ismatched (
redis://:pw@host), and so are lowercase%3a/%40in encoded URLs.?token=,?apiKey=,user[password]=, OAuth's?code=). Thenames are checked like keys. New
filters.url_query_paramssetting, on by default._keys, Symbol values and GlobalIDs.What the integrations write
Referer without its query and fragment, which can carry password reset, confirmation and magic link tokens or OAuth
codes.
urllines leave out the signed URL: anyone who has it can read the file until it expires.like a log line, instead of raw parameters, referers and mail recipients.
by parsing the line's JSON and writing it again, so an
X-Request-Idheader carrying a card number was logged assent.
LogStruct::Formatter#callnow takes the request ID and job context and filters them with the rest of theline, and the SemanticLogger formatter no longer parses and regenerates each line.
sql_log_bind_paramson, like request params. Before,only values that looked like tokens were filtered, so a
passwordbind was logged. Binary values are logged as theirsize, and
type_casted_bindsgiven as a lambda are read instead of raising.broadcast_toget scrubbed strings too.Docs
The filtering page, the SOC 2 page, the README and the site's sample log lines describe filtering as it now works.
They had described email keys,
[PASSWORD]scrubbing, a_sizefield and apayment_datakey that LogStructdidn't have, and said Rails' dotted filters and three-argument blocks don't apply.
Testing
New tests for each change. Sorbet and the unit suite (332 tests) pass at this branch's tip.
Stack for 0.2.0, merged in order: #16 → #17 → #18 → #19 → #20 → #21 → #22 → #23 → #24 → #25 → #26 → #27