Repository navigation
Conversation
…rsing split_header_words() rebuilt the remaining header text after every token, so parsing a long Set-Cookie2 header took time quadratic in its length. It now keeps a position in the text and matches each token there, which is linear and gives the same result. The header regular expressions lose their ^ anchor, since they are now applied with match() at a position.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes #159642
split_header_words()consumed one token at a time and rebuilt the rest of the header string after each one (throughunmatched()ortext.lstrip()[1:]), so parsing took time quadratic in the header length.CookieJar.extract_cookies()runs it on everySet-Cookie2header whenever the response also has aSet-Cookieheader, including under the default policy, so a server could keep a client busy for minutes with one large folded header.This change keeps a position in the text and applies each regular expression with
match(text, pos)instead of slicing. The^anchors are dropped from the header regular expressions becausematch()already anchors at the given position, and the comma and junk checks become two small compiled patterns. The result is the same as before; I also compared the old and new functions on 200,000 random short inputs built from the separator characters, quotes, backslashes and Unicode whitespace, with no differences.The new test
test_split_header_words_largeparses a 1.2 MB header made ofa=1; b="2",repeated 10**6 times and checks the result. It takes about a second with the change and did not finish within two minutes without it. It is marked with thecpuresource, liketest_unquote_largeintest_http_cookies.