🧠 Regex Fundamentals

🎯 What is a Regular Expression?

A regular expression (regex) is a sequence of characters that defines a search pattern. They are used for string matching, validation, extraction, and replacement. Regex is based on regular languages from automata theory — every regex corresponds to a finite automaton (DFA/NFA).

Why Regex Matters

Ubiquitous: Every language has regex. Powerful: Complex patterns in one line. Concise: Describe patterns compactly. Fast: Linear time with DFA engines. Versatile: Validation, search, replace, parse. Foundation: Compilers, networking, security.

📌 Pattern Matching📌 Regular Languages📌 Text Processing📌 Universally Supported
# Regular Expressions — the big picture # Regex is a mini-language for describing text patterns # REAL-LIFE EXAMPLE: Email validation # Pattern: something@something.something # WITHOUT regex (painful): def is_valid_email_naive(s): if '@' not in s: return False parts = s.split('@') if len(parts) != 2: return False if '.' not in parts[1]: return False # ... and so on — error-prone! # WITH regex (elegant): import re EMAIL_REGEX = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$' def is_valid_email(s): return bool(re.match(EMAIL_REGEX, s)) # Test emails = [ "user@example.com", # ✓ "test@domain.co.uk", # ✓ "invalid", # ✗ "user@invalid", # ✗ "a@b.com", # ✓ ] print("=== Email Validation ===") for email in emails: result = "✓ VALID" if is_valid_email(email) else "✗ INVALID" print(f" {email:<25} → {result}") # WHAT REGEX CAN DO print(""" === What Regex Can Do === 1. VALIDATION - Email, phone, URL, date, IP address - Credit cards, postal codes - Custom formats 2. SEARCH - Find patterns in text - grep, search, find - Highlight matches 3. EXTRACTION - Extract emails from text - Parse log files - Scrape data 4. REPLACEMENT - Sed, replace, substitute - Clean data - Reformat text 5. TOKENIZATION - Split into tokens - Lexical analysis - Language parsing 6. SPLITTING - Split strings by pattern - Parse CSV, TSV - Tokenize input 7. FILTERING - Whitelist/blacklist - Log filtering - Data cleaning """) # REGEX QUICK REFERENCE print(""" === Quick Reference === CHARACTERS: . any character (except newline) \d digit [0-9] \D non-digit \w word character [a-zA-Z0-9_] \W non-word \s whitespace [ \t\n\r\f\v] \S non-whitespace QUANTIFIERS: * 0 or more + 1 or more ? 0 or 1 {n} exactly n {n,} n or more {n,m} between n and m ANCHORS: ^ start of string/line $ end of string/line \b word boundary \B non-word boundary CLASSES: [abc] a, b, or c [^abc] not a, b, or c [a-z] range [a-zA-Z0-9] combined GROUPS: (...) capturing group (?:...) non-capturing (?P...) named group | alternation (or) ESCAPES: \. literal dot \* literal star \\ literal backslash \n newline \t tab """)

📝 Basic Syntax

🎯 The Building Blocks

Regex syntax consists of literals (exact characters), metacharacters (special symbols), and operators (quantifiers, groups).

Literals and Metacharacters

Literals: Match themselves (a, b, 1, 2). Metacharacters: Special meaning (. ^ $ * + ? { } [ ] \ | ( )). Escaping: Use \ to match literal metacharacter. Concatenation: Sequence matches in order. Alternation: | for "or". Precedence: Grouping → Quantifiers → Concat → Alternation.

📝 Literals⚡ Metacharacters\ Escaping| Alternation
# Basic Regex Syntax import re # 1. LITERALS — match exact characters print("=== Literals ===") print(f"re.search('cat', 'the cat sat') → {re.search('cat', 'the cat sat')}") print(f"re.search('dog', 'the cat sat') → {re.search('dog', 'the cat sat')}") print(f"re.search('123', 'abc123def') → {re.search('123', 'abc123def')}") # 2. METACHARACTERS print("\n=== Metacharacters ===") print(""" Metacharacters have special meaning: . any character (except newline) ^ start of string $ end of string * 0 or more + 1 or more ? 0 or 1 (or lazy) { } quantifier [ ] character class \ escape | alternation ( ) group To match a literal metacharacter, escape it with \\: \\. literal dot \\* literal star \\\\ literal backslash """) # 3. THE DOT print("=== The Dot (.) ===") print("Matches any character except newline") print(f"re.search('a.c', 'abc') → {re.search('a.c', 'abc')}") print(f"re.search('a.c', 'a c') → {re.search('a.c', 'a c')}") print(f"re.search('a.c', 'a\\nc') → {re.search('a.c', 'a\\nc')}") print(f"re.search('a.c', 'a\\nc', re.DOTALL) → {re.search('a.c', 'a\\nc', re.DOTALL)}") # 4. CONCATENATION print("\n=== Concatenation ===") print("Sequence of characters matches in order") print(f"re.search('abc', 'xabcx') → {re.search('abc', 'xabcx')}") print(f"re.search('abc', 'acb') → {re.search('abc', 'acb')}") # 5. ALTERNATION print("\n=== Alternation (|) ===") print("Matches either pattern") pattern = r'cat|dog' print(f"Pattern: {pattern}") for s in ['cat', 'dog', 'cow', 'catdog']: match = re.search(pattern, s) print(f" '{s}' → {match.group() if match else 'no match'}") # 6. ESCAPING print("\n=== Escaping ===") print("Match literal metacharacter with backslash") print(f"r'\\.' matches literal dot:") for s in ['a.b', 'axb']: match = re.search(r'\.', s) print(f" '{s}' → {'match' if match else 'no match'}") print(f"r'\\*' matches literal star:") for s in ['a*b', 'axb']: match = re.search(r'\*', s) print(f" '{s}' → {'match' if match else 'no match'}") # 7. PRECEDENCE print("\n=== Precedence ===") print(""" From highest to lowest: 1. Grouping: ( ) 2. Quantifiers: * + ? { } 3. Concatenation: ab 4. Alternation: | Example: ab|cd means (ab)|(cd), not a(b|c)d a|b* means a|(b*), not (a|b)* (a|b)* means (a|b)* — explicit grouping """) # 8. PRACTICAL EXAMPLES print("=== Practical Examples ===") examples = [ (r'hello', 'hello world', 'literal match'), (r'h.llo', 'hello', 'dot wildcard'), (r'colou?r', 'color', 'optional u'), (r'colou?r', 'colour', 'optional u'), (r'cat|dog', 'I have a dog', 'alternation'), (r'\d+', 'abc 123 def', 'one or more digits'), (r'\w+', 'hello world', 'one or more word chars'), ] for pattern, text, desc in examples: match = re.search(pattern, text) result = match.group() if match else 'no match' print(f" {pattern:<15} on '{text}': {result:<15} ({desc})") # 9. COMMON MISTAKES print("\n=== Common Mistakes ===") print(""" 1. FORGETTING TO ESCAPE ✗ re.search('.', 'a.b') — matches any char ✓ re.search('\\.', 'a.b') — matches literal dot 2. GREEDY VS LAZY ✗ '<.*>' on '' — matches '' ✓ '<.*?>' on '' — matches '' then '' 3. MISSING ANCHORS ✗ r'\\d+' on 'abc123' — matches '123' anywhere ✓ r'^\\d+$' on 'abc123' — requires full match 4. NOT USING RAW STRINGS ✗ '\\d' — Python interprets \\d as backspace+d? ✓ r'\\d' — raw string, passes \\d to regex engine 5. CONFUSING QUANTIFIERS ✗ 'a{1,3}' matches 'a', 'aa', 'aaa' ✓ 'a{3}' matches exactly 'aaa' 6. CHARACTER CLASS CONFUSION ✗ '[.]' matches literal dot (inside class, . is literal) ✓ '.' matches any character 7. GREEDY MATCHING ✗ '.*' matches everything ✓ '.*?' matches as little as possible """)

🔤 Character Classes

🎯 Matching Sets of Characters

Character classes let you match any one character from a set. They're defined with square brackets [ ] or with shorthand escapes like \d, \w, \s.

Class Types

Positive: [abc] matches a, b, or c. Negative: [^abc] matches anything except a, b, c. Range: [a-z] matches any lowercase. Shorthand: \d, \w, \s. Combined: [a-zA-Z0-9_]. Special: [.] matches literal dot inside class.

[abc] Set[^abc] Negate[a-z] Range\d \w \s Shorthand
# Character Classes in detail import re # 1. POSITIVE CHARACTER CLASSES print("=== Positive Character Classes ===") print("Match any ONE character from the set") pattern = r'[aeiou]' print(f"Pattern: {pattern} (vowels)") for s in ['cat', 'dog', 'apple', 'xyz']: match = re.search(pattern, s) print(f" '{s}' → {match.group() if match else 'no match'}") # 2. NEGATIVE CHARACTER CLASSES print("\n=== Negative Character Classes ===") print("Match any ONE character NOT in the set") pattern = r'[^aeiou]' print(f"Pattern: {pattern} (non-vowels)") for s in ['cat', 'aeiou', 'bcdfg']: match = re.search(pattern, s) print(f" '{s}' → {match.group() if match else 'no match'}") # 3. RANGES print("\n=== Ranges ===") print("[a-z] lowercase") print("[A-Z] uppercase") print("[0-9] digits") print("[a-zA-Z] letters") print("[a-zA-Z0-9] alphanumeric") pattern = r'[a-z]+' print(f"\nPattern: {pattern}") for s in ['Hello World', 'ABC', 'hello123']: matches = re.findall(pattern, s) print(f" '{s}' → {matches}") # 4. SHORTHAND CLASSES print("\n=== Shorthand Classes ===") print(""" \\d digit [0-9] \\D non-digit [^0-9] \\w word char [a-zA-Z0-9_] \\W non-word char [^a-zA-Z0-9_] \\s whitespace [ \\t\\n\\r\\f\\v] \\S non-whitespace [^ \\t\\n\\r\\f\\v] """) text = "Hello World 123!" print(f"Text: {text}") print(f" \\d+ → {re.findall(r'\\d+', text)}") print(f" \\w+ → {re.findall(r'\\w+', text)}") print(f" \\s+ → {re.findall(r'\\s+', text)}") print(f" \\D+ → {re.findall(r'\\D+', text)}") print(f" \\W+ → {re.findall(r'\\W+', text)}") print(f" \\S+ → {re.findall(r'\\S+', text)}") # 5. COMBINED CLASSES print("\n=== Combined Classes ===") print("[a-zA-Z0-9_] — same as \\w") print("[^a-zA-Z0-9_] — same as \\W") print("[ \\t\\n\\r\\f\\v] — same as \\s") pattern = r'[a-zA-Z0-9_]+' print(f"\nPattern: {pattern}") for s in ['hello_world', 'hello-world', 'hello world']: matches = re.findall(pattern, s) print(f" '{s}' → {matches}") # 6. SPECIAL CHARACTERS IN CLASSES print("\n=== Special Characters in Classes ===") print(""" Inside [ ], most metacharacters lose their special meaning: [.] matches literal dot [*] matches literal star [+] matches literal plus [?] matches literal question mark [()] matches literal parens [{}] matches literal braces But these ARE special inside [ ]: ^ negation (only at start) - range (between chars) ] end of class \\ escape """) # 7. PRACTICAL EXAMPLES print("=== Practical Examples ===") examples = [ (r'[aeiou]', 'Hello World', 'vowels'), (r'[^aeiou]', 'Hello', 'consonants'), (r'[A-Z][a-z]+', 'Hello World', 'capitalized words'), (r'[0-9]{3}-[0-9]{4}', 'Call 555-1234', 'phone pattern'), (r'[\\w.-]+@[\\w.-]+', 'user@example.com', 'email-ish'), (r'[^\\s]+', 'hello world', 'non-space runs'), ] for pattern, text, desc in examples: matches = re.findall(pattern, text) print(f" {pattern:<25} on '{text}'") print(f" → {matches} ({desc})") # 8. HEX AND UNICODE print("\n=== Hex and Unicode ===") print(""" \\x41 hex character (A) \\u0041 Unicode character (A) \\101 octal character (A) Character ranges with Unicode: [\\u0041-\\u005A] A-Z [\\u0061-\\u007A] a-z [\\u4E00-\\u9FFF] CJK characters """) # 9. CHARACTER CLASS SUBTRACTION (Java, .NET) print("\n=== Character Class Operations ===") print(""" Some engines support set operations: [a-z&&[^aeiou]] consonants (Java) [a-z--[aeiou]] consonants (some engines) Python doesn't support these directly, but you can simulate with lookahead. """) # 10. COMMON PATTERNS print("=== Common Character Class Patterns ===") print(""" \\d+ one or more digits \\d{3}-\\d{4} phone: 555-1234 [a-zA-Z]+ one or more letters [a-zA-Z0-9_]+ identifier [^\\s]+ non-whitespace token [\\w.-]+@[\\w.-]+ simple email \\d{4}-\\d{2}-\\d{2} ISO date [0-9a-fA-F]{6} hex color [\\x20-\\x7E] printable ASCII """)

🔢 Quantifiers

🎯 How Many Times?

Quantifiers specify how many times a pattern should repeat. They can be greedy (match as much as possible) or lazy (match as little as possible).

Quantifier Types

*: 0 or more. +: 1 or more. ?: 0 or 1. {n}: exactly n. {n,}: n or more. {n,m}: between n and m. Lazy: Append ? to make lazy. Possessive: Append + (some engines).

* 0++ 1+? 0/1{n,m} Range
# Quantifiers in detail import re # 1. BASIC QUANTIFIERS print("=== Basic Quantifiers ===") print(""" * 0 or more + 1 or more ? 0 or 1 (optional) {n} exactly n {n,} n or more {n,m} between n and m (inclusive) """) # Star print("Star (*) — 0 or more:") pattern = r'ab*c' for s in ['ac', 'abc', 'abbc', 'abbbc', 'adc']: match = re.search(pattern, s) print(f" '{s}' → {'match' if match else 'no match'}") # Plus print("\nPlus (+) — 1 or more:") pattern = r'ab+c' for s in ['ac', 'abc', 'abbc', 'abbbc']: match = re.search(pattern, s) print(f" '{s}' → {'match' if match else 'no match'}") # Question print("\nQuestion (?) — 0 or 1:") pattern = r'colou?r' for s in ['color', 'colour', 'colouur']: match = re.search(pattern, s) print(f" '{s}' → {'match' if match else 'no match'}") # Exact print("\nExact ({n}):") pattern = r'a{3}' for s in ['aa', 'aaa', 'aaaa']: match = re.search(pattern, s) print(f" '{s}' → {'match' if match else 'no match'}") # Range print("\nRange ({n,m}):") pattern = r'a{2,4}' for s in ['a', 'aa', 'aaa', 'aaaa', 'aaaaa']: match = re.search(pattern, s) print(f" '{s}' → {'match' if match else 'no match'}") # 2. GREEDY VS LAZY print("\n=== Greedy vs Lazy ===") print(""" Greedy (default): match as much as possible Lazy (add ?): match as little as possible Greedy: * + ? {n,m} Lazy: *? +? ?? {n,m}? """) text = "" print(f"Text: {text}") # Greedy greedy = re.search(r'<.*>', text) print(f"Greedy <.*> → '{greedy.group()}'") # Lazy lazy = re.search(r'<.*?>', text) print(f"Lazy <.*?> → '{lazy.group()}'") # All matches all_greedy = re.findall(r'<.*>', text) all_lazy = re.findall(r'<.*?>', text) print(f"Greedy findall → {all_greedy}") print(f"Lazy findall → {all_lazy}") # 3. PRACTICAL GREEDY VS LAZY print("\n=== Practical Greedy vs Lazy ===") text = 'start "hello" and "world" end' # Greedy greedy = re.findall(r'".*"', text) print(f"Greedy: {greedy}") # Lazy lazy = re.findall(r'".*?"', text) print(f"Lazy: {lazy}") # 4. QUANTIFIER PRECEDENCE print("\n=== Quantifier Precedence ===") print(""" Quantifiers apply to the immediately preceding element: ab* → a followed by 0+ b's (ab)* → 0+ repetitions of "ab" ab+ → a followed by 1+ b's a{2,3} → 2-3 a's """) # 5. NESTED QUANTIFIERS print("\n=== Nested Quantifiers ===") print(""" Can combine quantifiers with groups: (ab)+ 1+ repetitions of "ab" (a|b)* 0+ a's or b's (a+)+ 1+ groups of 1+ a's (dangerous!) (ab*)+ repetitions of "a" followed by b's Warning: Nested quantifiers can cause catastrophic backtracking! """) # 6. QUANTIFIER ON GROUPS print("\n=== Quantifiers on Groups ===") pattern = r'(ab)+' print(f"Pattern: {pattern}") for s in ['ab', 'abab', 'ababab', 'a', 'aba']: match = re.search(pattern, s) print(f" '{s}' → {match.group() if match else 'no match'}") # 7. COMMON PATTERNS print("\n=== Common Quantifier Patterns ===") print(""" \\d+ 1+ digits \\d{3} exactly 3 digits \\d{3,} 3+ digits \\d{3,5} 3-5 digits \\w+ 1+ word chars \\s* 0+ whitespace .* 0+ any chars (greedy) .*? 0+ any chars (lazy) a{2,4} 2-4 a's (colou)?r optional "colou" """) # 8. POSSESSIVE QUANTIFIERS print("\n=== Possessive Quantifiers ===") print(""" Some engines (Java, PCRE) support possessive quantifiers: *+ ++ ?+ {n,m}+ Possessive: match as much as possible, NO backtracking. - Faster (no backtracking) - Can change match behavior Python doesn't support possessive quantifiers. """) # 9. REAL-WORLD EXAMPLES print("\n=== Real-World Examples ===") examples = [ (r'\\d+', 'Order #12345', 'order number'), (r'\\d{4}-\\d{2}-\\d{2}', '2024-01-15', 'ISO date'), (r'\\w+@\\w+\\.\\w+', 'user@example.com', 'simple email'), (r'<[^>]+>', '

Hello

', 'HTML tags'), (r'\\s+', 'hello world', 'whitespace runs'), (r'colou?r', 'color and colour', 'optional u'), ] for pattern, text, desc in examples: matches = re.findall(pattern, text) print(f" {pattern:<25} on '{text}'") print(f" → {matches} ({desc})") # 10. PERFORMANCE NOTE print("\n=== Performance Note ===") print(""" Quantifier performance: FAST: \\d+ — simple, linear [a-z]+ — character class .*? — lazy, usually fine a{3,5} — bounded SLOW (dangerous): (a+)+ — nested quantifiers (a|a)* — ambiguous alternation .*.*.* — multiple wildcards (x+x+)+y — classic ReDoS Rule of thumb: - Avoid nested quantifiers - Avoid overlapping alternatives - Use possessive/lazy when possible - Test with long inputs """)

⚓ Anchors & Boundaries

🎯 Matching Positions, Not Characters

Anchors don't match characters — they match positions in the string. They're zero-width assertions that tell the regex engine where to look.

Anchor Types

^: Start of string (or line with MULTILINE). $: End of string (or line). \A: Start of string (never multiline). \Z: End of string. \b: Word boundary. \B: Non-word boundary. \G: End of previous match.

^ Start$ End\b WordZero-width
# Anchors and Boundaries import re # 1. START AND END print("=== ^ and $ ===") print(""" ^ start of string (or line with re.MULTILINE) $ end of string (or line with re.MULTILINE) """) # Without anchors — matches anywhere print("Without anchors:") print(f" r'cat' on 'the cat sat' → {re.search(r'cat', 'the cat sat')}") # With anchors print("\nWith anchors:") pattern = r'^cat$' for s in ['cat', 'the cat', 'cat sat', 'cat']: match = re.search(pattern, s) print(f" r'{pattern}' on '{s}' → {'match' if match else 'no match'}") # 2. FULL STRING MATCH print("\n=== Full String Match ===") print("Use ^...$ or re.fullmatch()") pattern = r'^\\d{3}-\\d{4}$' for s in ['555-1234', '555-12345', 'x555-1234', '555-1234x']: match = re.search(pattern, s) print(f" '{s}' → {'✓' if match else '✗'}") # 3. MULTILINE print("\n=== Multiline Mode ===") text = """first line second line third line""" print("Without MULTILINE:") print(f" ^second → {re.search(r'^second', text)}") print("With MULTILINE:") print(f" ^second → {re.search(r'^second', text, re.MULTILINE)}") # Find all lines starting with "t" matches = re.findall(r'^t\\w+', text, re.MULTILINE) print(f" Lines starting with 't': {matches}") # 4. STRING ANCHORS print("\n=== \\A and \\Z ===") print(""" \\A start of string (never multiline) \\Z end of string (never multiline) \\z end of string (absolute, some engines) Use \\A and \\Z when you want true string boundaries. """) text = "line1\nline2" print(f"Pattern r'\\Aline1' on '{text}': {re.search(r'\\Aline1', text)}") print(f"Pattern r'line2\\Z' on '{text}': {re.search(r'line2\\Z', text)}") # 5. WORD BOUNDARIES print("\n=== Word Boundaries (\\b) ===") print(""" \\b word boundary (between \\w and \\W) \\B non-word boundary Word boundary positions: - Before first word char - After last word char - Between word char and non-word char """) text = "cat cats concat cat." print(f"Text: {text}") print(f" r'\\bcat\\b' → {re.findall(r'\\bcat\\b', text)}") print(f" r'\\bcat' → {re.findall(r'\\bcat', text)}") print(f" r'cat\\b' → {re.findall(r'cat\\b', text)}") # 6. PRACTICAL USE OF BOUNDARIES print("\n=== Practical Use ===") text = "The cat scattered cats on the concat." print(f"Text: {text}") # Match whole word "cat" matches = re.findall(r'\bcat\b', text) print(f" Whole word 'cat': {matches}") # Match "cat" at start of word matches = re.findall(r'\bcat\w*', text) print(f" Words starting with 'cat': {matches}") # Match "cat" at end of word matches = re.findall(r'\w*cat\b', text) print(f" Words ending with 'cat': {matches}") # 7. ANCHORS IN PRACTICE print("\n=== Real-World Examples ===") examples = [ (r'^\\d+$', '12345', 'all digits'), (r'^\\d+$', '12a45', 'not all digits'), (r'^\\w+$', 'hello_world', 'word string'), (r'^\\w+$', 'hello world', 'contains space'), (r'^[A-Z]', 'Hello', 'starts with capital'), (r'\\.$', 'Hello.', 'ends with period'), (r'^$', '', 'empty string'), ] for pattern, text, desc in examples: match = re.search(pattern, text) result = "✓" if match else "✗" print(f" {pattern:<15} on '{text}': {result} ({desc})") # 8. LOOKING AHEAD — PREVIEW print("\n=== Preview: Lookahead/Lookbehind ===") print(""" Zero-width assertions that match positions: (?=...) positive lookahead (?!...) negative lookahead (?<=...) positive lookbehind (?

📦 Groups & Capturing

🎯 Grouping and Capturing Substrings

Groups use parentheses ( ) to:
1. Apply quantifiers to multiple characters
2. Capture matched substrings for extraction
3. Create backreferences

Group Types

Capturing: (pattern) — captures match. Non-capturing: (?:pattern) — groups without capture. Named: (?P<name>pattern) — named capture. Atomic: (?>pattern) — no backtracking. Conditional: (?(name)yes|no). Backreference: \1, \2, etc.

( ) Capture(?: ) Non-capture(?P<name>) Named\1 Backref
# Groups and Capturing import re # 1. BASIC CAPTURING GROUPS print("=== Basic Capturing Groups ===") pattern = r'(\\d{4})-(\\d{2})-(\\d{2})' text = 'Date: 2024-01-15' match = re.search(pattern, text) if match: print(f"Full match: {match.group(0)}") print(f"Group 1 (year): {match.group(1)}") print(f"Group 2 (month): {match.group(2)}") print(f"Group 3 (day): {match.group(3)}") print(f"All groups: {match.groups()}") # 2. GROUPS WITH QUANTIFIERS print("\n=== Groups with Quantifiers ===") # Without group: only applies to last char pattern = r'ab+' print(f"Pattern: {pattern}") print(f" 'ab' → {re.search(pattern, 'ab')}") print(f" 'abb' → {re.search(pattern, 'abb')}") # With group: applies to whole sequence pattern = r'(ab)+' print(f"\nPattern: {pattern}") for s in ['ab', 'abab', 'ababab', 'a']: match = re.search(pattern, s) print(f" '{s}' → {match.group() if match else 'no match'}") # 3. ALTERNATION WITH GROUPS print("\n=== Alternation with Groups ===") pattern = r'(cat|dog|bird)' text = 'I have a dog and a cat' matches = re.findall(pattern, text) print(f"Pattern: {pattern}") print(f"Text: {text}") print(f"Matches: {matches}") # 4. NON-CAPTURING GROUPS print("\n=== Non-Capturing Groups (?:...) ===") print("Use (?:...) when you need grouping but not capturing") # Capturing pattern_cap = r'(cat|dog)s?' print(f"Capturing {pattern_cap}:") matches = re.findall(pattern_cap, 'cats and dogs') print(f" {matches}") # Non-capturing pattern_noncap = r'(?:cat|dog)s?' print(f"Non-capturing {pattern_noncap}:") matches = re.findall(pattern_noncap, 'cats and dogs') print(f" {matches}") # 5. NAMED GROUPS print("\n=== Named Groups (?P...) ===") pattern = r'(?P\\d{4})-(?P\\d{2})-(?P\\d{2})' text = '2024-01-15' match = re.search(pattern, text) if match: print(f"Full match: {match.group(0)}") print(f"Year: {match.group('year')}") print(f"Month: {match.group('month')}") print(f"Day: {match.group('day')}") print(f"Group dict: {match.groupdict()}") # 6. GROUPS IN SUBSTITUTION print("\n=== Groups in Substitution ===") # Reformat date text = '2024-01-15' pattern = r'(\\d{4})-(\\d{2})-(\\d{2})' replacement = r'\\3/\\2/\\1' # day/month/year result = re.sub(pattern, replacement, text) print(f"Original: {text}") print(f"Reformatted: {result}") # Swap names text = 'John Smith' pattern = r'(\\w+) (\\w+)' replacement = r'\\2, \\1' result = re.sub(pattern, replacement, text) print(f"\nOriginal: {text}") print(f"Swapped: {result}") # 7. NESTED GROUPS print("\n=== Nested Groups ===") pattern = r'((\\d{4})-(\\d{2})-(\\d{2}))' text = '2024-01-15' match = re.search(pattern, text) if match: print(f"Group 0 (full): {match.group(0)}") print(f"Group 1 (whole date): {match.group(1)}") print(f"Group 2 (year): {match.group(2)}") print(f"Group 3 (month): {match.group(3)}") print(f"Group 4 (day): {match.group(4)}") # 8. PRACTICAL EXAMPLES print("\n=== Practical Examples ===") # Extract email parts text = 'Contact: user@example.com' pattern = r'(\\w+)@(\\w+)\\.(\\w+)' match = re.search(pattern, text) if match: print(f"Email: {match.group(0)}") print(f" User: {match.group(1)}") print(f" Domain: {match.group(2)}") print(f" TLD: {match.group(3)}") # Parse URL url = 'https://example.com:8080/path?query=value' pattern = r'(?Phttps?)://(?P[^:/]+)(?::(?P\\d+))?(?P/[^?]*)?' match = re.search(pattern, url) if match: print(f"\nURL: {url}") for key, value in match.groupdict().items(): print(f" {key}: {value}") # 9. BACKREFERENCES (PREVIEW) print("\n=== Backreferences (Preview) ===") print(""" \\1, \\2, etc. refer to captured groups: (\\w+) \\1 — repeated word (\\d)\\1 — repeated digit <(\\w+)>.*? — matching HTML tags Covered in detail in the next section. """) # 10. COMMON GROUP PATTERNS print("=== Common Group Patterns ===") print(""" (\\d{4})-(\\d{2})-(\\d{2}) date parts (\\w+)@(\\w+)\\.(\\w+) email parts (https?)://([^/]+)(/.*)? URL parts (\\d+)\\.(\\d+)\\.(\\d+)\\.(\\d+) IP address (?:Mr|Mrs|Ms)\\.?\\s+(\\w+) optional title (?P\\w+) named group (?P\\w+)\\s+(?P\\w+) first/last name """)

👀 Lookahead & Lookbehind

🎯 Zero-Width Assertions

Lookaround assertions check for a pattern without consuming characters. They match a position, not text, making them powerful for context-sensitive matching.

Lookaround Types

(?=...): Positive lookahead — followed by. (?!...): Negative lookahead — not followed by. (?<=...): Positive lookbehind — preceded by. (?<!...): Negative lookbehind — not preceded by. Zero-width: Don't consume input. Variable-length: Some engines restrict lookbehind length.

(?= ) Ahead(?! ) Negative(?<= ) Behind(?<! ) Negative
# Lookahead and Lookbehind import re # 1. POSITIVE LOOKAHEAD (?=...) print("=== Positive Lookahead (?=...) ===") print("Matches if followed by pattern, without consuming it") # Match "foo" only if followed by "bar" pattern = r'foo(?=bar)' for s in ['foobar', 'foobaz', 'fooqux']: match = re.search(pattern, s) print(f" r'{pattern}' on '{s}' → {match.group() if match else 'no match'}") # 2. NEGATIVE LOOKAHEAD (?!...) print("\n=== Negative Lookahead (?!...) ===") print("Matches if NOT followed by pattern") # Match "foo" only if NOT followed by "bar" pattern = r'foo(?!bar)' for s in ['foobar', 'foobaz', 'fooqux']: match = re.search(pattern, s) print(f" r'{pattern}' on '{s}' → {match.group() if match else 'no match'}") # 3. POSITIVE LOOKBEHIND (?<=...) print("\n=== Positive Lookbehind (?<=...) ===") print("Matches if preceded by pattern") # Match "bar" only if preceded by "foo" pattern = r'(?<=foo)bar' for s in ['foobar', 'bazbar', 'foobaz']: match = re.search(pattern, s) print(f" r'{pattern}' on '{s}' → {match.group() if match else 'no match'}") # 4. NEGATIVE LOOKBEHIND (?

🔗 Backreferences

🎯 Referencing Captured Groups

Backreferences let you match the same text that was previously captured by a group. They're written as \1, \2, etc., or \k<name> for named groups.

Backreference Types

\1, \2: Numeric backreference. \k<name>: Named backreference. (?P=name): Python named backref. Same text: Must match exactly. Recursion: Some engines support. Not regular: Backrefs go beyond regular languages.

\1 Numeric\k<name> Named(?P=name) PythonSame text
# Backreferences import re # 1. BASIC BACKREFERENCES print("=== Basic Backreferences ===") print("\\1, \\2, etc. match the same text as group N") # Match repeated words pattern = r'\\b(\\w+)\\s+\\1\\b' text = "hello hello world world foo bar" matches = re.findall(pattern, text) print(f"Pattern: {pattern}") print(f"Text: {text}") print(f"Repeated words: {matches}") # 2. REPEATED CHARACTERS print("\n=== Repeated Characters ===") # Match double letters pattern = r'(\\w)\\1' text = "hello world" matches = re.findall(pattern, text) print(f"Pattern: {pattern}") print(f"Text: {text}") print(f"Double letters: {matches}") # Match repeated digits pattern = r'(\\d)\\1' text = "112233 456 7788" matches = re.findall(pattern, text) print(f"Pattern: {pattern}") print(f"Text: {text}") print(f"Repeated digits: {matches}") # 3. MATCHING HTML TAGS print("\n=== Matching HTML Tags ===") # Match ... with matching tags pattern = r'<(\\w+)>.*?' text = '
Hello
World
Test
' matches = re.findall(pattern, text) print(f"Pattern: {pattern}") print(f"Text: {text}") print(f"Matching tags: {matches}") # 4. NAMED BACKREFERENCES print("\n=== Named Backreferences ===") # Python syntax: (?P...) and (?P=name) pattern = r'(?P\\w+)\\s+(?P=word)' text = "hello hello world" matches = re.findall(pattern, text) print(f"Pattern: {pattern}") print(f"Text: {text}") print(f"Repeated words: {matches}") # 5. PRACTICAL: DUPLICATE DETECTION print("\n=== Duplicate Detection ===") # Find duplicate lines text = """apple banana apple cherry banana""" lines = text.split('\n') seen = set() duplicates = [] for line in lines: if line in seen: duplicates.append(line) seen.add(line) print(f"Text:\n{text}") print(f"Duplicates: {duplicates}") # 6. PRACTICAL: PALINDROME CHECK print("\n=== Palindrome Check ===") def is_palindrome(s): # Simple palindrome check s = s.lower().replace(' ', '') return s == s[::-1] # Regex palindrome (only for specific length) def is_palindrome_regex(s): """Check if string is palindrome using regex.""" # For even-length palindromes pattern = r'^(.)(.)(.)(.)\\4\\3\\2\\1$' match = re.match(pattern, s) if match: return True # For odd-length palindromes pattern = r'^(.)(.)(.)(.)(.)\\4\\3\\2\\1$' match = re.match(pattern, s) return bool(match) test_words = ['racecar', 'abba', 'abcba', 'hello', 'aa', 'ab'] print("Palindrome checks:") for word in test_words: result = is_palindrome(word) print(f" '{word}' → {'✓' if result else '✗'}") # 7. BACKREFERENCE IN SUBSTITUTION print("\n=== Backreferences in Substitution ===") # Swap first and last names text = "John Smith" pattern = r'(\\w+) (\\w+)' replacement = r'\\2, \\1' result = re.sub(pattern, replacement, text) print(f"Original: {text}") print(f"Swapped: {result}") # Wrap matched text text = "hello world" pattern = r'(\\w+)' replacement = r'[\\1]' result = re.sub(pattern, replacement, text) print(f"\nOriginal: {text}") print(f"Wrapped: {result}") # 8. BACKREFERENCE LIMITATIONS print("\n=== Backreference Limitations ===") print(""" 1. NOT REGULAR Backreferences go beyond regular languages. They require unbounded memory. Example: L = {ww | w ∈ {a,b}*} This is NOT regular but can be matched with backrefs. 2. PERFORMANCE Backreferences can cause catastrophic backtracking. Use with caution. 3. ENGINE SUPPORT - Python: supports \\1, (?P=name) - JavaScript: supports \\1 - PCRE: supports \\1, \\k - RE2: DOES NOT support backreferences - Go: DOES NOT support backreferences 4. NO DFA Languages with backrefs can't be converted to DFA. Must use backtracking engine. """) # 9. COMMON BACKREFERENCE PATTERNS print("=== Common Backreference Patterns ===") print(""" (\\w+)\\s+\\1 repeated word (\\w)\\1 double letter (\\d)\\1 double digit <(\\w+)>.*? matching HTML tags (["']).*?\\1 matching quotes (\\w+)\\s+\\1\\s+\\1 word repeated 3 times (?P["']).*?(?P=quote) named quotes """)

🚩 Flags & Modifiers

🎯 Changing Regex Behavior

Flags (or modifiers) change how a regex engine interprets a pattern. They can enable case-insensitive matching, multiline mode, dot-all mode, and more.

Common Flags

i: Case-insensitive. m: Multiline (^ and $ per line). s: Dot-all (dot matches newline). x: Verbose (whitespace ignored). g: Global (find all). u: Unicode. a: ASCII-only. L: Locale.

i Ignore casem Multilines Dot-allx Verbose
# Flags and Modifiers import re # 1. IGNORECASE (i) print("=== IGNORECASE (i) ===") pattern = r'hello' text = "Hello World HELLO hello" # Without flag matches = re.findall(pattern, text) print(f"Without flag: {matches}") # With flag matches = re.findall(pattern, text, re.IGNORECASE) print(f"With IGNORECASE: {matches}") # 2. MULTILINE (m) print("\n=== MULTILINE (m) ===") text = """first line second line third line""" # Without MULTILINE pattern = r'^\\w+' matches = re.findall(pattern, text) print(f"Without MULTILINE: {matches}") # With MULTILINE matches = re.findall(pattern, text, re.MULTILINE) print(f"With MULTILINE: {matches}") # 3. DOTALL (s) print("\n=== DOTALL (s) ===") print("Makes . match newlines too") text = "line1\nline2\nline3" # Without DOTALL pattern = r'line1.line2' match = re.search(pattern, text) print(f"Without DOTALL: {match}") # With DOTALL match = re.search(pattern, text, re.DOTALL) print(f"With DOTALL: {match}") # 4. VERBOSE (x) print("\n=== VERBOSE (x) ===") print("Whitespace ignored, comments allowed") # Without VERBOSE pattern1 = r'\\d{4}-\\d{2}-\\d{2}' print(f"Without VERBOSE: {pattern1}") # With VERBOSE pattern2 = r""" \\d{4} # year - # separator \\d{2} # month - # separator \\d{2} # day """ print(f"With VERBOSE: {repr(pattern2)}") text = "2024-01-15" match = re.search(pattern2, text, re.VERBOSE) print(f"Match: {match.group() if match else 'no match'}") # 5. COMBINING FLAGS print("\n=== Combining Flags ===") text = "Hello\nWorld" # Multiple flags with | pattern = r'hello.*world' match = re.search(pattern, text, re.IGNORECASE | re.DOTALL) print(f"With IGNORECASE | DOTALL: {match}") # 6. INLINE FLAGS print("\n=== Inline Flags ===") print("(?i) case-insensitive") print("(?m) multiline") print("(?s) dot-all") print("(?x) verbose") # Inline flag pattern = r'(?i)hello' text = "HELLO World" match = re.search(pattern, text) print(f"Pattern: {pattern}") print(f"Text: {text}") print(f"Match: {match.group() if match else 'no match'}") # 7. FLAG SCOPE print("\n=== Flag Scope ===") print("(?i:...) apply flag to group only") print("(?-i:...) disable flag in group") pattern = r'(?i:hello) world' text = "HELLO World" match = re.search(pattern, text) print(f"Pattern: {pattern}") print(f"Text: {text}") print(f"Match: {match.group() if match else 'no match'}") # 8. PRACTICAL EXAMPLES print("\n=== Practical Examples ===") # Case-insensitive search text = "The Quick Brown Fox JUMPS over the LAZY dog" pattern = r'\\b\\w+\\b' matches = re.findall(pattern, text) print(f"All words: {matches[:5]}...") # Multiline log parsing log = """2024-01-15 ERROR: Connection failed 2024-01-15 INFO: Retrying 2024-01-15 ERROR: Timeout""" pattern = r'^.*ERROR.*$' errors = re.findall(pattern, log, re.MULTILINE) print(f"\nErrors:\n" + "\n".join(errors)) # Dot-all for multiline strings text = "start\nmiddle\nend" pattern = r'start.*end' match = re.search(pattern, text, re.DOTALL) print(f"\nDot-all match: {match.group() if match else 'no match'}") # 9. FLAG REFERENCE print("\n=== Flag Reference ===") print(""" Python re module flags: re.IGNORECASE (i) Case-insensitive matching re.MULTILINE (m) ^ and $ match at line breaks re.DOTALL (s) . matches newline re.VERBOSE (x) Whitespace and comments ignored re.ASCII (a) ASCII-only matching re.UNICODE (u) Unicode matching (default in Py3) re.LOCALE (L) Locale-dependent matching Can combine with |: re.IGNORECASE | re.MULTILINE Inline flags: (?i) case-insensitive (?m) multiline (?s) dot-all (?x) verbose (?i:...) scoped flag """) # 10. JAVASCRIPT FLAGS print("=== JavaScript Flags ===") print(""" JavaScript regex flags: g Global — find all matches i Ignore case m Multiline s Dot-all u Unicode y Sticky — match from lastIndex Example: /pattern/gi new RegExp('pattern', 'gi') """)

⚙️ Regex Engines

🎯 How Regex is Implemented

There are two main types of regex engines: DFA-based (linear time, no backrefs) and backtracking-based (full features, exponential worst case).

Engine Types

DFA: Thompson NFA, linear time O(n×m). Backtracking: Perl, Python, Java. Hybrid: PCRE, Oniguruma. RE2: Google's linear-time engine. Hyperscan: SIMD-accelerated. Rust regex: Linear time, no backrefs.

DFA LinearBacktracking FeaturesRE2 FastHyperscan SIMD
# Regex Engines — comparison and implementation print("=== Regex Engine Types ===") print(""" 1. DFA-BASED (Thompson NFA) - Linear time: O(n × m) - No backreferences - No lookaround (or limited) - Predictable performance - Examples: RE2, Rust regex, Go regexp 2. BACKTRACKING - Exponential worst case: O(2^n) - Full features (backrefs, lookaround) - Can be optimized (memoization) - Examples: Perl, Python re, Java, JavaScript 3. HYBRID - Try DFA first, fall back to backtracking - Best of both worlds - Examples: PCRE2, Oniguruma (Ruby) 4. SIMD-ACCELERATED - Parallel matching - Very fast - Examples: Hyperscan (Intel) """) # 1. PYTHON RE — BACKTRACKING print("\n=== Python re — Backtracking ===") import re import time # Demonstrate backtracking behavior def test_backtracking(): """Show backtracking with nested quantifiers.""" pattern = r'(a+)+b' text = 'a' * 20 + 'c' # no 'b' — will backtrack start = time.time() match = re.search(pattern, text) elapsed = time.time() - start print(f"Pattern: {pattern}") print(f"Text: 'a' * 20 + 'c'") print(f"Time: {elapsed:.4f}s") print(f"Match: {match}") test_backtracking() # 2. RE2 — LINEAR TIME print("\n=== RE2 — Linear Time ===") print(""" RE2 (Google) uses Thompson NFA: - Linear time guaranteed - No backreferences - No lookaround (limited) - Safe for untrusted input Python doesn't have RE2 built-in, but you can use the `re2` module: pip install re2 Or use `google-re2`: pip install google-re2 """) # 3. HYPERS CAN — SIMD print("\n=== Hyperscan — SIMD ===") print(""" Intel Hyperscan: - SIMD-accelerated - Multiple patterns simultaneously - 100+ GB/s throughput - Used in Snort, Suricata Available as: - C library - Python bindings (hyperscan) - Rust bindings """) # 4. RUST REGEX — LINEAR TIME print("\n=== Rust regex — Linear Time ===") print(""" Rust regex crate: - Guaranteed linear time - No backreferences - No lookaround (use fancy-regex for those) - Very fast Features: - Unicode support - SIMD acceleration - Lazy compilation """) # 5. ENGINE COMPARISON print("\n=== Engine Comparison ===") print(f"{'Engine':<20} {'Type':<15} {'Time':<15} {'Backrefs':<10} {'Lookaround':<12}") print("-" * 75) print(f"{'Python re':<20} {'Backtracking':<15} {'Exponential':<15} {'Yes':<10} {'Yes':<12}") print(f"{'JavaScript':<20} {'Backtracking':<15} {'Exponential':<15} {'Yes':<10} {'Yes':<12}") print(f"{'Java':<20} {'Backtracking':<15} {'Exponential':<15} {'Yes':<10} {'Yes':<12}") print(f"{'PCRE':<20} {'Hybrid':<15} {'Exponential':<15} {'Yes':<10} {'Yes':<12}") print(f"{'RE2':<20} {'DFA':<15} {'Linear':<15} {'No':<10} {'Limited':<12}") print(f"{'Rust regex':<20} {'DFA':<15} {'Linear':<15} {'No':<10} {'No':<12}") print(f"{'Go regexp':<20} {'DFA':<15} {'Linear':<15} {'No':<10} {'No':<12}") print(f"{'Hyperscan':<20} {'DFA+SIMD':<15} {'Linear':<15} {'No':<10} {'Limited':<12}") # 6. CHOOSING AN ENGINE print("\n=== Choosing an Engine ===") print(""" USE DFA/RE2 WHEN: ✓ Processing untrusted input ✓ Need guaranteed performance ✓ Pattern is regular (no backrefs) ✓ Matching many strings USE BACKTRACKING WHEN: ✓ Need backreferences ✓ Need lookaround ✓ Pattern is small ✓ Input is trusted HYBRID APPROACH: - Use DFA for initial filtering - Use backtracking for complex patterns - E.g., PCRE2, Oniguruma """) # 7. DEMONSTRATION: CATASTROPHIC BACKTRACKING print("\n=== Catastrophic Backtracking Demo ===") def demonstrate_redos(): """Show exponential backtracking.""" patterns = [ (r'(a+)+b', 'nested quantifiers'), (r'(a|a)*b', 'ambiguous alternation'), (r'(.*a){10}', 'multiple wildcards'), ] text = 'a' * 15 + 'c' for pattern, desc in patterns: start = time.time() try: re.search(pattern, text) except: pass elapsed = time.time() - start print(f" {pattern:<20} ({desc})") print(f" Time on 15 a's: {elapsed:.4f}s") demonstrate_redos() # 8. PRACTICAL RECOMMENDATIONS print("\n=== Practical Recommendations ===") print(""" 1. FOR WEB APPS - Use RE2 or Rust regex for user input - Avoid backtracking engines for validation - Set timeout on regex operations 2. FOR DATA PROCESSING - Use Hyperscan for high throughput - Use DFA engines for large datasets - Batch multiple patterns 3. FOR SCRIPTING - Python re is fine for trusted input - Use regex module for better performance - pip install regex 4. FOR PRODUCTION - Profile your regex - Test with adversarial input - Consider engine limitations """) # 9. REGEX MODULE (PYTHON) print("\n=== Python regex Module ===") print(""" The `regex` module is a drop-in replacement for `re`: pip install regex Features: - Better Unicode support - Variable-length lookbehind - Possessive quantifiers - Set operations - Better performance Example: import regex regex.search(r'(?<=a+)b', 'aaab') """)

⚡ Performance & ReDoS

🎯 Making Regex Fast and Safe

Regex performance can range from linear to exponential. Understanding and preventing ReDoS (Regular Expression Denial of Service) is critical for production systems.

Performance Factors

Engine: DFA vs backtracking. Pattern: Simple vs complex. Input: Length and content. Backtracking: Exponential worst case. ReDoS: Attack via crafted input. Optimization: Anchors, character classes, lazy quantifiers.

⚡ Linear🐌 Exponential🛡️ ReDoS📊 Profiling
# Regex Performance and ReDoS import re import time # 1. PERFORMANCE COMPARISON print("=== Performance Comparison ===") def measure(pattern, text, flags=0): start = time.time() re.search(pattern, text, flags) return time.time() - start text_short = 'a' * 100 text_long = 'a' * 1000 patterns = [ (r'a+', 'simple plus'), (r'a*b', 'star then b'), (r'(a+)+b', 'nested quantifiers'), (r'(a|a)*b', 'ambiguous alternation'), (r'a{1,100}b', 'bounded'), (r'[a-z]+b', 'character class'), ] print(f"{'Pattern':<20} {'Short (100)':<15} {'Long (1000)':<15}") print("-" * 50) for pattern, desc in patterns: t_short = measure(pattern, text_short + 'c') t_long = measure(pattern, text_long + 'c') print(f"{pattern:<20} {t_short:<15.6f} {t_long:<15.6f}") # 2. CATASTROPHIC BACKTRACKING print("\n=== Catastrophic Backtracking ===") print(""" Patterns that cause exponential backtracking: 1. NESTED QUANTIFIERS (a+)+ (a*)* (a+)* 2. AMBIGUOUS ALTERNATION (a|a)* (a|ab)* 3. MULTIPLE WILDCARDS .*.*.* (.*)* 4. OVERLAPPING ALTERNATIVES (a|ab|abc)+ """) def demonstrate_redos_attack(): """Demonstrate ReDoS with increasing input.""" pattern = r'(a+)+b' print(f"Pattern: {pattern}") print(f"{'Input Length':<15} {'Time (s)':<15} {'Growth':<15}") print("-" * 45) for n in [10, 15, 20, 25]: text = 'a' * n + 'c' start = time.time() re.search(pattern, text) elapsed = time.time() - start print(f"{n:<15} {elapsed:<15.6f}") demonstrate_redos_attack() # 3. REDOS PREVENTION print("\n=== ReDoS Prevention ===") print(""" 1. USE DFA ENGINE - RE2, Rust regex - Linear time guaranteed - No backreferences 2. SIMPLIFY PATTERNS ✗ (a+)+b ✓ a+b ✗ (a|a)*b ✓ a*b ✗ (.*a){10} ✓ .*a.*a.*a.*a.*a 3. USE ATOMIC GROUPS (?>a+) — no backtracking into group (PCRE, Java, .NET) 4. USE POSSESSIVE QUANTIFIERS a++ a*+ a?+ (PCRE, Java) 5. SET TIMEOUT - Python: signal.alarm (Unix) - Java: interruptible CharSequence - .NET: Regex.MatchTimeout 6. LIMIT INPUT LENGTH - Validate length first - Truncate if too long """) # 4. OPTIMIZATION TECHNIQUES print("\n=== Optimization Techniques ===") print(""" 1. USE ANCHORS ^pattern$ — requires full match Faster than searching whole string 2. USE CHARACTER CLASSES [abc] faster than (a|b|c) 3. USE POSSESSIVE QUANTIFIERS a++ vs a+ No backtracking 4. AVOID NESTED QUANTIFIERS ✗ (a+)+ ✓ a+ 5. USE NON-CAPTURING GROUPS (?:...) faster than (...) 6. USE LAZY QUANTIFIERS .*? vs .* Less backtracking 7. COMPILE ONCE re.compile(pattern) — reuse 8. USE ATOMIC GROUPS (?>...) — prevent backtracking """) # 5. BENCHMARK EXAMPLE print("\n=== Benchmark: Anchored vs Unanchored ===") text = 'x' * 1000 + 'hello' + 'x' * 1000 # Unanchored start = time.time() for _ in range(1000): re.search(r'hello', text) t1 = time.time() - start # Anchored start = time.time() for _ in range(1000): re.search(r'^.*hello.*$', text) t2 = time.time() - start print(f"Unanchored search: {t1:.6f}s") print(f"Anchored search: {t2:.6f}s") # 6. RE.COMPILE BENCHMARK print("\n=== re.compile Benchmark ===") pattern = r'\\d{4}-\\d{2}-\\d{2}' text = '2024-01-15' # Without compile start = time.time() for _ in range(10000): re.search(pattern, text) t1 = time.time() - start # With compile compiled = re.compile(pattern) start = time.time() for _ in range(10000): compiled.search(text) t2 = time.time() - start print(f"Without compile: {t1:.6f}s") print(f"With compile: {t2:.6f}s") print(f"Speedup: {t1/t2:.2f}x") # 7. RE.SEARCH VS RE.MATCH VS RE.FULLMATCH print("\n=== search vs match vs fullmatch ===") print(""" re.search() — find pattern anywhere re.match() — match at start only re.fullmatch() — match entire string Performance: fullmatch > match > search Use the most specific function! """) text = 'hello world' print(f"re.search(r'world', '{text}'): {re.search(r'world', text)}") print(f"re.match(r'world', '{text}'): {re.match(r'world', text)}") print(f"re.fullmatch(r'hello world', '{text}'): {re.fullmatch(r'hello world', text)}") # 8. PRACTICAL CHECKLIST print("\n=== Production Checklist ===") print(""" ✓ Use DFA engine for untrusted input ✓ Avoid nested quantifiers ✓ Avoid ambiguous alternation ✓ Use anchors when possible ✓ Compile patterns once ✓ Set timeout for regex operations ✓ Limit input length ✓ Profile with realistic data ✓ Test with adversarial input ✓ Monitor regex performance """) # 9. RE2 EXAMPLE print("\n=== RE2 Example ===") print(""" RE2 is Google's linear-time regex engine. Python: pip install google-re2 import re2 re2.search(pattern, text) Rust: regex = "1" use regex::Regex; let re = Regex::new(r"pattern").unwrap(); Go: import "regexp" re := regexp.MustCompile(`pattern`) """)

💼 Practical Patterns

🎯 Common Regex Patterns

A collection of production-ready regex patterns for common tasks: validation, extraction, parsing, and cleaning.

Pattern Categories

Validation: Email, phone, URL, IP. Extraction: Dates, numbers, tags. Parsing: Logs, CSV, config. Cleaning: Whitespace, HTML, duplicates. Security: Input sanitization. Formatting: Numbers, dates, text.

✅ Validation📤 Extraction📄 Parsing🧹 Cleaning
# Practical Regex Patterns import re # 1. EMAIL VALIDATION print("=== Email Validation ===") EMAIL = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$' emails = [ "user@example.com", "test.user@domain.co.uk", "invalid", "user@invalid", "a@b.com", "user+tag@example.com", ] for email in emails: match = re.match(EMAIL, email) print(f" {email:<30} → {'✓' if match else '✗'}") # 2. PHONE NUMBER print("\n=== Phone Number ===") # US phone numbers PHONE_US = r'^\(?(\d{3})\)?[-.\s]?(\d{3})[-.\s]?(\d{4})$' phones = ['(555) 123-4567', '555-123-4567', '555.123.4567', '5551234567', '555-1234'] for phone in phones: match = re.match(PHONE_US, phone) print(f" {phone:<20} → {'✓' if match else '✗'}") # 3. URL print("\n=== URL ===") URL = r'^(https?|ftp)://[^\s/$.?#].[^\s]*$' urls = ['https://example.com', 'http://test.org/path?q=1', 'ftp://files.com', 'not-a-url'] for url in urls: match = re.match(URL, url) print(f" {url:<35} → {'✓' if match else '✗'}") # 4. IP ADDRESS print("\n=== IP Address ===") IP = r'^(\d{1,3})\.(\d{1,3})\.(\d{1,3})\.(\d{1,3})$' ips = ['192.168.1.1', '10.0.0.1', '255.255.255.255', '999.999.999.999', '1.2.3'] for ip in ips: match = re.match(IP, ip) if match: parts = [int(g) for g in match.groups()] valid = all(0 <= p <= 255 for p in parts) print(f" {ip:<20} → {'✓' if valid else '✗ (invalid octet)'}") else: print(f" {ip:<20} → ✗") # 5. DATE EXTRACTION print("\n=== Date Extraction ===") DATE = r'(\d{4})-(\d{2})-(\d{2})' text = "Start: 2024-01-15, End: 2024-12-31" matches = re.findall(DATE, text) print(f"Text: {text}") print(f"Dates: {matches}") # 6. HTML TAG EXTRACTION print("\n=== HTML Tag Extraction ===") HTML_TAG = r'<(\w+)(?:\s+[^>]*)?>(.*?)' html = '
Hello
World Link' matches = re.findall(HTML_TAG, html) print(f"HTML: {html}") for tag, content in matches: print(f" <{tag}> → {content}") # 7. LOG PARSING print("\n=== Log Parsing ===") LOG = r'^(\d{4}-\d{2}-\d{2})\s+(\d{2}:\d{2}:\d{2})\s+(\w+)\s+(.*)$' log_line = "2024-01-15 10:30:45 ERROR Connection failed" match = re.match(LOG, log_line) if match: date, time, level, message = match.groups() print(f"Date: {date}") print(f"Time: {time}") print(f"Level: {level}") print(f"Message: {message}") # 8. CSV PARSING (simple) print("\n=== CSV Parsing (simple) ===") CSV_FIELD = r'([^,]+)' line = "John,Doe,30,New York" fields = re.findall(CSV_FIELD, line) print(f"Line: {line}") print(f"Fields: {fields}") # 9. WHITESPACE CLEANING print("\n=== Whitespace Cleaning ===") text = " Hello World " cleaned = re.sub(r'\s+', ' ', text).strip() print(f"Original: '{text}'") print(f"Cleaned: '{cleaned}'") # 10. NUMBER FORMATTING print("\n=== Number Formatting ===") # Add commas to numbers number = "1234567" formatted = re.sub(r'(\d)(?=(\d{3})+$)', r'\1,', number) print(f"Original: {number}") print(f"Formatted: {formatted}") # 11. PASSWORD STRENGTH print("\n=== Password Strength ===") def check_password(pwd): checks = { 'length': len(pwd) >= 8, 'lower': bool(re.search(r'[a-z]', pwd)), 'upper': bool(re.search(r'[A-Z]', pwd)), 'digit': bool(re.search(r'\d', pwd)), 'special': bool(re.search(r'[!@#$%^&*(),.?":{}|<>]', pwd)), } score = sum(checks.values()) return score, checks passwords = ['weak', 'Weak123', 'Strong1!', 'V3ry$tr0ng!'] for pwd in passwords: score, checks = check_password(pwd) print(f" {pwd:<15} Score: {score}/5") # 12. EXTRACT NUMBERS print("\n=== Extract Numbers ===") text = "Prices: $10.50, $20, $15.99, $100" numbers = re.findall(r'\$\d+(?:\.\d{2})?', text) print(f"Text: {text}") print(f"Prices: {numbers}") # 13. VALIDATE USERNAME print("\n=== Username Validation ===") USERNAME = r'^[a-zA-Z][a-zA-Z0-9_]{2,15}$' usernames = ['john_doe', 'user123', 'a', '123user', 'very_long_username_here'] for user in usernames: match = re.match(USERNAME, user) print(f" {user:<25} → {'✓' if match else '✗'}") # 14. EXTRACT QUOTED STRINGS print("\n=== Extract Quoted Strings ===") text = 'He said "hello" and then "goodbye"' strings = re.findall(r'"([^"]*)"', text) print(f"Text: {text}") print(f"Quoted: {strings}") # 15. PATTERN CHEAT SHEET print("\n=== Pattern Cheat Sheet ===") print(""" EMAIL: ^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}$ PHONE: ^\\(?\\d{3}\\)?[-.\\s]?\\d{3}[-.\\s]?\\d{4}$ URL: ^https?://[^\\s/$.?#].[^\\s]*$ IP: ^(\\d{1,3}\\.){3}\\d{1,3}$ DATE: ^\\d{4}-\\d{2}-\\d{2}$ TIME: ^\\d{2}:\\d{2}:\\d{2}$ ZIP: ^\\d{5}(-\\d{4})?$ SSN: ^\\d{3}-\\d{2}-\\d{4}$ CREDIT: ^\\d{4}[- ]?\\d{4}[- ]?\\d{4}[- ]?\\d{4}$ HEX: ^#?([0-9a-fA-F]{3}|[0-9a-fA-F]{6})$ USERNAME: ^[a-zA-Z][a-zA-Z0-9_]{2,15}$ PASSWORD: ^(?=.*[a-z])(?=.*[A-Z])(?=.*\\d).{8,}$ SLUG: ^[a-z0-9]+(?:-[a-z0-9]+)*$ """)

🌐 Regex by Language

🎯 Regex Across Programming Languages

While regex syntax is mostly standardized, each language has its own flavor and API. Here's a comparison of regex in popular languages.

Language Differences

Syntax: Mostly PCRE-compatible. API: Differs by language. Flags: Different names. Engine: Backtracking vs DFA. Unicode: Varying support. Performance: Varies significantly.

🐍 Python☕ Java🟨 JavaScript🦀 Rust
# Regex Across Languages # ============================================ # PYTHON # ============================================ print("=== Python (re module) ===") print(""" import re # Basic operations re.search(pattern, text) # find first match re.match(pattern, text) # match at start re.fullmatch(pattern, text) # match entire string re.findall(pattern, text) # find all matches re.finditer(pattern, text) # iterator of matches re.sub(pattern, repl, text) # replace re.split(pattern, text) # split # Flags re.IGNORECASE / re.I re.MULTILINE / re.M re.DOTALL / re.S re.VERBOSE / re.X re.ASCII / re.A re.UNICODE / re.U # Groups m = re.search(r'(\\d+)-(\\d+)', '12-34') m.group(0) # '12-34' m.group(1) # '12' m.group(2) # '34' m.groups() # ('12', '34') m.groupdict() # {} or named groups # Named groups m = re.search(r'(?P\\d{4})', '2024') m.group('year') # '2024' # Compile once pattern = re.compile(r'\\d+') pattern.search('abc 123') # Inline flags re.search(r'(?i)hello', 'HELLO') re.search(r'(?s).*', 'line1\\nline2') """) # ============================================ # JAVASCRIPT # ============================================ print("\n=== JavaScript ===") print(""" // Literal notation const re = /pattern/flags; // Constructor const re = new RegExp('pattern', 'flags'); // Methods re.test(str) // true/false re.exec(str) // match object or null str.match(re) // match array str.matchAll(re) // iterator str.replace(re, repl) str.split(re) // Flags g - global (find all) i - ignore case m - multiline s - dotAll u - unicode y - sticky // Example const re = /(\\d{4})-(\\d{2})-(\\d{2})/; const match = re.exec('2024-01-15'); match[0]; // '2024-01-15' match[1]; // '2024' match[2]; // '01' match[3]; // '15' // Named groups const re = /(?\\d{4})-(?\\d{2})/; const m = re.exec('2024-01'); m.groups.year; // '2024' // Replace with function 'abc123'.replace(/(\\d+)/, (match, p1) => `[${p1}]`); // 'abc[123]' """) # ============================================ # JAVA # ============================================ print("\n=== Java ===") print(""" import java.util.regex.*; // Compile Pattern p = Pattern.compile("pattern"); Matcher m = p.matcher(text); // Methods m.matches() // entire string m.find() // next match m.group() // current match m.group(1) // group 1 m.replaceAll() // replace m.replaceFirst() // Flags Pattern.CASE_INSENSITIVE Pattern.MULTILINE Pattern.DOTALL Pattern.COMMENTS Pattern.UNICODE_CASE // Example Pattern p = Pattern.compile("(\\\\d{4})-(\\\\d{2})"); Matcher m = p.matcher("2024-01-15"); if (m.find()) { System.out.println(m.group(1)); // 2024 System.out.println(m.group(2)); // 01 } // String methods "text".matches("pattern") "text".replaceAll("pattern", "replacement") "text".split("pattern") // Named groups (Java 7+) Pattern p = Pattern.compile("(?\\\\d{4})"); Matcher m = p.matcher("2024"); m.group("year"); // "2024" """) # ============================================ # RUST # ============================================ print("\n=== Rust (regex crate) ===") print(""" use regex::Regex; // Compile let re = Regex::new(r"pattern").unwrap(); // Methods re.is_match(text) // bool re.find(text) // Option re.captures(text) // Option re.replace(text, replacement) re.replace_all(text, replacement) re.split(text) // Example let re = Regex::new(r"(\\d{4})-(\\d{2})").unwrap(); let caps = re.captures("2024-01-15").unwrap(); println!("{}", &caps[1]); // 2024 println!("{}", &caps[2]); // 01 // Named groups let re = Regex::new(r"(?P\\d{4})").unwrap(); let caps = re.captures("2024").unwrap(); println!("{}", &caps["year"]); // 2024 // Flags via builder let re = RegexBuilder::new(r"pattern") .case_insensitive(true) .multi_line(true) .build() .unwrap(); // Note: Rust regex is DFA-based // No backreferences, no lookaround // Use fancy-regex for those features """) # ============================================ # GO # ============================================ print("\n=== Go (regexp package) ===") print(""" import "regexp" // Compile re := regexp.MustCompile(`pattern`) // Methods re.MatchString(text) // bool re.FindString(text) // string re.FindStringSubmatch(text) // []string re.FindAllString(text, -1) // []string re.ReplaceAllString(text, repl) re.Split(text, -1) // Example re := regexp.MustCompile(`(\\d{4})-(\\d{2})`) match := re.FindStringSubmatch("2024-01-15") fmt.Println(match[1]) // 2024 fmt.Println(match[2]) // 01 // Named groups (Go 1.22+) re := regexp.MustCompile(`(?P\\d{4})`) match := re.FindStringSubmatch("2024") fmt.Println(match[re.SubexpIndex("year")]) // Note: Go regex is DFA-based (RE2) // Linear time, no backreferences """) # ============================================ # COMPARISON TABLE # ============================================ print("\n=== Comparison Table ===") print(f"{'Language':<15} {'Engine':<15} {'Backrefs':<10} {'Lookaround':<12} {'Named':<8}") print("-" * 65) print(f"{'Python':<15} {'Backtracking':<15} {'Yes':<10} {'Yes':<12} {'(?P<>)':<8}") print(f"{'JavaScript':<15} {'Backtracking':<15} {'Yes':<10} {'Yes':<12} {'(?<>)':<8}") print(f"{'Java':<15} {'Backtracking':<15} {'Yes':<10} {'Yes':<12} {'(?<>)':<8}") print(f"{'Rust':<15} {'DFA':<15} {'No':<10} {'No*':<12} {'(?P<>)':<8}") print(f"{'Go':<15} {'DFA':<15} {'No':<10} {'No':<12} {'(?P<>)':<8}") print(f"{'PHP':<15} {'PCRE':<15} {'Yes':<10} {'Yes':<12} {'(?P<>)':<8}") print(f"{'Ruby':<15} {'Oniguruma':<15} {'Yes':<10} {'Yes':<12} {'(?<>)':<8}") print(f"{'C#/.NET':<15} {'Backtracking':<15} {'Yes':<10} {'Yes':<12} {'(?<>)':<8}") print(f"{'Perl':<15} {'Backtracking':<15} {'Yes':<10} {'Yes':<12} {'(?<>)':<8}") print("\n* Rust has fancy-regex crate for lookaround") # ============================================ # SYNTAX DIFFERENCES # ============================================ print("\n=== Syntax Differences ===") print(""" NAMED GROUPS: Python: (?P...) JavaScript: (?...) Java: (?...) Rust: (?P...) Go: (?P...) .NET: (?...) or (?'name'...) PHP: (?P...) or (?...) BACKREFERENCE: Numeric: \\1, \\2 Named: Python: (?P=name) JavaScript: \\k Java: \\k .NET: \\k PHP: \\k or (?P=name) INLINE FLAGS: Global: (?i) (?m) (?s) (?x) Scoped: (?i:...) (?-i:...) """)

🚀 Future & Best Practices

🎯 Regex in the Modern World

Regex continues to evolve with better engines, SIMD acceleration, and integration with modern systems. Here are the best practices and future directions.

Future Directions

DFA engines: Linear time becomes standard. SIMD: Hardware acceleration. GPU: Massively parallel. Streaming: Real-time matching. AI: Learned patterns. Verification: Formal methods.

⚡ DFA standard🔀 SIMD🎮 GPU🛡️ Safe by default
# Future & Best Practices print("=== Regex Best Practices ===") print(""" 1. PREFER SIMPLICITY ✗ (?P[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,}) ✓ Keep it simple, test thoroughly 2. USE RAW STRINGS Python: r'\\d+' (not '\\\\d+') Avoid double escaping 3. COMPILE ONCE pattern = re.compile(r'\\d+') Use compiled pattern repeatedly 4. ANCHOR WHEN POSSIBLE ^pattern$ — full string match Faster and more precise 5. AVOID CATASTROPHIC BACKTRACKING ✗ (a+)+b ✓ a+b 6. USE NON-CAPTURING GROUPS (?:...) when you don't need the capture 7. TEST WITH REAL DATA - Valid inputs - Invalid inputs - Edge cases - Adversarial inputs 8. DOCUMENT COMPLEX PATTERNS Use re.VERBOSE mode 9. SET TIMEOUTS For untrusted input 10. CONSIDER ALTERNATIVES - String methods for simple cases - Parsers for structured data - DFA engines for performance """) # 1. VERBOSE PATTERN EXAMPLE print("=== Verbose Pattern Example ===") import re EMAIL_VERBOSE = re.compile(r""" ^ # start of string [a-zA-Z0-9._%+-]+ # local part @ # @ symbol [a-zA-Z0-9.-]+ # domain \\. # dot [a-zA-Z]{2,} # TLD $ # end of string """, re.VERBOSE) print(f"Pattern compiled with re.VERBOSE") print(f"Test 'user@example.com': {EMAIL_VERBOSE.match('user@example.com')}") # 2. TIMEOUT PATTERN print("\n=== Timeout Pattern (Python) ===") print(""" import signal def timeout_handler(signum, frame): raise TimeoutError("Regex timed out") def safe_search(pattern, text, timeout=1): signal.signal(signal.SIGALRM, timeout_handler) signal.alarm(timeout) try: result = re.search(pattern, text) signal.alarm(0) return result except TimeoutError: return None """) # 3. MODERN REGEX ENGINES print("\n=== Modern Regex Engines ===") print(""" 1. RE2 (Google) - Linear time - C++ library - Used by: Google, Cloudflare 2. Hyperscan (Intel) - SIMD-accelerated - Multiple patterns - Used by: Snort, Suricata 3. Rust regex - Linear time - Safe, fast - Growing ecosystem 4. Oniguruma - Full features - Used by: Ruby, PHP 5. PCRE2 - Most feature-complete - Used by: PHP, Apache 6. .NET Regex - Good performance - Full features - Source generators """) # 4. FUTURE DIRECTIONS print("\n=== Future Directions ===") print(""" 1. DFA BY DEFAULT - Linear time for all - Backreferences deprecated? - Safe regex 2. HARDWARE ACCELERATION - FPGA regex - ASIC for network - 100+ Gbps 3. SIMD EVERYWHERE - 64+ states parallel - Multi-pattern - AVX-512, NEON 4. GPU MATCHING - Massive parallelism - Millions of matches - Genomics, security 5. STREAMING NATIVE - Infinite input - Constant memory - Real-time 6. AI-ASSISTED - Generate regex from examples - Learn patterns - Explain matches 7. FORMAL VERIFICATION - Prove regex correctness - Detect bugs - Optimize safely """) # 5. AI-ASSISTED REGEX print("\n=== AI-Assisted Regex ===") print(""" Tools that generate regex from examples: 1. RegexGPT - Describe pattern in English - Get regex 2. Copilot / ChatGPT - "Write regex for email validation" - Get suggestions 3. RegexBuddy - Visual regex builder - Test and debug 4. Debuggex - Visualize regex - Railroad diagrams 5. RegExr - Interactive testing - Community patterns """) # 6. REGEX IN MODERN SYSTEMS print("\n=== Regex in Modern Systems ===") print(""" 1. CLOUD SERVICES - AWS WAF regex rules - Cloudflare firewall - Azure API Management 2. CONTAINERS - Kubernetes selectors - Docker labels - Service mesh routing 3. OBSERVABILITY - Log parsing (Fluentd, Logstash) - Metrics extraction - Alert rules 4. SECURITY - WAF rules - IDS/IPS signatures - Input validation 5. DATA ENGINEERING - ETL pipelines - Data cleaning - Format conversion """) # 7. RECOMMENDATIONS print("\n=== Recommendations ===") print(""" FOR DEVELOPERS: 1. Learn regex basics thoroughly 2. Use DFA engines for untrusted input 3. Test with adversarial inputs 4. Profile performance 5. Document complex patterns FOR ARCHITECTS: 1. Choose engines wisely 2. Set performance budgets 3. Monitor regex performance 4. Plan for scale 5. Consider alternatives FOR SECURITY: 1. Never trust input 2. Use linear-time engines 3. Set timeouts 4. Limit input length 5. Audit regex rules """) # 8. FINAL THOUGHTS print("\n=== Final Thoughts ===") print(""" Regular expressions are: ✓ POWERFUL - Compact pattern description - Universal in programming ✓ VERSATILE - Validation, extraction, parsing - Every language supports them ✓ FAST - Linear time with DFA - Hardware acceleration ✓ FOUNDATIONAL - Based on automata theory - Regular languages ⚠ DANGEROUS - ReDoS attacks - Exponential backtracking - Subtle bugs Best practices: - Keep it simple - Test thoroughly - Use DFA engines - Set timeouts - Document patterns "With great power comes great responsibility." — Uncle Ben (and every regex user) Go build robust systems! """)