๐ Python Deep Foundations
๐ฏ Python's Design Philosophy & Execution Model
Python (Guido van Rossum, 1991) is a dynamically-typed, interpreted, high-level language with strong emphasis on readability. CPython compiles source to bytecode (.pyc) which runs on a stack-based VM. Memory management: reference counting + generational GC. Key pillars: "Batteries Included" standard library, duck typing, and EAFP (Easier to Ask for Forgiveness than Permission).
Deep Explanation
Python is both compiled and interpreted: Source code โ Bytecode โ PVM (Python Virtual Machine). This hybrid approach provides portability. The Global Interpreter Lock (GIL) ensures thread safety for CPython but limits true parallelism. Python's reference counting deallocates objects instantly when refcount hits zero, while generational garbage collection handles cyclic references.
Why it matters: Understanding this model helps optimize performance, debug memory leaks, and design efficient multi-threaded applications. The bytecode can be inspected using the dis module.
import sys
import dis
def reveal_internals():
x = 42
print(f"ID: {id(x)}, Type: {type(x).__name__}, Refcount: {sys.getrefcount(x)-1}")
print("Bytecode:")
dis.dis(reveal_internals)
reveal_internals()
print(f"Python Implementation: {sys.implementation.name}")๐ฆ Variables, Types & Memory Model
๐ฏ Variables as References
Variables are references to heap objects. Immutable types (int, str, tuple) cannot change; mutable types (list, dict, set) can. Interning for small ints (-5 to 256).
Memory & Reference Behavior
When you assign a = [1,2,3], Python creates a list object in heap memory and a becomes a reference to it. Assigning b = a copies only the reference, not the object. This is why mutable objects can be modified through multiple references.
Immutable objects (integers, strings, tuples) cannot be changed in-place. Operations return new objects. Python interns small integers (-5 to 256) and some strings for memory efficiency.
import sys
# Reference behavior
a = [1, 2, 3]
b = a
b.append(4)
print(a) # [1,2,3,4] - mutation via reference
# Interning
x = 256
y = 256
print(x is y) # True (cached)
# Type introspection
print(type(a), isinstance(a, list), id(a))
print(sys.getsizeof(a)) # Overhead + pointer array๐ข Operators (Bitwise, Walrus, etc)
๐ฏ Operator Categories & Overloading
Arithmetic: + - * / // % ** @. Comparison: == != > < >= <= is is not. Bitwise: & | ^ ~ << >>. Walrus operator := assigns and returns.
Advanced Operator Concepts
The walrus operator (:=) introduced in Python 3.8 allows assignment within expressions, reducing code duplication. Bitwise operators work at the binary level โ essential for flags, permissions, and low-level optimizations. Short-circuit evaluation (and/or) stops as soon as the result is determined.
# Walrus operator
if (n := len([1,2,3,4,5])) > 3:
print(f"Length {n} is large")
# Bitwise operations
flags = 0b1010
masked = flags & 0b1100
print(bin(masked)) # 0b1000
# Short-circuit evaluation
x = None
result = x or "default"
print(result) # default๐ Control Flow Mastery
๐ฏ Conditional Execution & Pattern Matching
if-elif-else, match-case (Python 3.10+), ternary expressions.
Structural Pattern Matching
Match-case goes beyond simple switch statements โ it supports sequence patterns, mapping patterns, class patterns, and guards. This enables elegant destructuring of complex data structures.
# Structural pattern matching
def handle_command(cmd):
match cmd.split():
case ["quit"]: return "Exiting"
case ["hello", name]: return f"Hello {name}"
case ["add", *numbers] if all(n.isdigit() for n in numbers): return sum(map(int, numbers))
case _: return "Unknown"
print(handle_command("add 10 20 30"))๐ Iteration Protocols & Comprehensions
๐ฏ Iterables, Iterators, Generators
List/dict/set comprehensions, generator expressions, itertools module.
Lazy Evaluation & Memory Efficiency
Generator expressions use parentheses (x**2 for x in range(10**6)) and produce values on the fly without storing entire list in memory. The itertools module provides high-performance iterator building blocks like cycle, chain, and combinations.
# Generator expression (lazy)
squares = (x**2 for x in range(10**6))
print(next(squares))
from itertools import cycle, islice
colors = cycle(['red', 'blue'])
print(list(islice(colors, 5)))โ๏ธ Functions: Decorators, Closures, Args
๐ฏ First-Class Functions
*args, **kwargs, closures, decorators, functools.lru_cache.
Decorators & Memoization
Decorators wrap functions to add behavior without modifying source code. Closures capture environment variables even after outer function returns. @lru_cache automatically caches function results based on arguments โ essential for expensive recursive computations like Fibonacci.
from functools import lru_cache
@lru_cache(maxsize=128)
def fibonacci(n):
if n < 2: return n
return fibonacci(n-1) + fibonacci(n-2)
print(fibonacci(100))๐ Advanced Collections
๐ฏ collections Module Power
deque, Counter, defaultdict, namedtuple, ChainMap.
Specialized Container Types
deque provides O(1) append/pop from both ends โ perfect for queues. Counter simplifies frequency counting. defaultdict eliminates key checks. namedtuple creates lightweight immutable data structures.
from collections import deque, Counter, defaultdict, namedtuple
dq = deque(maxlen=5)
dq.extend([1,2,3,4,5,6])
print(dq) # [2,3,4,5,6]
text = "hello world hello"
counts = Counter(text.split())
print(counts.most_common())๐ Iterators, Generators, Coroutines
๐ฏ yield, yield from, async generators
Generators, coroutines, and generator pipelines.
Generator Pipelines & Coroutines
Generators maintain state between yields. yield from delegates to sub-generators, flattening nested generators. Coroutines (using .send()) can receive values, enabling two-way communication โ the foundation for async patterns.
def coroutine_example():
while True:
received = yield
print(f"Received: {received}")
coro = coroutine_example()
next(coro)
coro.send("Hello")๐ Unicode, Regex, String Internals
๐ฏ Unicode & Encoding
Python 3 strings are Unicode. encode()/decode() convert between str and bytes. re module for regex.
Unicode & Regular Expressions
Python 3 uses Unicode (UTF-32 internally) โ all strings support international characters. Regular expressions support Unicode categories (\p{L}). String concatenation with join() is optimal for large operations.
import re
text = "Python ๐ ็ผ็จ"
print(len(text), text.encode('utf-8'))
pattern = re.compile(r'(?P\w+)@(?P\w+)\.\w+')
match = pattern.search('user@example.com')
print(match.groupdict()) ๐พ I/O, Context Managers, Serialization
๐ฏ with statement & protocols
Context managers, pickle, json, pathlib.
Resource Management & Serialization
Context managers (with) guarantee resource cleanup even if exceptions occur. pathlib provides object-oriented filesystem paths. pickle serializes Python objects, while json ensures interoperability with other languages.
from pathlib import Path
p = Path('.')
with open('test.txt', 'w') as f:
f.write('data')
import json
data = {'name': 'Python', 'version': 3.12}
print(json.dumps(data))๐ก๏ธ Exception Hierarchy & Custom Errors
๐ฏ Exception Classes & chaining
BaseException โ Exception โ (ValueError, TypeError, etc). raise ... from for chaining.
Exception Chaining & Best Practices
Exception chaining (raise ... from) preserves the original exception context for debugging. Custom exceptions should inherit from Exception and add domain-specific attributes. finally blocks ensure cleanup code always runs.
class BusinessError(Exception):
def __init__(self, message, code):
super().__init__(message)
self.code = code
try:
raise ValueError("Original")
except ValueError as e:
raise BusinessError("Business violation", 400) from e๐๏ธ OOP Deep Dive (Metaclasses, slots)
๐ฏ Classes as Objects, Metaclasses
Classes are instances of type (metaclass). __slots__ reduces memory. dataclasses, ABC.
Metaclasses & Memory Optimization
Metaclasses (type subclasses) control class creation โ used in frameworks like Django ORM. __slots__ prevents __dict__ creation, saving significant memory for thousands of instances. @dataclass automates boilerplate.
from dataclasses import dataclass
@dataclass(slots=True)
class Employee:
name: str
salary: float = 0.0
e = Employee("Alice", 75000)
print(e)๐ฆ Modules, Packages & Import System
๐ฏ Import Hooks & Namespace Packages
Modules cached in sys.modules. importlib for dynamic imports.
Import Machinery & Dynamic Loading
Python's import system searches sys.path, caches in sys.modules. importlib allows dynamic imports and module reloading โ useful for plugins and hot-reload in development. Namespace packages allow splitting packages across directories.
import importlib
math = importlib.import_module('math')
print(math.sqrt(144))
print(sys.modules.keys())โฉ Async IO, Concurrency, GIL
๐ฏ asyncio, threading, multiprocessing
asyncio for concurrent I/O, threading for I/O-bound, multiprocessing for CPU-bound.
Concurrency Models & GIL Reality
asyncio uses single-threaded cooperative multitasking โ ideal for thousands of network connections. threading works for I/O-bound tasks despite GIL. multiprocessing bypasses GIL via separate processes, true parallelism for CPU-intensive work.
import asyncio
async def fetch_data():
await asyncio.sleep(1)
return "Data"
async def main():
results = await asyncio.gather(fetch_data(), fetch_data())
print(results)
from concurrent.futures import ThreadPoolExecutor
with ThreadPoolExecutor(max_workers=4) as executor:
futures = [executor.submit(sum, range(10**6)) for _ in range(4)]