🧠 Fundamentals

🎯 What is Elasticsearch?

Elasticsearch is a distributed, RESTful search and analytics engine built on Apache Lucene. It stores data as JSON documents, provides near real-time search, and scales horizontally. It's the "E" in the ELK Stack (Elasticsearch, Logstash, Kibana).

Why Elasticsearch Matters

Search: Full-text, near real-time. Analytics: Aggregations. Scale: Petabytes. RESTful: JSON API. Distributed: Sharding, replication. Ecosystem: ELK, observability.

📌 Search Engine📌 Analytics📌 Distributed📌 RESTful
# Elasticsearch & ELK — the big picture # Distributed search and analytics engine. # CONCEPTUAL OVERVIEW print(""" === Elasticsearch === What it is: - Distributed search engine - Analytics engine - Based on Apache Lucene - JSON documents - RESTful API - Near real-time Key features: - Full-text search - Aggregations - Sharding - Replication - REST API - Schema-free What it's good for: - Log analytics - Full-text search - Application search - Observability - Security analytics - Business analytics What it's NOT: - Not a primary database - Not for transactions - Not for complex joins - Not for heavy writes """) # SIMPLE EXAMPLE print("=== Simple Example ===") print(""" # Index a document curl -X POST "localhost:9200/users/_doc" -H 'Content-Type: application/json' -d' { "name": "Alice", "age": 30, "city": "NYC", "tags": ["python", "spark"] }' # Search curl -X GET "localhost:9200/users/_search" -H 'Content-Type: application/json' -d' { "query": { "match": { "city": "NYC" } } }' # Aggregate curl -X GET "localhost:9200/users/_search" -H 'Content-Type: application/json' -d' { "size": 0, "aggs": { "by_city": { "terms": { "field": "city.keyword" } } } }' """) # KEY PROPERTIES print(""" === Key Properties === 1. DISTRIBUTED - Sharding - Replication - Horizontal scale 2. NEAR REAL-TIME - Index → searchable in ~1s - Refresh interval 3. RESTFUL - JSON over HTTP - Simple API - Any language 4. SCHEMA-FREE - Dynamic mapping - Flexible - But mappings recommended 5. FULL-TEXT SEARCH - Inverted index - Analyzers - Relevance scoring 6. AGGREGATIONS - Analytics - Buckets - Metrics 7. ECOSYSTEM - Logstash (ingest) - Kibana (visualize) - Beats (ship) """) # REAL-WORLD print("=== Real-World Scale ===") print(""" Elasticsearch: - Used by thousands of companies - Petabytes of data - Billions of documents Examples: - Netflix (logs) - Uber (logs, search) - eBay (search) - Wikipedia (search) - GitHub (search) - Slack (search) """)

🏗️ Architecture

🎯 Distributed Architecture

Elasticsearch is a distributed system with nodes forming a cluster. Data is stored in indices split into shards with replicas for fault tolerance.

Architecture

Node: Server. Cluster: Nodes. Index: Data store. Shard: Partition. Replica: Copy. Master: Cluster state.

🏗️ Node🏗️ Cluster🏗️ Index🏗️ Shard
# Elasticsearch Architecture print("=== Architecture Diagram ===") print(""" ┌─────────────────────────────────────────┐ │ Client │ │ (REST API, Kibana) │ └─────────────────┬───────────────────────┘ │ ▼ ┌─────────────────────────────────────────┐ │ Elasticsearch Cluster │ │ │ │ ┌─────────────┐ ┌─────────────┐ │ │ │ Node 1 │ │ Node 2 │ │ │ │ (Master) │ │ (Data) │ │ │ │ │ │ │ │ │ │ Shard 0(P) │ │ Shard 0(R) │ │ │ │ Shard 1(R) │ │ Shard 1(P) │ │ │ └─────────────┘ └─────────────┘ │ │ │ │ ┌─────────────┐ ┌─────────────┐ │ │ │ Node 3 │ │ Node 4 │ │ │ │ (Data) │ │ (Data) │ │ │ │ │ │ │ │ │ │ Shard 2(P) │ │ Shard 2(R) │ │ │ │ Shard 3(P) │ │ Shard 3(R) │ │ │ └─────────────┘ └─────────────┘ │ └─────────────────────────────────────────┘ """) print("\n=== Components ===") print(""" 1. NODE - Single server - Types: master, data, ingest, coordinating - Multiple roles possible - Join cluster 2. CLUSTER - Group of nodes - Shared state - Named (default: elasticsearch) 3. INDEX - Collection of documents - Like a database table - Multiple shards 4. SHARD - Partition of index - Primary or replica - Distributed 5. REPLICA - Copy of shard - For HA - Serves reads 6. MASTER NODE - Cluster state - Index creation - Shard allocation """) print("\n=== Node Types ===") print(""" Master-eligible: - Can be elected master - Cluster state Data: - Store data - Execute queries Ingest: - Pre-process documents - Ingest pipelines Coordinating: - Route requests - Merge results - All nodes by default """) print("\n=== Cluster Health ===") print(""" GET /_cluster/health Status: - green: All shards healthy - yellow: Replicas unassigned - red: Primary shards unassigned Response: { "cluster_name": "elasticsearch", "status": "green", "number_of_nodes": 3, "active_primary_shards": 5, "active_shards": 10, "unassigned_shards": 0 } """) # IMPLEMENTATION print("\n=== Simulating Elasticsearch ===") class ESNode: """Simulate Elasticsearch node.""" def __init__(self, node_id, node_type='data'): self.id = node_id self.type = node_type self.shards = {} # shard_id -> data def store_shard(self, index, shard_id, data): """Store shard.""" key = f"{index}_{shard_id}" self.shards[key] = data def get_stats(self): return { 'id': self.id, 'type': self.type, 'shards': len(self.shards), } class ESCluster: """Simulate Elasticsearch cluster.""" def __init__(self, name='elasticsearch'): self.name = name self.nodes = {} self.indices = {} self.master = None def add_node(self, node, is_master=False): """Add node.""" self.nodes[node.id] = node if is_master: self.master = node.id print(f"Cluster: Added node {node.id} ({node.type})") def create_index(self, name, shards=1, replicas=1): """Create index.""" self.indices[name] = { 'name': name, 'shards': shards, 'replicas': replicas, 'documents': [], } print(f"Cluster: Created index '{name}' ({shards} shards, {replicas} replicas)") def index_document(self, index, document): """Index a document.""" if index not in self.indices: return None self.indices[index]['documents'].append(document) # Route to shard (simple hash) shard_id = hash(str(document.get('id'))) % self.indices[index]['shards'] return shard_id def search(self, index, query): """Search documents.""" docs = self.indices.get(index, {}).get('documents', []) return [d for d in docs if query(d)] def get_health(self): """Get cluster health.""" return { 'cluster_name': self.name, 'status': 'green', 'number_of_nodes': len(self.nodes), 'number_of_indices': len(self.indices), } # Demo print("=== Elasticsearch Cluster Demo ===") cluster = ESCluster('my-cluster') # Add nodes cluster.add_node(ESNode('node1', 'master'), is_master=True) cluster.add_node(ESNode('node2', 'data')) cluster.add_node(ESNode('node3', 'data')) # Create index cluster.create_index('users', shards=3, replicas=1) # Index documents print("\nIndexing documents:") for i in range(10): doc = {'id': i, 'name': f'user_{i}', 'age': 20 + i, 'city': ['NYC', 'LA'][i % 2]} shard = cluster.index_document('users', doc) print(f" user_{i} → shard {shard}") # Search print("\nSearching (age > 25):") results = cluster.search('users', lambda d: d['age'] > 25) for r in results[:3]: print(f" {r}") # Health print(f"\nCluster health: {cluster.get_health()}") # ARCHITECTURE SUMMARY print("\n=== Architecture Summary ===") print(f"{'Component':<25} {'Role':<40}") print("-" * 65) print(f"{'Node':<25} {'Server':<40}") print(f"{'Cluster':<25} {'Group of nodes':<40}") print(f"{'Index':<25} {'Data store':<40}") print(f"{'Shard':<25} {'Partition':<40}") print(f"{'Replica':<25} {'Copy':<40}") print(f"{'Master':<25} {'Cluster state':<40}")

📄 Documents & Indices

🎯 JSON Documents

Elasticsearch stores data as JSON documents in indices. Documents are schema-free by default, but mappings define field types and analyzers.

Documents & Indices

Document: JSON. Index: Collection. _id: Unique ID. Field: Key. Dynamic: Auto-mapping. Explicit: Defined mapping.

📄 Document📄 Index📄 _id📄 Mapping
# Documents & Indices print("=== Document Model ===") print(""" Document: - JSON - Fields - Nested - Arrays - _id (unique) Example: { "_index": "users", "_id": "1", "_source": { "name": "Alice", "age": 30, "address": { "city": "NYC", "zip": "10001" }, "tags": ["python", "spark"] } } Index: - Collection of documents - Like a database table - Multiple shards """) print("\n=== Operations ===") print(""" # Create index PUT /users # Index document (auto ID) POST /users/_doc { "name": "Alice", "age": 30 } # Index document (specific ID) PUT /users/_doc/1 { "name": "Alice", "age": 30 } # Get document GET /users/_doc/1 # Update document POST /users/_update/1 { "doc": { "age": 31 } } # Delete document DELETE /users/_doc/1 # Bulk operations POST /_bulk { "index": { "_index": "users", "_id": "1" } } { "name": "Alice", "age": 30 } { "index": { "_index": "users", "_id": "2" } } { "name": "Bob", "age": 25 } """) print("\n=== Index Management ===") print(""" # Create index with settings PUT /users { "settings": { "number_of_shards": 3, "number_of_replicas": 1 }, "mappings": { "properties": { "name": { "type": "text" }, "age": { "type": "integer" }, "city": { "type": "keyword" }, "created_at": { "type": "date" } } } } # Get index GET /users # Delete index DELETE /users # List indices GET /_cat/indices?v """) print("\n=== Bulk Operations ===") print(""" Bulk: - Batch operations - High throughput - Single request Actions: - index: Create/replace - create: Create only - update: Update - delete: Delete Example: POST /_bulk {"index": {"_index": "users", "_id": "1"}} {"name": "Alice", "age": 30} {"delete": {"_index": "users", "_id": "2"}} {"update": {"_index": "users", "_id": "3"}} {"doc": {"age": 31}} """) # DOCUMENT SUMMARY print("\n=== Document Summary ===") print(f"{'Concept':<25} {'Description':<40}") print("-" * 65) print(f"{'Document':<25} {'JSON':<40}") print(f"{'Index':<25} {'Collection':<40}") print(f"{'_id':<25} {'Unique ID':<40}") print(f"{'Mapping':<25} {'Field types':<40}") print(f"{'Dynamic':<25} {'Auto-mapping':<40}") print(f"{'Bulk':<25} {'Batch operations':<40}")

📐 Mappings & Analyzers

🎯 Defining Data Types

Mappings define field types and how they're indexed. Analyzers control how text is tokenized and normalized for search.

Mappings & Analyzers

Mapping: Field types. Text: Analyzed. Keyword: Exact. Analyzer: Tokenizer + filters. Standard: Default. Custom: User-defined.

📐 Mapping📐 Text📐 Keyword📐 Analyzer
# Mappings & Analyzers print("=== Field Types ===") print(""" Common types: - text: Analyzed, full-text - keyword: Exact, aggregations - integer, long, float, double - boolean - date - object: Nested - nested: Array of objects - geo_point: Lat/lon Example: { "mappings": { "properties": { "name": { "type": "text" }, "city": { "type": "keyword" }, "age": { "type": "integer" }, "created_at": { "type": "date" } } } } """) print("\n=== Text vs Keyword ===") print(f"{'Aspect':<25} {'text':<25} {'keyword':<25}") print("-" * 75) print(f"{'Analyzed':<25} {'Yes':<25} {'No':<25}") print(f"{'Full-text':<25} {'Yes':<25} {'No':<25}") print(f"{'Aggregations':<25} {'No':<25} {'Yes':<25}") print(f"{'Sorting':<25} {'No':<25} {'Yes':<25}") print(f"{'Use case':<25} {'Search':<25} {'Filters, aggs':<25}") print("\n=== Analyzers ===") print(""" Analyzer: - Character filters - Tokenizer - Token filters Standard analyzer: - "The Quick Brown Fox" - → [the, quick, brown, fox] Custom analyzer: { "settings": { "analysis": { "analyzer": { "my_analyzer": { "type": "custom", "tokenizer": "standard", "filter": ["lowercase", "asciifolding"] } } } } } """) print("\n=== Built-in Analyzers ===") print(""" - standard: Default - simple: Letters only - whitespace: Split on whitespace - stop: Remove stop words - keyword: No tokenization - pattern: Regex - language: Language-specific """) print("\n=== Dynamic Mapping ===") print(""" Dynamic mapping: - Auto-detect types - No mapping needed - But explicit is better Example: POST /users/_doc { "name": "Alice", "age": 30 } → Mapping auto-created Control: "dynamic": true // Auto "dynamic": false // Ignore new "dynamic": "strict" // Reject """) print("\n=== Mapping Best Practices ===") print(""" 1. Define mappings explicitly 2. Use keyword for filters 3. Use text for search 4. Disable dynamic for strict 5. Use multi-fields for both 6. Choose analyzers carefully 7. Consider norms (disable for analytics) """) # MAPPING SUMMARY print("\n=== Mapping Summary ===") print(f"{'Type':<25} {'Use Case':<40}") print("-" * 65) print(f"{'text':<25} {'Full-text search':<40}") print(f"{'keyword':<25} {'Filters, aggregations':<40}") print(f"{'integer':<25} {'Numbers':<40}") print(f"{'date':<25} {'Dates':<40}") print(f"{'nested':<25} {'Array of objects':<40}") print(f"{'geo_point':<25} {'Geospatial':<40}")

🔍 Query DSL

🎯 Searching Elasticsearch

The Query DSL is Elasticsearch's JSON-based query language. It supports many query types: match, term, range, bool, and more.

Query DSL

Match: Full-text. Term: Exact. Range: Numeric/date. Bool: Combine. Compound: Complex. Relevance: Scoring.

🔍 match🔍 term🔍 range🔍 bool
# Query DSL print("=== Query Types ===") print(""" 1. MATCH - Full-text search - Analyzed { "query": { "match": { "name": "alice" } } } 2. TERM - Exact match - Not analyzed { "query": { "term": { "city": "NYC" } } } 3. RANGE - Numeric/date ranges { "query": { "range": { "age": { "gte": 25, "lte": 35 } } } } 4. BOOL - Combine queries - must, should, must_not, filter { "query": { "bool": { "must": [{ "match": { "name": "alice" } }], "filter": [{ "term": { "city": "NYC" } }] }}} 5. WILDCARD - Pattern matching { "query": { "wildcard": { "name": "ali*" } } } 6. FUZZY - Typo tolerance { "query": { "fuzzy": { "name": "alise" } } } """) print("\n=== Bool Query ===") print(""" Bool: - must: AND (scored) - should: OR (scored) - must_not: NOT - filter: AND (no score) Example: { "query": { "bool": { "must": [ { "match": { "title": "elasticsearch" } } ], "filter": [ { "term": { "status": "published" } }, { "range": { "date": { "gte": "2024-01-01" } } } ], "must_not": [ { "term": { "author": "spam" } } ], "should": [ { "match": { "tags": "search" } } ] } } } """) print("\n=== Full Example ===") print(""" GET /users/_search { "query": { "bool": { "must": [ { "match": { "name": "alice" } } ], "filter": [ { "range": { "age": { "gte": 25 } } } ] } }, "sort": [ { "age": "desc" } ], "from": 0, "size": 10, "_source": ["name", "age", "city"] } """) print("\n=== Search Features ===") print(""" - Pagination: from, size - Sorting: sort - Highlighting: highlight - Source filtering: _source - Script fields: script_fields - Suggesters: suggest - Multi-search: _msearch - Scroll: for large results """) # IMPLEMENTATION print("\n=== Simulating Query DSL ===") class QueryDSL: """Simulate Elasticsearch Query DSL.""" def __init__(self, documents): self.documents = documents def match(self, field, value): """Match query (simplified).""" return [d for d in self.documents if value.lower() in str(d.get(field, '')).lower()] def term(self, field, value): """Term query (exact).""" return [d for d in self.documents if d.get(field) == value] def range(self, field, gte=None, lte=None): """Range query.""" results = [] for d in self.documents: val = d.get(field) if val is None: continue if gte is not None and val < gte: continue if lte is not None and val > lte: continue results.append(d) return results def bool_query(self, must=None, filter=None, must_not=None, should=None): """Bool query.""" results = self.documents.copy() if must: for q in must: results = [d for d in results if d in q] if filter: for q in filter: results = [d for d in results if d in q] if must_not: for q in must_not: results = [d for d in results if d not in q] return results # Demo print("=== Query DSL Demo ===") # Sample documents users = [ {'id': 1, 'name': 'Alice', 'age': 30, 'city': 'NYC'}, {'id': 2, 'name': 'Bob', 'age': 25, 'city': 'LA'}, {'id': 3, 'name': 'Charlie', 'age': 35, 'city': 'NYC'}, {'id': 4, 'name': 'Diana', 'age': 28, 'city': 'LA'}, {'id': 5, 'name': 'Eve', 'age': 32, 'city': 'Chicago'}, ] dsl = QueryDSL(users) # Match print("1. Match (name contains 'a'):") results = dsl.match('name', 'a') for r in results: print(f" {r['name']}") # Term print("\n2. Term (city = 'NYC'):") results = dsl.term('city', 'NYC') for r in results: print(f" {r['name']}") # Range print("\n3. Range (age 28-35):") results = dsl.range('age', gte=28, lte=35) for r in results: print(f" {r['name']} ({r['age']})") # Bool print("\n4. Bool (city=NYC AND age>30):") must = dsl.term('city', 'NYC') filter_q = dsl.range('age', gte=30) results = dsl.bool_query(must=must, filter=filter_q) for r in results: print(f" {r['name']} ({r['age']})") # QUERY SUMMARY print("\n=== Query Summary ===") print(f"{'Query':<25} {'Description':<40}") print("-" * 65) print(f"{'match':<25} {'Full-text search':<40}") print(f"{'term':<25} {'Exact match':<40}") print(f"{'range':<25} {'Range query':<40}") print(f"{'bool':<25} {'Combine queries':<40}") print(f"{'wildcard':<25} {'Pattern matching':<40}") print(f"{'fuzzy':<25} {'Typo tolerance':<40}")

📊 Aggregations

🎯 Analytics in Elasticsearch

Aggregations are Elasticsearch's analytics engine. They compute metrics, build buckets, and enable complex analytics on your data.

Aggregations

Metrics: Compute values. Buckets: Group data. Pipeline: Chain aggs. Terms: Group by field. Date histogram: Time buckets. Nested: Combine.

📊 Metrics📊 Buckets📊 Pipeline📊 Terms
# Aggregations print("=== What are Aggregations? ===") print(""" Aggregations: - Analytics - Metrics - Buckets - Pipeline Types: - Metrics: Compute values - Buckets: Group documents - Pipeline: Chain aggregations """) print("\n=== Metric Aggregations ===") print(""" Metrics: - avg, sum, min, max - count, cardinality - stats, extended_stats - percentiles Example: { "size": 0, "aggs": { "avg_age": { "avg": { "field": "age" } }, "max_age": { "max": { "field": "age" } }, "count": { "value_count": { "field": "age" } } } } """) print("\n=== Bucket Aggregations ===") print(""" Buckets: - terms: Group by field - date_histogram: Time - range: Numeric ranges - histogram: Numeric - filters: Multiple filters Example: { "size": 0, "aggs": { "by_city": { "terms": { "field": "city.keyword" }, "aggs": { "avg_age": { "avg": { "field": "age" } } } } } } """) print("\n=== Full Example ===") print(""" GET /users/_search { "size": 0, "aggs": { "by_city": { "terms": { "field": "city.keyword", "size": 10 }, "aggs": { "avg_age": { "avg": { "field": "age" } }, "age_ranges": { "range": { "field": "age", "ranges": [ { "to": 30 }, { "from": 30, "to": 40 }, { "from": 40 } ] } } } } } } """) print("\n=== Date Histogram ===") print(""" { "size": 0, "aggs": { "events_per_day": { "date_histogram": { "field": "timestamp", "calendar_interval": "day" } } } } """) print("\n=== Pipeline Aggregations ===") print(""" Pipeline: - Chain aggregations - Derivative - Cumulative sum - Moving average Example: { "aggs": { "sales_per_month": { "date_histogram": { "field": "date", "calendar_interval": "month" }, "aggs": { "sales": { "sum": { "field": "amount" } }, "sales_deriv": { "derivative": { "buckets_path": "sales" } } } } } } """) print("\n=== Aggregation Use Cases ===") print(""" - Dashboards - Reports - KPIs - Trends - Distributions - Correlations - Real-time analytics """) # IMPLEMENTATION print("\n=== Simulating Aggregations ===") class Aggregations: """Simulate Elasticsearch aggregations.""" def __init__(self, documents): self.documents = documents def avg(self, field): """Average.""" values = [d[field] for d in self.documents if field in d] return sum(values) / len(values) if values else 0 def sum(self, field): """Sum.""" return sum(d[field] for d in self.documents if field in d) def max(self, field): """Max.""" return max(d[field] for d in self.documents if field in d) def min(self, field): """Min.""" return min(d[field] for d in self.documents if field in d) def terms(self, field): """Terms aggregation.""" counts = {} for d in self.documents: val = d.get(field) counts[val] = counts.get(val, 0) + 1 return sorted(counts.items(), key=lambda x: -x[1]) def date_histogram(self, field, interval='day'): """Date histogram.""" counts = {} for d in self.documents: date = d.get(field) if date: key = date.split('T')[0] counts[key] = counts.get(key, 0) + 1 return sorted(counts.items()) # Demo print("=== Aggregation Demo ===") # Sample data users = [ {'name': 'Alice', 'age': 30, 'city': 'NYC', 'date': '2024-01-01'}, {'name': 'Bob', 'age': 25, 'city': 'LA', 'date': '2024-01-01'}, {'name': 'Charlie', 'age': 35, 'city': 'NYC', 'date': '2024-01-02'}, {'name': 'Diana', 'age': 28, 'city': 'LA', 'date': '2024-01-02'}, {'name': 'Eve', 'age': 32, 'city': 'Chicago', 'date': '2024-01-03'}, ] aggs = Aggregations(users) print("1. Metrics:") print(f" avg age: {aggs.avg('age')}") print(f" sum age: {aggs.sum('age')}") print(f" max age: {aggs.max('age')}") print(f" min age: {aggs.min('age')}") print("\n2. Terms (by city):") for city, count in aggs.terms('city'): print(f" {city}: {count}") print("\n3. Date histogram (by day):") for date, count in aggs.date_histogram('date'): print(f" {date}: {count}") # AGGREGATION SUMMARY print("\n=== Aggregation Summary ===") print(f"{'Type':<25} {'Description':<40}") print("-" * 65) print(f"{'avg, sum':<25} {'Metrics':<40}") print(f"{'max, min':<25} {'Metrics':<40}") print(f"{'terms':<25} {'Group by field':<40}") print(f"{'date_histogram':<25} {'Time buckets':<40}") print(f"{'range':<25} {'Numeric ranges':<40}") print(f"{'pipeline':<25} {'Chain aggs':<40}")

🔀 Sharding & Replication

🎯 Scaling Elasticsearch

Sharding splits an index into partitions for horizontal scale. Replication copies shards for fault tolerance and read capacity.

Sharding & Replication

Primary shard: Data. Replica shard: Copy. Shard count: Fixed at creation. Replica count: Configurable. Allocation: Distributed. Rebalancing: Automatic.

🔀 Primary🔀 Replica🔀 Allocation🔀 Rebalancing
# Sharding & Replication print("=== What is Sharding? ===") print(""" Sharding: - Split index into partitions - Distribute across nodes - Horizontal scale Primary shard: - Original data - Fixed at creation - Cannot change Replica shard: - Copy of primary - Configurable - For HA and reads """) print("\n=== Shard Allocation ===") print(""" Example: Index: users Shards: 3 Replicas: 1 Node 1: Shard 0 (P), Shard 1 (R) Node 2: Shard 1 (P), Shard 2 (R) Node 3: Shard 2 (P), Shard 0 (R) Total shards: 3 primary + 3 replica = 6 """) print("\n=== Shard Configuration ===") print(""" # Create index PUT /users { "settings": { "number_of_shards": 3, "number_of_replicas": 1 } } # Cannot change primary shards after creation # Can change replicas PUT /users/_settings { "number_of_replicas": 2 } """) print("\n=== Choosing Shard Count ===") print(""" Rule of thumb: - Shard size: 10-50 GB - Shards per node: 20-25 - Total shards: nodes * 20 Example: 3 nodes 30 shards (10 per node) 1 replica Over-sharding: - Too many shards - Overhead - Slow Under-sharding: - Too few shards - Cannot scale """) print("\n=== Routing ===") print(""" Routing: - Document → shard - Default: hash(_id) - Custom routing Formula: shard = hash(routing) % num_primary_shards Custom routing: PUT /users/_doc/1?routing=user_1 { ... } """) print("\n=== Cluster Health ===") print(""" GET /_cluster/health Status: - green: All shards healthy - yellow: Replicas unassigned - red: Primary shards unassigned GET /_cat/shards?v GET /_cat/nodes?v """) # IMPLEMENTATION print("\n=== Simulating Sharding ===") class ShardedIndex: """Simulate sharded Elasticsearch index.""" def __init__(self, name, num_shards=3, num_replicas=1): self.name = name self.num_shards = num_shards self.num_replicas = num_replicas self.primary_shards = [[] for _ in range(num_shards)] self.replica_shards = [[] for _ in range(num_shards)] def route(self, doc_id): """Route document to shard.""" return hash(str(doc_id)) % self.num_shards def index(self, doc_id, document): """Index document.""" shard_id = self.route(doc_id) self.primary_shards[shard_id].append(document) # Replicate self.replica_shards[shard_id].append(document) return shard_id def search(self, query): """Search across shards.""" results = [] for shard in self.primary_shards: results.extend([d for d in shard if query(d)]) return results def get_stats(self): return { 'name': self.name, 'shards': self.num_shards, 'replicas': self.num_replicas, 'primary_docs': [len(s) for s in self.primary_shards], 'replica_docs': [len(s) for s in self.replica_shards], } # Demo print("=== Sharding Demo ===") index = ShardedIndex('users', num_shards=3, num_replicas=1) print("Indexing documents:") for i in range(10): doc = {'id': i, 'name': f'user_{i}', 'age': 20 + i} shard = index.index(i, doc) print(f" user_{i} → shard {shard}") print(f"\nIndex stats: {index.get_stats()}") # Search print("\nSearching (age > 25):") results = index.search(lambda d: d['age'] > 25) for r in results[:3]: print(f" {r}") # SHARDING SUMMARY print("\n=== Sharding Summary ===") print(f"{'Concept':<25} {'Description':<40}") print("-" * 65) print(f"{'Primary shard':<25} {'Original data':<40}") print(f"{'Replica shard':<25} {'Copy':<40}") print(f"{'Shard count':<25} {'Fixed at creation':<40}") print(f"{'Replica count':<25} {'Configurable':<40}") print(f"{'Routing':<25} {'Document → shard':<40}") print(f"{'Rebalancing':<25} {'Automatic':<40}")

📦 ELK Stack

🎯 The Complete Stack

The ELK Stack combines Elasticsearch (search), Logstash (ingest), and Kibana (visualize). It's the standard for log analytics and observability.

ELK Stack

Elasticsearch: Search & store. Logstash: Ingest & transform. Kibana: Visualize. Beats: Ship data. Pipeline: Beats → Logstash → ES → Kibana.

📦 Elasticsearch📦 Logstash📦 Kibana📦 Beats
# ELK Stack print("=== What is ELK? ===") print(""" ELK: - Elasticsearch - Logstash - Kibana Pipeline: Data → Beats → Logstash → Elasticsearch → Kibana Components: - Beats: Ship data - Logstash: Process data - Elasticsearch: Store and search - Kibana: Visualize """) print("\n=== Data Flow ===") print(""" 1. BEATS - Collect data - Ship to Logstash/ES - Lightweight 2. LOGSTASH - Receive data - Filter, transform - Send to ES 3. ELASTICSEARCH - Index data - Store - Search 4. KIBANA - Query ES - Visualize - Dashboards """) print("\n=== Use Cases ===") print(""" 1. Log Analytics - Application logs - Server logs - Security logs 2. Observability - Metrics - Traces - Logs 3. Search - Application search - Site search - Document search 4. Security - SIEM - Threat detection - Compliance 5. Business Analytics - KPIs - Dashboards - Reports """) print("\n=== Alternative Names ===") print(""" ELK: - Elasticsearch - Logstash - Kibana Elastic Stack: - ELK + Beats - Official name OpenSearch: - AWS fork (2021) - Apache 2.0 - Compatible """) print("\n=== Example Pipeline ===") print(""" # Filebeat configuration filebeat.inputs: - type: log paths: - /var/log/*.log output.logstash: hosts: ["localhost:5044"] # Logstash configuration input { beats { port => 5044 } } filter { grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } } output { elasticsearch { hosts => ["localhost:9200"] index => "logs-%{+YYYY.MM.dd}" } } # Kibana # - Create index pattern # - Build visualizations # - Create dashboards """) # ELK SUMMARY print("\n=== ELK Summary ===") print(f"{'Component':<25} {'Role':<40}") print("-" * 65) print(f"{'Elasticsearch':<25} {'Search & store':<40}") print(f"{'Logstash':<25} {'Ingest & transform':<40}") print(f"{'Kibana':<25} {'Visualize':<40}") print(f"{'Beats':<25} {'Ship data':<40}")

🔷 Logstash

🎯 Data Processing Pipeline

Logstash is a server-side data processing pipeline that ingests data from multiple sources, transforms it, and sends it to Elasticsearch.

Logstash

Input: Sources. Filter: Transform. Output: Destinations. Plugins: Many. Pipelines: Multiple. Grok: Parse.

🔷 Input🔷 Filter🔷 Output🔷 Grok
# Logstash print("=== What is Logstash? ===") print(""" Logstash: - Data processing pipeline - Input, filter, output - 200+ plugins - Multiple pipelines - JVM-based Pipeline: Input → Filter → Output """) print("\n=== Input Plugins ===") print(""" - beats: Receive from Beats - file: Read files - kafka: Kafka topics - jdbc: Databases - http: HTTP requests - tcp/udp: Network - s3: S3 buckets - redis: Redis """) print("\n=== Filter Plugins ===") print(""" - grok: Parse unstructured - mutate: Modify fields - date: Parse dates - json: Parse JSON - geoip: Add location - useragent: Parse UA - csv: Parse CSV - ruby: Custom code """) print("\n=== Output Plugins ===") print(""" - elasticsearch: Send to ES - kafka: Kafka topics - file: Write files - s3: S3 buckets - stdout: Console - http: HTTP - mongodb: MongoDB """) print("\n=== Example Configuration ===") print(""" input { beats { port => 5044 } kafka { bootstrap_servers => "localhost:9092" topics => ["logs"] } } filter { # Parse Apache logs grok { match => { "message" => "%{COMBINEDAPACHELOG}" } } # Parse timestamp date { match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"] } # Add geo data geoip { source => "clientip" } # Remove fields mutate { remove_field => ["message", "@version"] } } output { elasticsearch { hosts => ["localhost:9200"] index => "logs-%{+YYYY.MM.dd}" } stdout { codec => rubydebug } } """) print("\n=== Grok Patterns ===") print(""" Common patterns: - %{WORD} - %{NUMBER} - %{IP} - %{TIMESTAMP_ISO8601} - %{COMBINEDAPACHELOG} - %{SYSLOGTIMESTAMP} Custom: %{WORD:username} %{NUMBER:status:int} """) # LOGSTASH SUMMARY print("\n=== Logstash Summary ===") print(f"{'Section':<25} {'Description':<40}") print("-" * 65) print(f"{'Input':<25} {'Data sources':<40}") print(f"{'Filter':<25} {'Transform data':<40}") print(f"{'Output':<25} {'Destinations':<40}") print(f"{'Grok':<25} {'Parse unstructured':<40}") print(f"{'Plugins':<25} {'200+ available':<40}")

📊 Kibana

🎯 Visualization Platform

Kibana is the visualization and exploration UI for Elasticsearch. It provides dashboards, charts, maps, and tools for analytics.

Kibana

Discover: Explore data. Visualize: Charts. Dashboard: Combine. Dev Tools: Query. Maps: Geo. Canvas: Reports.

📊 Discover📊 Visualize📊 Dashboard📊 Dev Tools
# Kibana print("=== What is Kibana? ===") print(""" Kibana: - Visualization UI - Explore data - Create charts - Build dashboards - Query Elasticsearch Features: - Discover - Visualize - Dashboard - Dev Tools - Maps - Canvas - Machine Learning """) print("\n=== Discover ===") print(""" Discover: - Explore raw documents - Search - Filter - Time range - Fields Use: - Debug data - Explore logs - Find patterns """) print("\n=== Visualize ===") print(""" Visualizations: - Line chart - Bar chart - Pie chart - Data table - Metric - Gauge - Heat map - Tag cloud - Maps Based on: - Aggregations - Queries """) print("\n=== Dashboard ===") print(""" Dashboard: - Combine visualizations - Interactive - Filters - Auto-refresh - Share Use: - Monitoring - Analytics - Reporting """) print("\n=== Dev Tools ===") print(""" Dev Tools: - Console - Query ES - Debug - Test Example: GET /users/_search { "query": { "match_all": {} } } """) print("\n=== Kibana Examples ===") print(""" 1. Log Dashboard - Log volume over time - Top errors - Top sources 2. Metrics Dashboard - CPU, memory - Response times - Error rates 3. Business Dashboard - Sales - Users - Revenue 4. Security Dashboard - Threats - Alerts - Anomalies """) # KIBANA SUMMARY print("\n=== Kibana Summary ===") print(f"{'Feature':<25} {'Description':<40}") print("-" * 65) print(f"{'Discover':<25} {'Explore data':<40}") print(f"{'Visualize':<25} {'Charts':<40}") print(f"{'Dashboard':<25} {'Combine':<40}") print(f"{'Dev Tools':<25} {'Query':<40}") print(f"{'Maps':<25} {'Geo':<40}") print(f"{'Canvas':<25} {'Reports':<40}")

📡 Beats

🎯 Lightweight Data Shippers

Beats are lightweight agents that ship data from edge machines to Logstash or Elasticsearch. Each Beat is specialized for a data type.

Beats

Filebeat: Logs. Metricbeat: Metrics. Packetbeat: Network. Winlogbeat: Windows. Auditbeat: Audit. Heartbeat: Uptime.

📡 Filebeat📡 Metricbeat📡 Packetbeat📡 Heartbeat
# Beats print("=== What are Beats? ===") print(""" Beats: - Lightweight shippers - Single-purpose - Edge agents - Ship to Logstash/ES Common Beats: - Filebeat: Logs - Metricbeat: Metrics - Packetbeat: Network - Winlogbeat: Windows events - Auditbeat: Audit - Heartbeat: Uptime - Functionbeat: Serverless """) print("\n=== Filebeat ===") print(""" Filebeat: - Ship log files - Lightweight - Backpressure handling - Multiple inputs Configuration: filebeat.inputs: - type: log paths: - /var/log/*.log fields: service: myapp output.elasticsearch: hosts: ["localhost:9200"] """) print("\n=== Metricbeat ===") print(""" Metricbeat: - Ship metrics - System, Docker, Kubernetes - Modules Configuration: metricbeat.modules: - module: system metricsets: - cpu - memory - network period: 10s output.elasticsearch: hosts: ["localhost:9200"] """) print("\n=== Packetbeat ===") print(""" Packetbeat: - Network packets - Protocols: HTTP, DNS, MySQL - Real-time Configuration: packetbeat.interfaces.device: eth0 packetbeat.protocols: - type: http ports: [80, 8080] - type: dns ports: [53] """) print("\n=== Beats vs Logstash ===") print(f"{'Aspect':<25} {'Beats':<25} {'Logstash':<25}") print("-" * 75) print(f"{'Purpose':<25} {'Ship':<25} {'Process':<25}") print(f"{'Resources':<25} {'Light':<25} {'Heavy':<25}") print(f"{'Language':<25} {'Go':<25} {'JVM':<25}") print(f"{'Transforms':<25} {'Limited':<25} {'Extensive':<25}") print(f"{'Deployment':<25} {'Edge':<25} {'Server':<25}") # BEATS SUMMARY print("\n=== Beats Summary ===") print(f"{'Beat':<25} {'Data':<40}") print("-" * 65) print(f"{'Filebeat':<25} {'Logs':<40}") print(f"{'Metricbeat':<25} {'Metrics':<40}") print(f"{'Packetbeat':<25} {'Network':<40}") print(f"{'Winlogbeat':<25} {'Windows events':<40}") print(f"{'Auditbeat':<25} {'Audit':<40}") print(f"{'Heartbeat':<25} {'Uptime':<40}")

🚀 Performance

🎯 Fast Search

Elasticsearch performance depends on sharding, mappings, queries, and hardware. Proper tuning is essential.

Performance

Shards: Right size. Mappings: Right types. Queries: Filter vs query. Refresh: Interval. Cache: Query, request. Hardware: RAM, SSD.

🚀 Shards🚀 Mappings🚀 Queries🚀 Cache
# Performance print("=== Performance Factors ===") print(""" 1. SHARDS - Right size: 10-50 GB - Shards per node: 20-25 - Not too many 2. MAPPINGS - keyword for filters - text for search - Disable norms for analytics 3. QUERIES - Filter context (cache) - Query context (score) - Avoid wildcards - Use filters 4. REFRESH - Default: 1s - Increase for writes - Decrease for search 5. CACHE - Query cache - Request cache - Field data cache 6. HARDWARE - RAM: 64GB+ - SSD: Fast - CPU: Multi-core """) print("\n=== Query Optimization ===") print(""" Filter vs Query: - Filter: No scoring, cached - Query: Scoring, not cached Use filter when: - Exact match - Range - No relevance needed Example: { "query": { "bool": { "must": [ { "match": { "title": "search" } } // Scored ], "filter": [ { "term": { "status": "active" } } // Cached ] } } } """) print("\n=== Refresh Interval ===") print(""" Refresh: - Index → searchable - Default: 1s - Affects performance Tuning: - Bulk indexing: -1 - Then refresh manually - Search: 1s Example: PUT /users/_settings { "index.refresh_interval": "30s" } """) print("\n=== Bulk Indexing ===") print(""" Bulk: - Batch operations - Much faster - 5-15 MB per batch Best practices: - Use bulk API - Disable refresh - Disable replicas - Increase translog Example: PUT /users/_settings { "index.refresh_interval": "-1", "index.number_of_replicas": 0 } # Bulk index # ... PUT /users/_settings { "index.refresh_interval": "1s", "index.number_of_replicas": 1 } """) print("\n=== Monitoring ===") print(""" Metrics: - Cluster health - Node stats - Index stats - Query latency - Indexing rate - JVM heap Tools: - Kibana Monitoring - Elasticsearch APIs - Prometheus - Grafana """) # PERFORMANCE SUMMARY print("\n=== Performance Summary ===") print(f"{'Factor':<25} {'Impact':<40}") print("-" * 65) print(f"{'Shards':<25} {'High':<40}") print(f"{'Mappings':<25} {'High':<40}") print(f"{'Queries':<25} {'High':<40}") print(f"{'Refresh':<25} {'Medium':<40}") print(f"{'Cache':<25} {'Medium':<40}") print(f"{'Hardware':<25} {'High':<40}")

⚖️ vs Alternatives

🎯 Comparing Search Engines

Elasticsearch vs Solr, OpenSearch, and other search/analytics engines.

Comparison

Elasticsearch: Most popular. Solr: Mature. OpenSearch: AWS fork. Meilisearch: Lightweight. Typesense: Simple. Choose: Based on needs.

⚖️ Elasticsearch⚖️ Solr⚖️ OpenSearch⚖️ Meilisearch
# Elasticsearch vs Alternatives print("=== Comparison ===") print(f"{'Aspect':<25} {'Elasticsearch':<20} {'Solr':<20} {'OpenSearch':<20}") print("-" * 85) print(f"{'License':<25} {'SSPL/Elastic':<20} {'Apache':<20} {'Apache':<20}") print(f"{'Language':<25} {'Java':<20} {'Java':<20} {'Java':<20}") print(f"{'Based on':<25} {'Lucene':<20} {'Lucene':<20} {'Lucene':<20}") print(f"{'Search':<25} {'Excellent':<20} {'Excellent':<20} {'Excellent':<20}") print(f"{'Analytics':<25} {'Excellent':<20} {'Good':<20} {'Excellent':<20}") print(f"{'Ecosystem':<25} {'Largest':<20} {'Mature':<20} {'Growing':<20}") print(f"{'Cloud':<25} {'Elastic Cloud':<20} {'Various':<20} {'AWS':<20}") print("\n=== Elasticsearch vs Solr ===") print(f"{'Aspect':<25} {'Elasticsearch':<20} {'Solr':<20}") print("-" * 65) print(f"{'API':<25} {'REST/JSON':<20} {'REST/XML':<20}") print(f"{'Distributed':<25} {'Native':<20} {'SolrCloud':<20}") print(f"{'Real-time':<25} {'Yes':<20} {'Yes':<20}") print(f"{'Analytics':<25} {'Excellent':<20} {'Good':<20}") print(f"{'Community':<25} {'Larger':<20} {'Mature':<20}") print(f"{'Use case':<25} {'Modern':<20} {'Enterprise':<20}") print("\n=== Elasticsearch vs OpenSearch ===") print(f"{'Aspect':<25} {'Elasticsearch':<20} {'OpenSearch':<20}") print("-" * 65) print(f"{'License':<25} {'SSPL/Elastic':<20} {'Apache 2.0':<20}") print(f"{'Origin':<25} {'Elastic':<20} {'AWS fork':<20}") print(f"{'Compatible':<25} {'—':<20} {'Mostly':<20}") print(f"{'Ecosystem':<25} {'Largest':<20} {'Growing':<20}") print(f"{'Cloud':<25} {'Elastic Cloud':<20} {'AWS':<20}") print("\n=== When to Use Which ===") print(""" USE ELASTICSEARCH: ✓ Modern search ✓ Analytics ✓ Observability ✓ Largest ecosystem USE SOLR: ✓ Enterprise ✓ Mature ✓ Custom USE OPENSEARCH: ✓ AWS ✓ Apache license ✓ Compatible USE MEILISEARCH: ✓ Lightweight ✓ Simple ✓ Fast """)

🚀 Future

🎯 Where Elasticsearch Is Going

The future is AI-powered search, vector search, observability, and serverless.

Future Directions

Vector: Semantic search. AI: RAG. Serverless: Elastic Cloud. Observability: Unified. Security: SIEM. Open: Standards.

🚀 Vector🚀 AI🚀 Serverless🚀 Observability
# Future of Elasticsearch print("=== 1. Vector Search ===") print(""" Vector Search: - Embeddings - Similarity - Semantic search - RAG Features: - Dense vector - Sparse vector - kNN search - Hybrid search Future: - AI-native - Semantic - RAG pipelines """) print("\n=== 2. AI and RAG ===") print(""" AI: - RAG (Retrieval Augmented Generation) - LLM integration - Semantic search - Recommendations Features: - Elastic Learned Sparse Encoder - ELSER - LangChain - OpenAI integration Future: - AI-powered search - Natural language - Conversational """) print("\n=== 3. Serverless ===") print(""" Serverless: - Elastic Cloud Serverless - Pay per use - Auto-scaling - No infrastructure Benefits: - No ops - Elastic - Cost-effective Future: - Serverless default - Event-driven """) print("\n=== 4. Observability ===") print(""" Observability: - Logs - Metrics - Traces - APM Features: - Elastic Observability - APM - Uptime - Logs Future: - Unified - AIOps - Auto-detect """) print("\n=== 5. Security ===") print(""" Security: - SIEM - Threat detection - Compliance - Endpoint Features: - Elastic Security - SIEM - Endpoint - Cloud security Future: - AI-driven - Automated - Unified """) print("\n=== 6. Open Standards ===") print(""" Open: - OpenSearch - Apache 2.0 - Community Future: - Standardization - Interoperability - Open ecosystem """) # TIMELINE print("\n=== Timeline ===") print(""" Past: 2010: Elasticsearch created 2012: Elastic company 2015: ELK Stack 2018: Elastic Cloud 2021: License change 2021: OpenSearch fork Present: 2024: Vector search 2025: AI-native Future: 2030: AI-powered? 2035: Serverless? 2040: New paradigms? """) print("\n=== Final Thoughts ===") print(""" Elasticsearch is: - Search engine - Analytics engine - Distributed - RESTful - Evolving The future is: - Vector search - AI-powered - Serverless - Observability - Open "Search everything, everywhere." Keep exploring. Keep building. """)