๐ RNN Core Philosophy
๐ฏ Modeling Sequential Data
Recurrent Neural Networks (RNNs) are designed for sequential data. They maintain a hidden state that captures information from previous time steps, making them ideal for time series, text, speech, and any data where order matters.
Why RNNs?
Memory through recurrence โ RNNs process sequences step by step, updating their internal state at each time step. This allows them to capture temporal dependencies and variable-length sequences. However, vanilla RNNs suffer from vanishing gradients, which LSTMs solve.
from tensorflow.keras.layers import SimpleRNN, LSTM, GRU
from tensorflow.keras.models import Sequential
import numpy as np
# Simple RNN
model = Sequential([
SimpleRNN(64, input_shape=(10, 32)), # 10 timesteps, 32 features
Dense(1, activation='sigmoid')
])
print("RNN for sequence classification")
print(f"Model summary: {model.summary()}")๐งฌ Vanilla RNN Architecture
๐ฏ The Recurrent Unit
hโ = f(Wโโ ยท hโโโ + Wโโ ยท xโ + bโ). The hidden state hโ captures information from the sequence up to time t.
Recurrence in Action
At each step, the RNN combines the current input with the previous hidden state. This creates a chain of dependencies through time, enabling the network to maintain context.
import numpy as np
def simple_rnn_step(x_t, h_prev, Wxh, Whh, b):
h_t = np.tanh(np.dot(Wxh, x_t) + np.dot(Whh, h_prev) + b)
return h_t
# Example: processing a sequence
X = np.random.randn(5, 10) # 5 timesteps, 10 features
h = np.zeros(20) # initial hidden state
Wxh = np.random.randn(20, 10)
Whh = np.random.randn(20, 20)
b = np.zeros(20)
for t in range(len(X)):
h = simple_rnn_step(X[t], h, Wxh, Whh, b)
print(f"Final hidden state shape: {h.shape}")๐ Backprop Through Time (BPTT)
๐ฏ Training RNNs
BPTT unrolls the RNN through time and applies backpropagation to the unrolled network. Gradients flow backward through all time steps.
Unrolling the Network
BPTT is computationally expensive for long sequences. Truncated BPTT processes sequences in chunks to reduce complexity.
# BPTT conceptual in Keras
# Keras handles BPTT automatically
model.compile(optimizer='adam', loss='mse')
# For long sequences, use truncated backprop through time (tbptt)
# RNN layers have a 'unroll' parameter for debugging
rnn_layer = SimpleRNN(64, unroll=True) # unrolls sequence for debugging๐ Vanishing & Exploding Gradients
๐ฏ The Fundamental Challenge
Vanishing gradients: gradients become exponentially small. Exploding gradients: gradients become exponentially large. Both hinder learning long-term dependencies.
Gradient Problems
Vanishing gradients prevent RNNs from learning long-range dependencies (e.g., context from 100 steps ago). LSTM and GRU were designed to solve this with gating mechanisms.
# Gradient clipping in Keras
from tensorflow.keras.optimizers import Adam
optimizer = Adam(learning_rate=0.001, clipnorm=1.0) # clip gradients
model.compile(optimizer=optimizer, loss='mse')
# Gradient clipping prevents exploding gradients๐ง LSTM Architecture
๐ฏ Long Short-Term Memory
LSTM introduces a cell state (Cโ) that acts as a long-term memory, regulated by three gates: input, forget, and output gates.
The Cell State
The cell state runs through the entire sequence, with only minor linear interactions. This allows information to flow unchanged over long distances, solving the vanishing gradient problem.
from tensorflow.keras.layers import LSTM
# LSTM layer
lstm_model = Sequential([
LSTM(64, input_shape=(10, 32), return_sequences=True),
LSTM(32),
Dense(1, activation='sigmoid')
])
# LSTM has 4 times more parameters than SimpleRNN
print(f"LSTM layer parameters: {lstm_model.layers[0].count_params()}")๐ช LSTM Gates
๐ฏ Input, Forget, Output Gates
Forget gate: fโ = ฯ(W_f ยท [hโโโ, xโ] + b_f). Input gate: iโ = ฯ(W_i ยท [hโโโ, xโ] + b_i). Output gate: oโ = ฯ(W_o ยท [hโโโ, xโ] + b_o).
Gate Functions
Forget gate decides what to discard. Input gate decides what to store. Output gate decides what to output. Together, they control the flow of information.
# LSTM gate equations (simplified)
import numpy as np
def lstm_step(x, h_prev, c_prev, params):
# Concatenate input and previous hidden state
concat = np.concatenate([h_prev, x])
# Gates
f = sigmoid(np.dot(W_f, concat) + b_f) # forget
i = sigmoid(np.dot(W_i, concat) + b_i) # input
o = sigmoid(np.dot(W_o, concat) + b_o) # output
c_tilde = tanh(np.dot(W_c, concat) + b_c) # candidate
c = f * c_prev + i * c_tilde
h = o * tanh(c)
return h, cโก GRU (Gated Recurrent Unit)
๐ฏ Simpler than LSTM
GRU combines forget and input gates into a single "update gate". It also merges cell state and hidden state, making it more computationally efficient.
GRU Advantages
GRU has fewer parameters than LSTM, trains faster, and often performs similarly. It's a good choice for smaller datasets or when speed is critical.
from tensorflow.keras.layers import GRU
gru_model = Sequential([
GRU(64, input_shape=(10, 32)),
Dense(1, activation='sigmoid')
])
# GRU has 3 gates (vs LSTM's 4)
print(f"GRU parameters: {gru_model.layers[0].count_params()}")๐ Bidirectional RNNs
๐ฏ Context from Both Directions
Bidirectional RNNs process sequences forward and backward, combining information from past and future context.
When to Use
Ideal for tasks where future context matters (e.g., text classification, named entity recognition). Not suitable for online/real-time prediction.
from tensorflow.keras.layers import Bidirectional
bidir_model = Sequential([
Bidirectional(LSTM(64, return_sequences=True), input_shape=(10, 32)),
Bidirectional(LSTM(32)),
Dense(1, activation='sigmoid')
])
# Bidirectional doubles the output dimension (forward + backward)๐๏ธ Stacked RNNs & Deep Models
๐ฏ Deep Recurrent Networks
Stack multiple RNN layers to learn hierarchical temporal representations. Each layer processes the output of the previous layer.
Layer Hierarchy
Lower layers capture short-term patterns, higher layers capture long-term dependencies. Return_sequences=True is needed between layers.
stacked_model = Sequential([
LSTM(64, return_sequences=True, input_shape=(10, 32)),
LSTM(64, return_sequences=True),
LSTM(32),
Dense(1, activation='sigmoid')
])๐ฏ Attention Mechanisms
๐ฏ Focusing on Important Parts
Attention allows the model to focus on relevant parts of the input sequence, learning to assign weights to different time steps.
Attention Benefits
Attention addresses the bottleneck in Seq2Seq models. It enables the decoder to access all encoder hidden states, improving performance on long sequences.
# Custom attention layer (conceptual)
def attention(query, keys, values):
# Compute attention scores
scores = np.dot(query, keys.T) / np.sqrt(d_k)
weights = softmax(scores)
context = np.dot(weights, values)
return context, weights
# Modern transformers use self-attention
from tensorflow.keras.layers import MultiHeadAttention
attention_layer = MultiHeadAttention(num_heads=8, key_dim=64)๐ค Seq2Seq & Encoder-Decoder
๐ฏ Sequence-to-Sequence Learning
Encoder processes input sequence to a context vector. Decoder generates output sequence from the context.
Applications
Machine translation, text summarization, image captioning, time series forecasting. The encoder and decoder can be RNNs, LSTMs, or transformers.
# Encoder
encoder_inputs = Input(shape=(None, 32))
encoder_lstm = LSTM(64, return_state=True)
encoder_outputs, state_h, state_c = encoder_lstm(encoder_inputs)
encoder_states = [state_h, state_c]
# Decoder
decoder_inputs = Input(shape=(None, 32))
decoder_lstm = LSTM(64, return_sequences=True, return_state=True)
decoder_outputs, _, _ = decoder_lstm(decoder_inputs, initial_state=encoder_states)
decoder_dense = Dense(vocab_size, activation='softmax')
decoder_outputs = decoder_dense(decoder_outputs)
seq2seq = Model([encoder_inputs, decoder_inputs], decoder_outputs)๐ Text Processing & Embeddings
๐ฏ Tokenization & Embeddings
Convert text to tokens, then to embeddings (word2vec, GloVe, BERT). Embeddings capture semantic relationships.
Preprocessing Pipeline
Tokenization โ Padding โ Embedding Layer โ RNN/LSTM. Use pre-trained embeddings for better performance.
from tensorflow.keras.layers import Embedding
embedding_model = Sequential([
Embedding(input_dim=10000, output_dim=128, input_length=50),
LSTM(64),
Dense(1, activation='sigmoid')
])
# Use pre-trained GloVe embeddings
# embedding_layer = Embedding(vocab_size, 100, weights=[embedding_matrix], trainable=False)๐ฆ Keras RNN/LSTM API
๐ฏ SimpleRNN, LSTM, GRU
Unified API with support for return_sequences, return_state, stateful RNNs.
Key Parameters
units: number of hidden units. return_sequences: True for stacked RNNs. return_state: True for Seq2Seq. stateful: preserve state between batches.
# Stateful LSTM for online learning
stateful_lstm = LSTM(64, stateful=True, batch_input_shape=(1, 10, 32))
# Reset states between sequences
stateful_lstm.reset_states()
# Use with larger batches
lstm_layer = LSTM(64, return_sequences=True, return_state=True)
outputs, state_h, state_c = lstm_layer(inputs)๐ Deployment & Production
๐ฏ Serving RNNs
TensorFlow Serving, ONNX, TorchScript. Optimize with quantization, pruning, and model distillation.
Production Considerations
State management in production: preserve state across sequences. Use batch inference for throughput. Monitor for data drift.
# Save and load RNN model
model.save('rnn_model.h5')
# Convert to TFLite
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
with open('model.tflite', 'wb') as f:
f.write(tflite_model)
# For stateful models, ensure proper state reset in production