๐ง Database Fundamentals
๐ฏ What is a Biological Database?
A biological database is an organized collection of biological data (sequences, structures, pathways, expression) that is stored, managed, and made accessible for querying and analysis. Biological databases are essential for storing the massive amounts of data generated by genomics, proteomics, and structural biology. They enable researchers to retrieve, compare, and analyze biological information.
Why Databases Matter
Storage: Massive data. Retrieval: Fast access. Integration: Cross-reference. Annotation: Curated knowledge. Analysis: Comparative. Reproducibility: Standardized.
# Biological Databases โ the big picture
# Organized collections of biological data.
print("""
=== Biological Databases ===
What they are:
- Organized collections of biological data
- Sequences, structures, pathways, expression
- Stored, managed, accessible
- Enable retrieval and analysis
Why they matter:
- Storage of massive data
- Fast retrieval
- Cross-referencing
- Curated annotation
- Comparative analysis
- Reproducibility
Major databases:
- NCBI (GenBank, RefSeq, PubMed)
- UniProt (protein sequences)
- Ensembl (genomes)
- PDB (protein structures)
- KEGG (pathways)
- Reactome (pathways)
- GEO (expression)
- Pfam (protein families)
- InterPro (protein domains)
Applications:
- Sequence retrieval
- BLAST search
- Phylogenetics
- Structure analysis
- Pathway analysis
- Expression analysis
""")
print("=== Historical Context ===")
print("""
1960s: First protein sequence databases (Atlas of Protein Sequence)
1970s: PDB (1971), GenBank (1982)
1980s: EMBL, Swiss-Prot
1990s: NCBI, Entrez, KEGG
2000s: Ensembl, UniProt, GEO
2010s: Big data, cloud, APIs
2020s: AI, knowledge graphs, integration
""")
print("=== Why It's Hard ===")
print("""
1. DATA VOLUME
- Petabytes of data
- Growing exponentially
2. DATA HETEROGENEITY
- Different formats
- Different standards
3. DATA QUALITY
- Errors
- Redundancy
- Incomplete annotation
4. INTEGRATION
- Cross-referencing
- Identifiers
- Ontologies
5. ACCESS
- APIs
- Rate limits
- Authentication
""")๐ฏ Why Databases Matter
๐ฏ The Value of Biological Databases
Biological databases are the backbone of modern biology. They store, organize, and make accessible the data that drives discovery.
Key Benefits
Storage: Petabytes. Retrieval: Fast. Integration: Cross-ref. Annotation: Curated. Analysis: Comparative. Sharing: Standardized.
# Why Databases Matter
print("=== 1. Data Storage ===")
print("""
Storage:
- GenBank: > 3 billion sequences
- UniProt: > 200 million proteins
- PDB: > 200,000 structures
- GEO: > 5 million samples
Growth:
- Exponential
- Doubling every ~2 years
- Big data challenge
""")
print("\n=== 2. Fast Retrieval ===")
print("""
Retrieval:
- Query by ID
- Query by sequence
- Query by keyword
- BLAST search
Example:
Entrez: search NCBI
UniProt: search proteins
PDB: search structures
""")
print("\n=== 3. Cross-Referencing ===")
print("""
Cross-referencing:
- Link databases
- Same entity, different DBs
- Integration
Example:
UniProt โ PDB (structure)
UniProt โ Ensembl (genome)
UniProt โ KEGG (pathway)
GenBank โ PubMed (literature)
""")
print("\n=== 4. Curated Annotation ===")
print("""
Annotation:
- Manual curation
- Automated annotation
- Quality control
Example:
Swiss-Prot (curated)
RefSeq (curated)
Pfam (curated)
""")
print("\n=== 5. Comparative Analysis ===")
print("""
Comparative:
- BLAST
- Multiple alignment
- Phylogenetics
- Structure comparison
Example:
BLAST against GenBank
โ Find homologs
""")
print("\n=== 6. Reproducibility ===")
print("""
Reproducibility:
- Standardized data
- Versioning
- Accession numbers
- Citation
Example:
GenBank accession: NC_000001
โ Same sequence everywhere
""")
print("\n=== 7. Knowledge Discovery ===")
print("""
Discovery:
- Data mining
- Machine learning
- Knowledge graphs
- Drug discovery
Example:
Drug target identification
โ Database mining
""")
print("\n=== 8. Clinical Applications ===")
print("""
Clinical:
- Variant interpretation
- Disease genes
- Pharmacogenomics
Example:
ClinVar
โ Clinical variants
โ Diagnosis
""")
print("\n=== 9. Education ===")
print("""
Education:
- Teaching
- Training
- Resources
Example:
NCBI tutorials
UniProt tutorials
PDB tutorials
""")
print("\n=== 10. Research ===")
print("""
Research:
- Hypothesis generation
- Data analysis
- Publication
Example:
Database search
โ New hypothesis
โ Experiment
โ Publication
""")๐ Types of Databases
๐ฏ Database Classification
Biological databases can be classified in several ways: by data type, by curation level, by scope, and by access method.
Types of Databases
Primary: Raw data. Secondary: Curated. Composite: Integrated. Specialized: Specific. Sequence: DNA/protein. Structure: 3D.
# Types of Databases
print("=== 1. Primary Databases ===")
print("""
Primary:
- Raw experimental data
- Submitted by researchers
- May be redundant
- May have errors
Examples:
- GenBank (DNA)
- PDB (structures)
- GEO (expression)
- SRA (reads)
- PRIDE (proteomics)
Characteristics:
- Direct submission
- No curation
- Redundant
""")
print("\n=== 2. Secondary Databases ===")
print("""
Secondary:
- Derived from primary
- Curated
- Annotated
- Non-redundant
Examples:
- UniProt (proteins)
- RefSeq (reference)
- Pfam (families)
- InterPro (domains)
- KEGG (pathways)
Characteristics:
- Curated
- Annotated
- Integrated
""")
print("\n=== 3. Composite Databases ===")
print("""
Composite:
- Integrate multiple sources
- Unified interface
- Cross-referenced
Examples:
- Entrez (NCBI)
- Ensembl (genomes)
- UCSC (genomes)
- InterPro (domains)
Characteristics:
- Integration
- Cross-referencing
- Unified query
""")
print("\n=== 4. Specialized Databases ===")
print("""
Specialized:
- Specific data type
- Specific organism
- Specific disease
Examples:
- FlyBase (Drosophila)
- TAIR (Arabidopsis)
- SGD (yeast)
- OMIM (human disease)
- ClinVar (clinical variants)
Characteristics:
- Focused
- Deep annotation
- Community-specific
""")
print("\n=== 5. Sequence Databases ===")
print("""
Sequence:
- DNA, RNA, protein
- Primary + secondary
Examples:
- GenBank (DNA)
- RefSeq (reference)
- UniProt (protein)
- EMBL (DNA)
- DDBJ (DNA)
Types:
- Nucleotide
- Protein
- Genome
""")
print("\n=== 6. Structure Databases ===")
print("""
Structure:
- 3D structures
- Experimental + predicted
Examples:
- PDB (experimental)
- AlphaFold DB (predicted)
- CATH (classification)
- SCOP (classification)
- EMDB (EM)
Types:
- X-ray
- NMR
- Cryo-EM
- Predicted
""")
print("\n=== 7. Pathway Databases ===")
print("""
Pathway:
- Metabolic pathways
- Signaling pathways
- Regulatory networks
Examples:
- KEGG
- Reactome
- WikiPathways
- BioCyc
- MetaCyc
Types:
- Metabolic
- Signaling
- Regulatory
""")
print("\n=== 8. Expression Databases ===")
print("""
Expression:
- Gene expression
- Transcriptomics
- Proteomics
Examples:
- GEO (microarray/RNA-seq)
- ArrayExpress
- GTEx (human tissues)
- TCGA (cancer)
- Human Protein Atlas
Types:
- Microarray
- RNA-seq
- Proteomics
""")
print("\n=== 9. Literature Databases ===")
print("""
Literature:
- Scientific papers
- Abstracts
- Citations
Examples:
- PubMed
- PMC
- arXiv
- bioRxiv
- Google Scholar
Types:
- Abstracts
- Full text
- Preprints
""")
print("\n=== 10. Classification ===")
print(f"{'Type':<20} {'Description':<30} {'Examples':<30}")
print("-" * 80)
print(f"{'Primary':<20} {'Raw data':<30} {'GenBank, PDB':<30}")
print(f"{'Secondary':<20} {'Curated':<30} {'UniProt, RefSeq':<30}")
print(f"{'Composite':<20} {'Integrated':<30} {'Entrez, Ensembl':<30}")
print(f"{'Specialized':<20} {'Organism/disease':<30} {'FlyBase, OMIM':<30}")
print(f"{'Sequence':<20} {'DNA/protein':<30} {'GenBank, UniProt':<30}")
print(f"{'Structure':<20} {'3D structures':<30} {'PDB, AlphaFold':<30}")
print(f"{'Pathway':<20} {'Pathways':<30} {'KEGG, Reactome':<30}")
print(f"{'Expression':<20} {'Expression':<30} {'GEO, GTEx':<30}")
print(f"{'Literature':<20} {'Papers':<30} {'PubMed, PMC':<30}")๐งฌ Nucleotide Databases
๐ฏ DNA and RNA Databases
Nucleotide databases store DNA and RNA sequences. They are essential for genomics, transcriptomics, and molecular biology.
Nucleotide Databases
GenBank: Primary. RefSeq: Curated. EMBL: Europe. DDBJ: Japan. Ensembl: Genomes. UCSC: Genomes.
# Nucleotide Databases
print("=== 1. GenBank ===")
print("""
GenBank:
- NCBI
- Primary database
- All submitted sequences
- Redundant
- > 3 billion sequences
Access:
- Web: ncbi.nlm.nih.gov/genbank
- Entrez
- FTP
Formats:
- GenBank
- FASTA
- XML
Example:
Accession: NC_000001
โ Human chromosome 1
""")
print("\n=== 2. RefSeq ===")
print("""
RefSeq:
- NCBI
- Secondary database
- Curated
- Non-redundant
- Reference sequences
Access:
- Web: ncbi.nlm.nih.gov/refseq
- Entrez
Prefixes:
- NC_ (genome)
- NM_ (mRNA)
- NP_ (protein)
- NR_ (non-coding)
Example:
NM_000546
โ TP53 mRNA
""")
print("\n=== 3. EMBL ===")
print("""
EMBL:
- EBI (Europe)
- Primary database
- Part of INSDC
- Nucleotide sequences
Access:
- Web: ebi.ac.uk/ena
- API
Partners:
- GenBank (NCBI)
- DDBJ (Japan)
- INSDC (international)
""")
print("\n=== 4. DDBJ ===")
print("""
DDBJ:
- Japan
- Primary database
- Part of INSDC
- Nucleotide sequences
Access:
- Web: ddbj.nig.ac.jp
- API
""")
print("\n=== 5. Ensembl ===")
print("""
Ensembl:
- EBI + Wellcome Sanger
- Genome database
- Annotation
- Comparative genomics
Access:
- Web: ensembl.org
- REST API
- FTP
Features:
- Genomes
- Genes
- Variants
- Comparative
- Regulation
Example:
ENSG00000141510
โ TP53 gene
""")
print("\n=== 6. UCSC Genome Browser ===")
print("""
UCSC:
- University of California
- Genome browser
- Annotation
- Comparative
Access:
- Web: genome.ucsc.edu
- API
Features:
- Genomes
- Tracks
- Annotation
- Comparative
""")
print("\n=== 7. Other Nucleotide Databases ===")
print("""
1. SILVA
- rRNA
- Microbial
2. Rfam
- RNA families
- Non-coding
3. miRBase
- microRNA
4. GtRNAdb
- tRNA
5. PlasmoDB
- Plasmodium
6. VectorBase
- Vectors
""")
print("\n=== 8. Accession Numbers ===")
print("""
Accession:
- Unique identifier
- Stable
- Versioned
Examples:
- NC_000001.11 (chromosome)
- NM_000546.6 (mRNA)
- NP_000537.3 (protein)
- ENSG00000141510 (gene)
Format:
- Prefix
- Number
- Version
""")
print("\n=== 9. Format ===")
print("""
GenBank format:
LOCUS NC_000001 248956422 bp DNA
DEFINITION Homo sapiens chromosome 1
ACCESSION NC_000001
VERSION NC_000001.11
KEYWORDS RefSeq
SOURCE Homo sapiens
FEATURES ...
ORIGIN ...
//
FASTA format:
>NC_000001.11 Homo sapiens chromosome 1
NNNNNNNNNNNNNNNNNNNN...
""")
print("\n=== 10. Use Cases ===")
print("""
Use cases:
- Sequence retrieval
- BLAST search
- Primer design
- Phylogenetics
- Variant analysis
- Gene annotation
- Comparative genomics
""")๐งช Protein Databases
๐ฏ Protein Sequence and Function
Protein databases store protein sequences, structures, families, and functional annotations. They are essential for proteomics, structural biology, and functional genomics.
Protein Databases
UniProt: Sequences. Pfam: Families. InterPro: Domains. PDB: Structures. STRING: Interactions. PROSITE: Motifs.
# Protein Databases
print("=== 1. UniProt ===")
print("""
UniProt:
- Universal Protein Resource
- Most comprehensive
- Protein sequences
- Functional annotation
Components:
- Swiss-Prot (curated)
- TrEMBL (unreviewed)
- UniParc (archive)
Access:
- Web: uniprot.org
- REST API
- FTP
Example:
P04637
โ Human TP53
""")
print("\n=== 2. Pfam ===")
print("""
Pfam:
- Protein families
- HMM profiles
- Domains
- Curated
Access:
- Web: pfam.xfam.org
- API
- FTP
Example:
PF00001
โ 7 transmembrane receptor
""")
print("\n=== 3. InterPro ===")
print("""
InterPro:
- Integrative
- Protein families
- Domains
- Functional sites
Access:
- Web: ebi.ac.uk/interpro
- API
Members:
- Pfam
- PROSITE
- PRINTS
- SMART
- PANTHER
Example:
IPR001841
โ Zinc finger
""")
print("\n=== 4. PDB ===")
print("""
PDB:
- Protein Data Bank
- 3D structures
- Experimental
- X-ray, NMR, Cryo-EM
Access:
- Web: rcsb.org
- API
- FTP
Example:
1TUP
โ p53 DNA-binding domain
""")
print("\n=== 5. STRING ===")
print("""
STRING:
- Protein-protein interactions
- Functional associations
- Confidence scores
Access:
- Web: string-db.org
- API
Example:
TP53
โ Interaction network
""")
print("\n=== 6. PROSITE ===")
print("""
PROSITE:
- Protein motifs
- Patterns
- Profiles
- Functional sites
Access:
- Web: prosite.expasy.org
- API
Example:
PS00001
โ N-glycosylation site
""")
print("\n=== 7. Other Protein Databases ===")
print("""
1. PRIDE
- Proteomics
- Mass spectrometry
2. PeptideAtlas
- Peptides
- Proteomics
3. Human Protein Atlas
- Expression
- Localization
4. neXtProt
- Human proteins
- Annotation
5. MEROPS
- Proteases
- Peptidases
""")
print("\n=== 8. UniProt Format ===")
print("""
UniProt format:
ID TP53_HUMAN Reviewed; 393 AA.
AC P04637; A0A024R1X2; ...
DT 01-AUG-1988, integrated into UniProtKB/Swiss-Prot.
DE Cellular tumor antigen p53
GN Name=TP53; Synonyms=P53;
OS Homo sapiens (Human).
OX NCBI_TaxID=9606;
...
SQ SEQUENCE 393 AA;
MEEPQSDPSV EPPLSQETFS DLWKLLPENN VLSPLPSQAM DDLMLSPDDI ...
//
""")
print("\n=== 9. Use Cases ===")
print("""
Use cases:
- Protein sequence retrieval
- BLAST search
- Domain analysis
- Structure prediction
- Interaction networks
- Functional annotation
- Phylogenetics
- Drug discovery
""")
print("\n=== 10. Cross-References ===")
print("""
Cross-references:
- UniProt โ PDB
- UniProt โ Ensembl
- UniProt โ KEGG
- UniProt โ Pfam
- UniProt โ InterPro
- UniProt โ STRING
Example:
UniProt P04637
โ PDB: 1TUP, 2OCJ, ...
โ Ensembl: ENSG00000141510
โ KEGG: hsa:7157
""")๐ Genome Databases
๐ฏ Whole Genome Data
Genome databases store complete genome sequences, annotations, and comparative data. They are essential for genomics, evolutionary biology, and precision medicine.
Genome Databases
Ensembl: Vertebrates. UCSC: Browser. NCBI: Genomes. JGI: Plants/microbes. TAIR: Arabidopsis. FlyBase: Drosophila.
# Genome Databases
print("=== 1. Ensembl ===")
print("""
Ensembl:
- EBI + Sanger
- Vertebrate genomes
- Annotation
- Comparative genomics
Access:
- Web: ensembl.org
- REST API
- FTP
Features:
- Genomes
- Genes
- Transcripts
- Variants
- Regulation
- Comparative
Example:
ENSG00000141510
โ TP53
""")
print("\n=== 2. UCSC Genome Browser ===")
print("""
UCSC:
- University of California
- Genome browser
- Many genomes
- Tracks
Access:
- Web: genome.ucsc.edu
- API
- FTP
Features:
- Genomes
- Tracks
- Annotation
- Comparative
- Table Browser
Example:
hg38
โ Human genome
""")
print("\n=== 3. NCBI Genome ===")
print("""
NCBI Genome:
- NCBI
- All genomes
- Assembly
- Annotation
Access:
- Web: ncbi.nlm.nih.gov/genome
- Entrez
- FTP
Features:
- Assemblies
- Annotation
- BLAST
- Downloads
Example:
GCF_000001405.40
โ Human GRCh38
""")
print("\n=== 4. JGI ===")
print("""
JGI (Joint Genome Institute):
- DOE
- Plants
- Microbes
- Fungi
Access:
- Web: jgi.doe.gov
- Phytozome
- MycoCosm
Features:
- Genomes
- Annotation
- Comparative
- Tools
Example:
Phytozome
โ Plant genomes
""")
print("\n=== 5. TAIR ===")
print("""
TAIR:
- Arabidopsis
- Model plant
- Annotation
- Tools
Access:
- Web: arabidopsis.org
Features:
- Genome
- Genes
- Markers
- Expression
- Literature
""")
print("\n=== 6. FlyBase ===")
print("""
FlyBase:
- Drosophila
- Model organism
- Annotation
- Tools
Access:
- Web: flybase.org
Features:
- Genome
- Genes
- Mutants
- Expression
- Literature
""")
print("\n=== 7. SGD ===")
print("""
SGD (Saccharomyces Genome Database):
- Yeast
- Model organism
- Annotation
- Tools
Access:
- Web: yeastgenome.org
Features:
- Genome
- Genes
- Proteins
- Interactions
- Expression
""")
print("\n=== 8. WormBase ===")
print("""
WormBase:
- C. elegans
- Model organism
- Annotation
- Tools
Access:
- Web: wormbase.org
Features:
- Genome
- Genes
- Expression
- Phenotypes
""")
print("\n=== 9. Other Genome Databases ===")
print("""
1. ZFIN
- Zebrafish
2. MGI
- Mouse
3. RGD
- Rat
4. Gramene
- Grasses
5. Phytozome
- Plants
6. Ensembl Genomes
- Non-vertebrates
7. NCBI Assembly
- All assemblies
""")
print("\n=== 10. Use Cases ===")
print("""
Use cases:
- Genome browsing
- Gene retrieval
- Variant analysis
- Comparative genomics
- Phylogenetics
- Annotation
- Primer design
- CRISPR design
""")๐๏ธ Structure Databases
๐ฏ 3D Structural Data
Structure databases store 3D structures of biological macromolecules. They are essential for structural biology, drug discovery, and understanding molecular function.
Structure Databases
PDB: Experimental. AlphaFold: Predicted. EMDB: EM maps. CATH: Classification. SCOP: Classification. BMRB: NMR.
# Structure Databases
print("=== 1. PDB ===")
print("""
PDB (Protein Data Bank):
- Experimental structures
- X-ray, NMR, Cryo-EM
- > 200,000 structures
- Worldwide
Access:
- Web: rcsb.org
- API
- FTP
Formats:
- PDB
- mmCIF
- PDBML
Example:
1TUP
โ p53 DNA-binding domain
""")
print("\n=== 2. AlphaFold DB ===")
print("""
AlphaFold DB:
- Predicted structures
- DeepMind + EBI
- > 200 million
- Proteomes
Access:
- Web: alphafold.ebi.ac.uk
- API
- FTP
Coverage:
- Human
- Model organisms
- UniProt
Example:
AF-P04637-F1
โ TP53 predicted structure
""")
print("\n=== 3. EMDB ===")
print("""
EMDB (Electron Microscopy Data Bank):
- Cryo-EM maps
- 3D reconstructions
- EBI
Access:
- Web: emdb-empiar.org
- API
Example:
EMD-1234
โ Cryo-EM map
""")
print("\n=== 4. CATH ===")
print("""
CATH:
- Protein structure classification
- Class, Architecture, Topology, Homology
- Hierarchical
Access:
- Web: cathdb.info
- API
Example:
1.10.8.10
โ Classification
""")
print("\n=== 5. SCOP ===")
print("""
SCOP:
- Structural Classification of Proteins
- Hierarchical
- Manual curation
Access:
- Web: scop.mrc-lmb.cam.ac.uk
- API
Example:
a.1.1.1
โ Classification
""")
print("\n=== 6. BMRB ===")
print("""
BMRB (Biological Magnetic Resonance Data Bank):
- NMR data
- Chemical shifts
- Metabolomics
Access:
- Web: bmrb.io
- API
Example:
15000
โ NMR entry
""")
print("\n=== 7. Other Structure Databases ===")
print("""
1. PDBe
- PDB Europe
2. RCSB PDB
- PDB US
3. PDBj
- PDB Japan
4. ModBase
- Comparative models
5. SWISS-MODEL
- Homology models
6. AlphaFold
- Predicted
""")
print("\n=== 8. PDB Format ===")
print("""
PDB format:
HEADER DNA-BINDING PROTEIN
TITLE CRYSTAL STRUCTURE OF P53
COMPND MOL_ID: 1;
SOURCE ORGANISM_SCIENTIFIC: HOMO SAPIENS;
KEYWDS DNA-BINDING, TUMOR SUPPRESSOR
EXPDTA X-RAY DIFFRACTION
AUTHOR Y. CHO, S. GORINA, P.D. JEFFREY, N.P. PAVLETICH
REVDAT ...
ATOM 1 N MET A 1 35.123 45.678 55.789 1.00 50.00 N
...
END
""")
print("\n=== 9. mmCIF Format ===")
print("""
mmCIF:
- Modern format
- PDBx/mmCIF
- Larger structures
- More metadata
Example:
_entry.id 1TUP
_struct.title 'CRYSTAL STRUCTURE OF P53'
_atom_site.group_PDB ATOM
_atom_site.id 1
_atom_site.label_atom_id N
_atom_site.label_comp_id MET
_atom_site.label_asym_id A
_atom_site.label_seq_id 1
_atom_site.Cartn_x 35.123
_atom_site.Cartn_y 45.678
_atom_site.Cartn_z 55.789
""")
print("\n=== 10. Use Cases ===")
print("""
Use cases:
- Structure visualization
- Docking
- Drug design
- Structure prediction
- Molecular dynamics
- Comparative modeling
- Function prediction
""")๐ Pathway Databases
๐ฏ Pathways and Networks
Pathway databases store metabolic, signaling, and regulatory pathways. They are essential for systems biology, drug discovery, and understanding cellular processes.
Pathway Databases
KEGG: Metabolic. Reactome: Curated. WikiPathways: Community. BioCyc: Many. MetaCyc: Metabolic. PANTHER: Classification.
# Pathway Databases
print("=== 1. KEGG ===")
print("""
KEGG (Kyoto Encyclopedia of Genes and Genomes):
- Metabolic pathways
- Signaling pathways
- Disease pathways
- Drug pathways
Access:
- Web: kegg.jp
- REST API
Databases:
- PATHWAY
- GENES
- COMPOUND
- DRUG
- DISEASE
- BRITE
Example:
hsa04115
โ p53 signaling pathway
""")
print("\n=== 2. Reactome ===")
print("""
Reactome:
- Curated pathways
- Human + model organisms
- Expert-written
- Open source
Access:
- Web: reactome.org
- API
Features:
- Pathways
- Reactions
- Proteins
- Compounds
- Disease
Example:
R-HSA-69541
โ TP53 regulation
""")
print("\n=== 3. WikiPathways ===")
print("""
WikiPathways:
- Community-curated
- Open
- Many organisms
- Wiki-style
Access:
- Web: wikipathways.org
- API
Features:
- Pathways
- Community
- Curation
- GPML format
Example:
WP1742
โ p53 pathway
""")
print("\n=== 4. BioCyc ===")
print("""
BioCyc:
- Many organisms
- Metabolic pathways
- Genome databases
- Curated
Access:
- Web: biocyc.org
- API
Features:
- Pathways
- Compounds
- Genes
- Reactions
Example:
HUMAN
โ HumanCyc
""")
print("\n=== 5. MetaCyc ===")
print("""
MetaCyc:
- Metabolic pathways
- Non-redundant
- Curated
- Reference
Access:
- Web: metacyc.org
- API
Features:
- Pathways
- Compounds
- Enzymes
- Reactions
Example:
PWY-1234
โ Pathway
""")
print("\n=== 6. PANTHER ===")
print("""
PANTHER:
- Protein classification
- Pathways
- Families
- Curated
Access:
- Web: pantherdb.org
- API
Features:
- Pathways
- Families
- Classification
- Ontology
Example:
P00001
โ Pathway
""")
print("\n=== 7. Other Pathway Databases ===")
print("""
1. SMPDB
- Small molecule pathways
- Disease
2. PharmGKB
- Pharmacogenomics
- Drug pathways
3. INOH
- Signaling
- Networks
4. NetPath
- Signaling
- Curated
5. Pathway Commons
- Integration
- Many sources
""")
print("\n=== 8. KEGG Format ===")
print("""
KEGG format:
ENTRY hsa04115 Pathway
NAME p53 signaling pathway
DESCRIPTION ...
CLASS Cellular Processes; Cell growth and death
PATHWAY_MAP hsa04115
GENE 7157 TP53
...
COMPOUND C00002 ATP
...
REFERENCE ...
//
KGML (KEGG Markup Language):
...
""")
print("\n=== 9. Use Cases ===")
print("""
Use cases:
- Pathway analysis
- Enrichment analysis
- Systems biology
- Drug discovery
- Disease mechanisms
- Metabolic engineering
- Network analysis
""")
print("\n=== 10. Cross-References ===")
print("""
Cross-references:
- KEGG โ UniProt
- KEGG โ Ensembl
- KEGG โ PDB
- Reactome โ UniProt
- Reactome โ Ensembl
- WikiPathways โ Ensembl
Example:
KEGG hsa04115
โ UniProt P04637
โ Ensembl ENSG00000141510
""")๐ Expression Databases
๐ฏ Gene Expression Data
Expression databases store gene expression data from microarrays, RNA-seq, and proteomics. They are essential for transcriptomics, functional genomics, and disease research.
Expression Databases
GEO: NCBI. ArrayExpress: EBI. GTEx: Human tissues. TCGA: Cancer. HPA: Protein. CCLE: Cell lines.
# Expression Databases
print("=== 1. GEO ===")
print("""
GEO (Gene Expression Omnibus):
- NCBI
- Microarray + RNA-seq
- > 5 million samples
- Free
Access:
- Web: ncbi.nlm.nih.gov/geo
- Entrez
- FTP
Features:
- Datasets
- Samples
- Platforms
- Series
Example:
GSE12345
โ Expression dataset
""")
print("\n=== 2. ArrayExpress ===")
print("""
ArrayExpress:
- EBI
- Microarray + RNA-seq
- Curated
- Free
Access:
- Web: ebi.ac.uk/arrayexpress
- API
Features:
- Experiments
- Samples
- Protocols
- Data
Example:
E-MTAB-1234
โ Experiment
""")
print("\n=== 3. GTEx ===")
print("""
GTEx (Genotype-Tissue Expression):
- Human tissues
- Expression + genotype
- Multiple tissues
- Free
Access:
- Web: gtexportal.org
- API
Features:
- Expression
- eQTLs
- Tissues
- Genotypes
Example:
GTEX-1234
โ Sample
""")
print("\n=== 4. TCGA ===")
print("""
TCGA (The Cancer Genome Atlas):
- Cancer
- Multi-omics
- Many cancer types
- Free
Access:
- Web: cancer.gov/tcga
- GDC
Features:
- Expression
- Mutations
- Copy number
- Methylation
Example:
TCGA-01-1234
โ Sample
""")
print("\n=== 5. Human Protein Atlas ===")
print("""
Human Protein Atlas:
- Protein expression
- Localization
- Tissue
- Cell
Access:
- Web: proteinatlas.org
- API
Features:
- Expression
- Localization
- Tissue
- Cancer
Example:
TP53
โ Protein expression
""")
print("\n=== 6. CCLE ===")
print("""
CCLE (Cancer Cell Line Encyclopedia):
- Cancer cell lines
- Expression
- Mutations
- Drug response
Access:
- Web: broadinstitute.org/ccle
- API
Features:
- Expression
- Mutations
- Copy number
- Drug
Example:
A549
โ Cell line
""")
print("\n=== 7. Other Expression Databases ===")
print("""
1. ENCODE
- Functional genomics
- Epigenomics
2. Roadmap Epigenomics
- Epigenomes
- Human
3. FANTOM
- Transcriptomics
- Mouse/human
4. Single Cell Portal
- Single-cell
- Broad
5. Human Cell Atlas
- Single-cell
- Human
""")
print("\n=== 8. GEO Format ===")
print("""
GEO:
Series: GSE12345
Samples: GSM123456, GSM123457, ...
Platform: GPL12345
Sample:
GSM123456
Title: Control sample
Organism: Homo sapiens
Tissue: Liver
Treatment: None
Data:
Raw: .CEL, .fastq
Processed: .txt, .csv
""")
print("\n=== 9. Use Cases ===")
print("""
Use cases:
- Differential expression
- Co-expression
- Biomarker discovery
- Drug response
- Disease mechanisms
- Tissue specificity
- Single-cell
""")
print("\n=== 10. Analysis ===")
print("""
Analysis:
- Differential expression
- Clustering
- PCA
- Pathway enrichment
- Network analysis
- Machine learning
Tools:
- GEO2R
- R (limma, DESeq2)
- Python (scikit-learn)
""")๐ ๏ธ Tools (Entrez, UniProt API)
๐ฏ Accessing Databases
Most biological databases offer APIs and tools for programmatic access. Here are the most important ones.
Access Tools
Entrez: NCBI. UniProt API: REST. Ensembl API: REST. PDB API: REST. KEGG API: REST. Biopython: Python.
# Tools
print("=== 1. NCBI Entrez (E-utilities) ===")
print("""
Entrez:
- NCBI API
- Access to all NCBI databases
- E-utilities
Databases:
- nucleotide
- protein
- pubmed
- gene
- genome
- structure
E-utilities:
- esearch: search
- efetch: retrieve
- esummary: summary
- elink: cross-reference
URL:
https://eutils.ncbi.nlm.nih.gov/entrez/eutils/
Example:
esearch.fcgi?db=protein&term=TP53&retmax=10
""")
print("\n=== 2. UniProt REST API ===")
print("""
UniProt API:
- RESTful
- Protein data
- Search, retrieve, map
Endpoints:
- /uniprotkb/search
- /uniprotkb/{accession}
- /uniprotkb/stream
- /idmapping/run
URL:
https://rest.uniprot.org/
Example:
https://rest.uniprot.org/uniprotkb/P04637
""")
print("\n=== 3. Ensembl REST API ===")
print("""
Ensembl API:
- RESTful
- Genome data
- Genes, variants, sequences
Endpoints:
- /lookup/id/{id}
- /sequence/id/{id}
- /vep/{species}/hgvs/{hgvs}
- /overlap/region/{species}/{region}
URL:
https://rest.ensembl.org/
Example:
https://rest.ensembl.org/lookup/id/ENSG00000141510
""")
print("\n=== 4. PDB API ===")
print("""
PDB API:
- RESTful
- Structure data
- Search, retrieve
Endpoints:
- /rest/v1/entry/{id}
- /rest/v1/search
- /rest/v1/core/{id}
URL:
https://data.rcsb.org/
Example:
https://data.rcsb.org/rest/v1/core/entry/1TUP
""")
print("\n=== 5. KEGG API ===")
print("""
KEGG API:
- RESTful
- Pathway data
- Genes, compounds
Endpoints:
- /get/{db}/{entry}
- /list/{db}
- /find/{db}/{query}
URL:
https://rest.kegg.jp/
Example:
https://rest.kegg.jp/get/hsa04115
""")
print("\n=== 6. Biopython ===")
print("""
Biopython:
- Python library
- Entrez
- Parsing
- Analysis
Modules:
- Bio.Entrez
- Bio.SeqIO
- Bio.PDB
- Bio.Blast
Example:
from Bio import Entrez
Entrez.email = "your@email.com"
handle = Entrez.esearch(db="protein", term="TP53")
record = Entrez.read(handle)
""")
print("\n=== 7. Other Tools ===")
print("""
1. Requests
- Python HTTP
- REST APIs
2. curl
- Command line
- HTTP
3. wget
- Download
- FTP
4. Entrez Direct
- Command line
- NCBI
5. Dataset
- NCBI
- Command line
""")
print("\n=== 8. Rate Limits ===")
print("""
Rate limits:
- NCBI: 3 requests/second (without API key)
- NCBI: 10 requests/second (with API key)
- UniProt: reasonable
- Ensembl: 15 requests/second
- PDB: reasonable
- KEGG: reasonable
Best practices:
- Use API key
- Cache results
- Respect limits
- Batch requests
""")
print("\n=== 9. Example Queries ===")
print("""
NCBI Entrez:
esearch.fcgi?db=protein&term=TP53[gene]+AND+human[orgn]
efetch.fcgi?db=protein&id=NP_000537&rettype=fasta
UniProt:
https://rest.uniprot.org/uniprotkb/search?query=TP53+AND+organism_id:9606
Ensembl:
https://rest.ensembl.org/lookup/id/ENSG00000141510?expand=1
PDB:
https://data.rcsb.org/rest/v1/core/entry/1TUP
KEGG:
https://rest.kegg.jp/get/hsa04115
""")
print("\n=== 10. Best Practices ===")
print("""
1. Use API key
2. Cache results
3. Respect rate limits
4. Handle errors
5. Use HTTPS
6. Read documentation
7. Cite databases
8. Version control
""")๐ Python Integration
๐ฏ Accessing Databases from Python
Python is the language of choice for bioinformatics. Here's how to access biological databases from Python.
Python
Biopython: Entrez. Requests: REST. Pandas: Data. Matplotlib: Plot. Jupyter: Notebook. Automate: Pipeline.
# Python Integration
print("=== 1. NCBI Entrez with Biopython ===")
print("""
from Bio import Entrez
# Set email (required)
Entrez.email = "your.email@example.com"
# Search
handle = Entrez.esearch(db="protein", term="TP53 AND human", retmax=10)
record = Entrez.read(handle)
handle.close()
print(record["IdList"])
# Fetch
handle = Entrez.efetch(db="protein", id=record["IdList"], rettype="fasta")
sequences = handle.read()
handle.close()
""")
print("\n=== 2. UniProt REST API ===")
print("""
import requests
def get_uniprot(accession):
url = f"https://rest.uniprot.org/uniprotkb/{accession}.json"
response = requests.get(url)
if response.status_code == 200:
return response.json()
return None
# Example
data = get_uniprot("P04637")
if data:
print(data["primaryAccession"])
print(data["proteinDescription"]["recommendedName"]["fullName"]["value"])
""")
print("\n=== 3. Ensembl REST API ===")
print("""
import requests
def get_ensembl_gene(gene_id):
url = f"https://rest.ensembl.org/lookup/id/{gene_id}?expand=1"
headers = {"Content-Type": "application/json"}
response = requests.get(url, headers=headers)
if response.status_code == 200:
return response.json()
return None
# Example
gene = get_ensembl_gene("ENSG00000141510")
if gene:
print(gene["display_name"])
print(gene["description"])
""")
print("\n=== 4. PDB API ===")
print("""
import requests
def get_pdb_entry(pdb_id):
url = f"https://data.rcsb.org/rest/v1/core/entry/{pdb_id}"
response = requests.get(url)
if response.status_code == 200:
return response.json()
return None
# Example
entry = get_pdb_entry("1TUP")
if entry:
print(entry["struct"]["title"])
""")
print("\n=== 5. KEGG API ===")
print("""
import requests
def get_kegg_pathway(pathway_id):
url = f"https://rest.kegg.jp/get/{pathway_id}"
response = requests.get(url)
if response.status_code == 200:
return response.text
return None
# Example
pathway = get_kegg_pathway("hsa04115")
if pathway:
print(pathway[:500])
""")
print("\n=== 6. Parse FASTA ===")
print("""
from Bio import SeqIO
def parse_fasta(filepath):
sequences = []
for record in SeqIO.parse(filepath, "fasta"):
sequences.append({
"id": record.id,
"description": record.description,
"sequence": str(record.seq)
})
return sequences
# Example
# sequences = parse_fasta("sequences.fasta")
""")
print("\n=== 7. Parse GenBank ===")
print("""
from Bio import SeqIO
def parse_genbank(filepath):
records = []
for record in SeqIO.parse(filepath, "genbank"):
records.append({
"id": record.id,
"name": record.name,
"description": record.description,
"sequence": str(record.seq),
"features": [(f.type, f.location) for f in record.features]
})
return records
# Example
# records = parse_genbank("sequence.gb")
""")
print("\n=== 8. Parse PDB ===")
print("""
from Bio.PDB import PDBParser
def parse_pdb(filepath):
parser = PDBParser()
structure = parser.get_structure("protein", filepath)
for model in structure:
for chain in model:
for residue in chain:
print(residue)
return structure
# Example
# structure = parse_pdb("1tup.pdb")
""")
print("\n=== 9. Complete Pipeline ===")
print("""
from Bio import Entrez, SeqIO
import requests
import pandas as pd
def database_pipeline(gene):
\"\"\"Complete database pipeline.\"\"\"
# Search NCBI
Entrez.email = "your.email@example.com"
handle = Entrez.esearch(db="protein", term=f"{gene} AND human", retmax=5)
record = Entrez.read(handle)
handle.close()
# Fetch
handle = Entrez.efetch(db="protein", id=record["IdList"], rettype="fasta")
sequences = list(SeqIO.parse(handle, "fasta"))
handle.close()
# Get UniProt
uniprot = get_uniprot("P04637")
# Get Ensembl
ensembl = get_ensembl_gene("ENSG00000141510")
# Combine
data = {
"gene": gene,
"ncbi_count": len(sequences),
"uniprot": uniprot["primaryAccession"] if uniprot else None,
"ensembl": ensembl["display_name"] if ensembl else None
}
return data
# Use
# data = database_pipeline("TP53")
# print(data)
""")
print("\n=== 10. Visualization ===")
print("""
import matplotlib.pyplot as plt
import pandas as pd
def plot_expression(data):
\"\"\"Plot expression data.\"\"\"
df = pd.DataFrame(data)
plt.figure(figsize=(10, 6))
plt.bar(df["tissue"], df["expression"], color='purple')
plt.xlabel("Tissue")
plt.ylabel("Expression")
plt.title("Gene Expression")
plt.xticks(rotation=45)
plt.tight_layout()
plt.savefig("expression.png")
plt.show()
# Use
# plot_expression(data)
""")โ๏ธ Database Comparison
๐ฏ Which Database?
Choosing the right database depends on your data type and analysis goals.
Comparison
NCBI: Comprehensive. UniProt: Proteins. Ensembl: Genomes. PDB: Structures. KEGG: Pathways. GEO: Expression.
| Database | Type | Data | Access | Best For |
|---|---|---|---|---|
| NCBI | Composite | Sequences | Web, API | General |
| UniProt | Secondary | Proteins | Web, API | Protein sequences |
| Ensembl | Composite | Genomes | Web, API | Genomes |
| PDB | Primary | Structures | Web, API | 3D structures |
| KEGG | Secondary | Pathways | Web, API | Pathways |
| Reactome | Secondary | Pathways | Web, API | Curated pathways |
| GEO | Primary | Expression | Web, API | Expression |
| Pfam | Secondary | Families | Web, API | Protein families |
| InterPro | Composite | Domains | Web, API | Domains |
| AlphaFold | Secondary | Structures | Web, API | Predicted |
# Decision Guide
print("=== Choose Database by Data Type ===")
print("""
DNA sequences:
โ GenBank
โ RefSeq
โ Ensembl
Protein sequences:
โ UniProt
โ RefSeq
Protein structures:
โ PDB
โ AlphaFold DB
Protein families:
โ Pfam
โ InterPro
Genomes:
โ Ensembl
โ UCSC
โ NCBI Genome
Pathways:
โ KEGG
โ Reactome
โ WikiPathways
Expression:
โ GEO
โ ArrayExpress
โ GTEx
Interactions:
โ STRING
โ BioGRID
Diseases:
โ OMIM
โ ClinVar
""")
print("\n=== Speed Comparison ===")
print(f"{'Database':<20} {'Size':<20} {'Access':<15} {'API':<10}")
print("-" * 65)
print(f"{'NCBI':<20} {'3B+ sequences':<20} {'Web, API':<15} {'Yes':<10}")
print(f"{'UniProt':<20} {'200M+ proteins':<20} {'Web, API':<15} {'Yes':<10}")
print(f"{'Ensembl':<20} {'100+ genomes':<20} {'Web, API':<15} {'Yes':<10}")
print(f"{'PDB':<20} {'200K+ structures':<20} {'Web, API':<15} {'Yes':<10}")
print(f"{'KEGG':<20} {'500+ pathways':<20} {'Web, API':<15} {'Yes':<10}")
print("\n=== When to Use Which ===")
print("""
USE NCBI:
โ General
โ Sequences
โ Literature
โ BLAST
USE UniProt:
โ Proteins
โ Functional annotation
โ Cross-references
USE Ensembl:
โ Genomes
โ Comparative
โ Variants
USE PDB:
โ Structures
โ Drug design
โ Visualization
USE KEGG:
โ Pathways
โ Metabolic
โ Disease
USE GEO:
โ Expression
โ Differential
โ Biomarkers
""")โ FAQ
๐ฏ Frequently Asked Questions
Answers to common questions about biological databases.
What is a biological database?
A biological database is an organized collection of biological data (sequences, structures, pathways, expression) that is stored, managed, and made accessible for querying and analysis. Examples include NCBI, UniProt, Ensembl, PDB, and KEGG.
What is the difference between primary and secondary databases?
Primary databases store raw experimental data (e.g., GenBank, PDB, GEO). Secondary databases store derived, curated, or integrated data (e.g., UniProt, Pfam, KEGG, Reactome). Primary data is submitted by researchers; secondary data is curated and annotated.
Which biological database is best for protein sequences?
UniProt is the most comprehensive protein sequence database, with Swiss-Prot (curated) and TrEMBL (unreviewed). For protein structures, use PDB. For protein families and domains, use Pfam and InterPro.
How do I access biological databases programmatically?
Most databases offer APIs: NCBI Entrez (E-utilities), UniProt REST API, Ensembl REST API, PDB API, and KEGG REST API. You can access them from Python using Biopython (Entrez), requests, or dedicated libraries.
What is the difference between GenBank and RefSeq?
GenBank is a primary database containing all submitted sequences (may include redundancy and errors). RefSeq is a secondary, curated database of reference sequences for genomes, transcripts, and proteins, with non-redundant, well-annotated entries.
What is an accession number?
An accession number is a unique identifier assigned to a sequence, structure, or other biological record in a database. It is stable and versioned. Examples: NC_000001.11, NM_000546.6, P04637.
What is the difference between KEGG and Reactome?
KEGG is a comprehensive database of metabolic and signaling pathways, with a focus on metabolism and disease. Reactome is a curated, expert-written database of human and model organism pathways, with detailed molecular reactions.
How do I download data from biological databases?
Most databases offer FTP downloads, API access, and web interfaces. Use FTP for bulk downloads, APIs for programmatic access, and web for browsing. Examples: NCBI FTP, UniProt FTP, Ensembl FTP.
What is the best database for gene expression data?
GEO (Gene Expression Omnibus) is the largest public repository for gene expression data (microarray, RNA-seq). For human tissue-specific expression, use GTEx. For cancer, use TCGA. For protein expression, use Human Protein Atlas.
How do I cite a biological database?
Always cite the database and its primary publication. For example: "UniProt Consortium (2023) UniProt: the Universal Protein Knowledgebase. Nucleic Acids Research." Check each database's website for citation guidelines.
# Quick Reference
print("=== Quick Answers ===")
print("""
Q: What is a biological database?
A: Organized collection of biological data.
Q: Primary vs secondary?
A: Primary = raw; secondary = curated.
Q: Best for protein?
A: UniProt.
Q: Best for structures?
A: PDB, AlphaFold.
Q: Best for genomes?
A: Ensembl, UCSC.
Q: Best for pathways?
A: KEGG, Reactome.
Q: Best for expression?
A: GEO, GTEx.
Q: How to access?
A: Web, API, FTP, Biopython.
Q: GenBank vs RefSeq?
A: GenBank = primary; RefSeq = curated.
Q: Accession number?
A: Unique identifier.
Q: How to cite?
A: Cite database and publication.
Q: Free?
A: Yes โ all major databases are free.
""")๐ Future
๐ฏ Where Databases Are Going
The future is AI-integrated, knowledge graphs, cloud-native, and multi-omics.
Future Directions
AI: Integration. Graphs: Knowledge. Cloud: Native. Multi-omics: Integrated. Real-time: Updates. FAIR: Data principles.
# Future of Biological Databases
print("=== 1. AI Integration ===")
print("""
Trend: AI + databases
Ideas:
- AI-powered search
- Intelligent annotation
- Predictive models
- Knowledge discovery
Examples:
- AlphaFold DB
- GPT for biology
- AI curation
Future:
- Smart databases
- Automated annotation
- Predictive
""")
print("\n=== 2. Knowledge Graphs ===")
print("""
Trend: Knowledge graphs
Current:
- Relational databases
- Flat files
Future:
- Knowledge graphs
- Linked data
- Semantic web
Examples:
- Bio2RDF
- Wikidata
- SPOKE
- Hetionet
Future:
- Integrated knowledge
- Reasoning
- Discovery
""")
print("\n=== 3. Cloud-Native ===")
print("""
Trend: Cloud databases
Current:
- Local servers
- FTP downloads
Future:
- Cloud-native
- Serverless
- Auto-scaling
- Pay per use
Examples:
- AWS Open Data
- Google Cloud
- Azure
- Terra
Future:
- Petabyte-scale
- Real-time
- Global
""")
print("\n=== 4. Multi-Omics ===")
print("""
Trend: Multi-omics integration
Current:
- Separate databases
- Genomics, proteomics, etc.
Future:
- Integrated
- Multi-omics
- Systems biology
Examples:
- TCGA
- GTEx
- Human Cell Atlas
- UK Biobank
Future:
- Unified
- Integrated
- Systems-level
""")
print("\n=== 5. Real-Time Data ===")
print("""
Trend: Real-time updates
Current:
- Periodic releases
- Static
Future:
- Real-time
- Streaming
- Live
Examples:
- COVID-19 databases
- GISAID
- Nextstrain
Future:
- Real-time
- Continuous
- Dynamic
""")
print("\n=== 6. FAIR Principles ===")
print("""
Trend: FAIR data
FAIR:
- Findable
- Accessible
- Interoperable
- Reusable
Future:
- All databases FAIR
- Standardized
- Machine-readable
Examples:
- DataCite
- DOIs
- Metadata standards
""")
# TIMELINE
print("\n=== Timeline ===")
print("""
Past:
1970s: PDB, GenBank
1980s: EMBL, Swiss-Prot
1990s: NCBI, Entrez, KEGG
2000s: Ensembl, UniProt
2010s: Big data, APIs
2020s: AI, knowledge graphs
Present:
2025: Cloud-native
2026: AI-integrated
Future:
2030: Knowledge graphs
2035: Multi-omics
2040: Real-time?
""")
print("\n=== Final Thoughts ===")
print("""
Biological databases are:
- Essential for research
- Growing exponentially
- Becoming more integrated
- AI-powered
- Cloud-native
The future is:
- AI-integrated
- Knowledge graphs
- Cloud-native
- Multi-omics
- Real-time
- FAIR
"Data is the new oil โ but
biological data is the new gold."
Keep exploring. Keep building.
""")