Refers to datasets that are too large, fast, or complex for traditional databases to manage efficiently.
Characteristics
- Volume
In the scale of terabytes or petabytes. - Velocity
High ingest/update rates and need for low-latency ingestion or real-time processing. - Variety
Structured, semi-structured, and unstructured data (logs, JSON, images, time series). - Veracity
Data quality and trustworthiness issues at scale. - Value
Extracting meaningful analytics or driving applications from the data.
Big Data Storage Systems
Distributed File Systems (DFS)
Stores data across many machines. Offers a single logical view. Provides redundancy via replication to handle failures.
Example: Google FS, Hadoop HDFS.
Hadoop HDFS features:
- NameNode stores metadata (file to block mapping).
- DataNode stores actual block data.
- Block size is typically 64 MB.
- Write-once-read-many model.
Sharding
Partitions data using a shard key. Improves scalability but adds complexity:
- Application must route queries manually.
- Rebalancing shards is difficult.
- Increases failure points → more replicas needed.
Parallel & Distributed Databases
Use multiple nodes to run queries in parallel. When data is stored on multiple nodes, replication ensures availability.
MapReduce
Programming model for large-scale parallel computation. Allows fault-tolerant processing of data.
map()– transforms input into key–value pairs.reduce()– aggregates values by key.
Algebraic Operations & Spark
- Modern systems (Spark, Tez) extend MapReduce with relational-like operations.
- An RDD (Resilient Distributed Dataset) is a fault-tolerant, distributed data abstraction.
- Supports transformations like map, filter, reduce, join.
Streaming Data
- Continuous data flow (stocks, sensors, IoT).
- Queries on streams use windows (time/tuple-based).
- Window types:
- Tumbling (non-overlapping)
- Hopping (fixed-size overlap)
- Sliding (moving window)
- Session (per user session)
Architectures
- Lambda architecture: stream + batch layers for processing.
- Pub-Sub systems: e.g., Apache Kafka for topic-based streaming.
11. Graph Data Model
- Represents entities and their relationships as nodes and edges.
- Equivalent to ER model in structure.
- Graph Databases: e.g., Neo4J.
- Enable efficient edge traversal queries.
- Use languages like Cypher for pattern-based queries.