Volume

Refers to the amount of data accumulated over time; modern Big Data systems often manage extremely large datasets measured in terabytes or petabytes.

Velocity

The speed at which data is generated, transmitted, and processed, especially in real‑time applications.

Variety

The diversity of data types and formats—structured, unstructured, and semi‑structured—that increases complexity in storage and analysis.

Veracity

The integrity, trustworthiness, and accuracy of data, especially important when data sources are noisy or unreliable.

Value

The usefulness or business benefit derived from analyzing data.

Variability

The volatility or inconsistency of data flows, formats, or meanings over time.

Big Data Challenges

Big Data systems face challenges in storage, transmission, and computation due to large volumes, high velocity, and diverse data types.

Data Science

The scientific study of creating, validating, and transforming data to produce meaning.

Data Scientist

A professional who uses scientific methods to extract meaning from raw data.

Curse of Dimensionality

As the number of features increases, the amount of data required to generalize accurately grows exponentially, making learning tasks more difficult.

Inductive Research

A data‑driven approach where patterns and insights emerge from the data itself.

Deductive Research

A theory‑driven approach where data is used to validate or reject a hypothesis.

Abductive Research

A mixed approach combining inductive and deductive reasoning.

Human‑Generated Data

Data produced through human interaction with systems, such as online services or digital devices.

Machine‑Generated Data

Data produced automatically by software or hardware in response to real‑world events.

Structured Data

Data organized in relational models capturing relationships between entities.

Unstructured Data

Data that does not conform to a predefined schema; makes up ~80% of enterprise data.

Semi‑Structured Data

Data with some organizational properties (hierarchical or graph‑based) but not fully relational.

Supervised Learning

Machine learning trained on labeled data and tested on unseen data.

Unsupervised Learning

Machine learning that identifies patterns without labeled training data.

Semi‑Supervised Learning

Learning using a small labeled dataset combined with a large unlabeled dataset.

Descriptive Analytics

Answers “what happened?” using historical data, often via dashboards or reports.

Diagnostic Analytics

Answers “why did it happen?” by identifying causes behind past events.

Predictive Analytics

Uses historical data to estimate future outcomes.

Prescriptive Analytics

Recommends actions based on analytical results and explains why a particular option is best.

Business Intelligence (BI)

Analyzes business process data to provide insights that improve organizational performance.

Key Performance Indicator (KPI)

A quantifiable metric used to measure success relative to strategic goals.

Quantitative Data Analysis

Uses mathematical or statistical criteria to analyze data.

Qualitative Data Analysis

Uses human visual perception and interpretation to analyze data.

Dimensionality Reduction

Projects high‑dimensional data into lower dimensions while preserving important information.

Compressed Sensing

Acquires and reconstructs data at rates below the Nyquist limit using sparsity and incoherence.

Manifold Learning

Nonlinear dimensionality reduction that uncovers low‑dimensional structures in high‑dimensional data.

Kernel Learning

Uses kernel functions to map data into higher dimensions where it becomes linearly separable.

Reinforcement Learning

Algorithms learn by interacting with an environment and maximizing rewards.

Deep Learning

Uses multi‑layer neural networks to perform complex learning tasks.

Data Procurement

Obtaining data for analysis; external acquisition (government/commercial sources) can be expensive.

Privacy in Big Data

Analytics may reveal confidential information; requires privacy regulations, tagging, and anonymization.

Security in Big Data

Ensures secure data networks and repositories; NoSQL systems often lack robust built‑in security and rely on plaintext HTTP APIs.

Provenance

Metadata describing the source, processing, and storage of data; essential for authenticity and auditing.

Governance Requirements

Policies regulating data tagging, privacy, anonymization, cleansing, filtering, and archiving.

Cloud Adoption

Organizations use cloud environments for scalable storage and processing when in‑house resources are insufficient.

Big Data Analytics Lifecycle

Consists of nine stages: 1. Business Case Evaluation 2. Data Identification 3. Data Acquisition & Filtering 4. Data Extraction 5. Data Validation & Cleansing 6. Data Aggregation & Representation 7. Data Analysis 8. Data Visualization 9. Utilization of Analysis Results

1) Business Case Evaluation

Defines justification, motivation, and goals for analysis; identifies KPIs or SMART goals.

2) Data Identification

Determines required datasets and their sources; diverse sources increase chances of finding patterns.

3) Data Acquisition & Filtering

Removes corrupt, irrelevant, or noisy data; filtered data may still be useful for other analyses.

4) Data Extraction

Converts disparate internal/external data into formats compatible with Big Data solutions.

5) Data Validation & Cleansing

Removes invalid data to prevent skewed or false results; external data may lack structure.

6) Data Aggregation & Representation

Combines data across datasets; challenges include structure differences, semantic mismatches, and large processing volumes.

7) Data Analysis

Uses mining and statistical techniques to discover patterns, correlations, and anomalies; may be iterative.

Confirmatory Analysis

Deductive approach testing a hypothesis using data.

Exploratory Analysis

Inductive approach discovering patterns without predefined assumptions.

8) Data Visualization

Graphically communicates results for interpretation; enables visual analysis and discovery of new questions.

9) Utilization of Analysis Results

Converts analytical insights into business value; produces models capturing patterns and relationships.

Data Mining

Automated techniques that sift through large datasets to identify hidden patterns; foundational for predictive analytics and BI.

A/B Testing

Statistical method comparing control and treatment versions to determine which performs better; common in marketing.

Correlation Analysis

Determines whether two variables are related and characterizes the strength/direction of the relationship.

What is the order of complexity of the categories of data analytics?

(High) 1. Prescriptive 2. Predictive 3. Diagnostic 4. Descriptive (Low)

What is the order of value of the categories of data analytics?

(Foresight) 1. Prescriptive 2. Predictive 3. Diagnostic 4. Descriptive (Hindsight)