Big data

Definition

Big data refers to extremely large and complex data sets that cannot be managed or processed effectively using traditional databases or tools. These data sets are characterised by high volume, velocity, and variety, often requiring advanced technologies for storage, analysis, and visualisation.

For example, social media platforms generate massive streams of structured and unstructured data every second, which are analysed to identify trends, user behaviour, and sentiment.

Advanced

Big data solutions rely on distributed computing frameworks such as Hadoop and Apache Spark, which enable parallel processing across clusters of machines. Storage often leverages cloud-based architectures that support scalability and accessibility.

Advanced big data practices include real-time analytics, machine learning integration, and predictive modelling. Data lakes and warehouses are used to organise structured and unstructured data for analysis. Security, compliance, and governance also play a central role, as industries like healthcare and finance must ensure responsible data use.

Why it matters

• Enables organisations to identify patterns and insights at scale.
• Improves decision-making through predictive and prescriptive analytics.
• Supports innovation in AI, IoT, and automation technologies.
• Enhances customer experience by analysing behaviour and preferences.

Use cases

• Analysing sensor data in IoT devices for predictive maintenance.
• Studying consumer trends from e-commerce and social media.
• Detecting fraud in financial transactions.
• Supporting personalised healthcare through patient data analysis.

Metrics

• Data processing speed and latency.
• Storage capacity utilisation in data lakes and warehouses.
• Accuracy of predictive models derived from big data.
• Cost efficiency of big data infrastructure.

Issues

• High costs of infrastructure and skilled personnel.
• Data quality issues leading to unreliable insights.
• Privacy and compliance risks when handling personal data.
• Complexity in integrating multiple data sources.

Example

A retailer collects transaction data, online browsing behaviour, and loyalty program activity. By applying big data analytics, the company forecasts product demand, reduces supply chain inefficiencies, and launches targeted promotions.