Big and Fast Data
Georgios Gousios
09 September 2021
Big and Fast Data
What is big data?
An overloaded, fuzzy term
“Data too large to be efficiently processed on a single computer”
“Massive amounts of diverse, unstructured data prodiced by high-performance applications”
How big is “Big?”
Typical numbers associated with Big Data
- 2.5 Exabytes (\(10^3\) TB) produced daily
- IoT: 8.4 Billion devices with internet access
- Amazon: 600 orders per second (2016) up from 35 in 2012
- Alibaba in 2015: 400 orders per second
- Google in 2016: 2+ trillion searches or 65k per second
- Each query involves > 1000 machines
- Each search touches 200+ services
- Alibaba in 2018 Singles Day: 1 billion revenue in 1 min and 25 secs.
How big is “Big?” – Instagram
- 1B daily users, clicking around the app
- 95M photos daily
- Most followed user: 181M followers
How big is “Big?” – FaceBook
Warning: numbers (\(\Uparrow\)) from 2014! Today on FB:
- 2 Billion users
- 1.32 Billion active users per day
- 350 million photos per day (148k/min)
- Every min: 510k comments, 293k status updates
The many Vs of Big data
Main Vs, by Doug Laney
- Volume: large amounts of data
- Variety: data comes in many different forms from diverse sources
- Velocity: the content is changing quickly
More Vs
- Value: data alone is not enough; how can value be derived from it?
- Veracity: can we trust the data? How accurate is it?
- Validity: ensure that the interpreted data is sound
- Visibility: data from diverse sources need to be stitched together
Volume
We call Big Data big because it is really big:
- 90% of all the data ever was created in the last 2 years
- By 2020, each person will generate 1.7MB per sec
- The Big data / data analytics industry will be worth €200 Billion in 2020