## Big Data
> "describes *large* amount of data (structured or unstructured) that
> are difficult to process using traditional database and software"
## Big Data
> "Big data usually includes data sets with sizes beyond the ability of commonly used
> software tools to capture, curate, manage, and process data within a tolerable elapsed time."
## Big Data
Note:
* 1 TB: ~2000h CD quality / 17,000 hours at 128kbs
* Library of congress ~10TB of printed data (26 millions books)
* 1 PB: 1PB of audio (mp3) would take 2000 years to play
* 1 EB: 1 gram of DNA can theoretically hold 455 exabytes
* 1 ZB: is equivalent to 152 million years of high-definition video
## Big Data - The 3 Vs... (and more)
### Volume
* large data sets
* data are not sampled
### Velocity
* rapidly changing
* available in real-time
### Variety
* different type: text, images, audio, video, ...
* (un)structured: JSON, XML / images, audio, music
### ... and more Vs
* **Veracity**
* how much trust can be put in the data
* **Value**
* eventually drives revenues or new features for companies
* **Variability**
* no fixed data or schema
* evolution in time
## Big Data in Music
-
## Where ?
### Content
* Audio
* Lyrics
* Metadata
### Events
* Listening patterns
* Application events
* User activity
* Social media data
### Derived data
* Crowd sourced data
* Recommendations
* Playlists
* User content
## Example: Big Data @ Spotify
* 42PB Storage
* 200TB data generated / day
* 1300 Servers
Note:
* Figures in 2015, probably around 1600 servers now depending on literature
### Algorithms & Data Structures
* Algorithmic & Complexity
* Data Structures
### Complexity
Note:
* 1 billion elements 10^9 / 1GHz 10^9
* O(n) linear: 1 billion operations / 1s
* O(n2) quadratic: 10^18 operations / 10^19 s = ~32 years
* O(log(n)) logarithmic: 20 operations / 0.01 microseconds
### Data structures
### Program optimization
* CPU
* Memory
* IO
* Network
### Parallelism
* Parallelism
* Multithreading
* Multiprocessing
### Vertical Scaling
* More CPU
* More Memory
* More Storage
Note:
* AWS: x1.32xlarge: 128cpu/1952GB RAM/2x1920GBSSD
* Dell: 96cpu/6TB RAM/24 disks (x8TB = 192TB)/
### New paradigms
* Dedicated hardware - bespoke designs
* GPU
* FPGA (Field Programmable Gate Array)
* New paradigm
* DNA Computing
* Quantum Computing
## Horizontal scaling
* Clusters
* Sharding
* Share Nothing
* Distributed Systems
## Big Data & Hadoop
* Fundations
* Google File System (2003)
* Google MapReduce (2004)
* Google BigTable (2005/2006)
* Open Source implementation
* Apache Nutch (web crawler)
* Development moved to the Hadoop project in 2006
### MapReduce
> A programming model for processing and generating large data sets with a parallel,
distributed algorithm on a cluster
> Takes advantage of the locality of data, processing it near the place it is
stored in order to reduce the distance over which it must be transmitted.
Note:
* Distribution of work
* Data locality and compute
### MapReduce
Parallel computations on a cluster
### MapReduce
Data locality
### MapReduce
### MapReduce - Word Count
def map(document):
for word in document:
emit(word, 1)
def reduce(word, values):
count = 0
for value in values:
count += value
emit(word, count)
### MapReduce - Word Count
### MapReduce - What Now?
Relatively simple computational model
*but*
Many problems can be translated to it!
* SQL
* ETL (Extract / Transform / Load)
* Machine Learning
* Bespoke analysis
* ...
## MapReduce - Limitations
* MapReduce tasks independent from each others
* Network/IO intensive in some cases (Shuffle)
* Lack of iterative/in-memory computation
## New frameworks - DAG
> Direct Acyclic Graph
## New frameworks - DAG
* Generalization of MapReduce concept
* Jobs are aware of all the tasks involved
* Allows global optimization
* Better use of resources
Implementations:
* Dremel, Spark, Tez, Drill, ...
## Big Data & Spark
* Fundations
* Berkeley's AMPLab from 2009
* Open sourced and moved as an Apache project in 2013
* Improvements on the MapReduce paradigm
* In memory cluster computing
* Iterative algorithms
* Interactive & Exploratory analysis
* Batch & Streaming
Note:
* DAG system
* Java, Python, Scala,
* Popular for data science and analysis
## Spark - RDD (Resilient Distributed Datasets)
> a fault-tolerant collection of elements that can be operated on in parallel
## Spark - Driver & Workers
## Spark - High Level Libraries
* SQL
* Streaming
* Machine Learning
* Graph