# Planning
* **Tuesday 29/11**
* Talk & QA: *Music & Web - Architecture & Technology Overview*
* Lab Work 1: Working with APIs
* Lab Work 2 & 3: Introduction to Pandas & data analysis
* **Tuesday 6/12**
* Talk & QA: *Music & Big Data - Overview of challenges & technologies*
* Lab Work 4: Introduction to Spark & Data processing
Note:
* General description and organization of the 2 sessions
## Music & Web
—
### Architecture & Technology Overview
Note:
Objectives is to give an overview of elements related to the distribution
of digital music and the new area of IT, Research & Science used in the
context of music.
## Digital transformation
* Vinyl
* Cassette
* CDs
* MP3s & co
## Consumption models
* Ownership
* Physical libraries
* Digital libraries (local/remote)
* Access
* Subscription
* Pay As You Go
# Internet
* Online stores
* Cloud libraries
* New services
* Recommendation
* Discovery
* Music is social
## Components of a Music service
—
Thinking of building a Music service? :)
Note:
* Put in place the story to present different concepts
* Deep dive into architecture of a generic music service/system
## 10,000 feet view
Note:
* Very simplistic approach
* "consumer" view
* Web service
* Different types of clients (phones, laptop, tablet, ...)
## Dive 1 - Generic architecture
Note:
* Interactions are performed through APIs
* Frontends and applications communicates to APIs, that talk to private backends/api
* Some storage in place to store information
## API
An *Application Programming Interface* (API) is a set of subroutine definitions,
protocols, and tools for building software and applications.
## API
They are *everywhere*!
* OS (POSIX, Windows API, Cocoa, iOS, Android, ...)
* Software libraries (C++, Scala, Java, Python, Javascript, ...)
* Protocols, Remote APIs (JDBC, ...)
* Web API (SOAP, REST, ...)
## API
API is not an implementation, only defines the *interface*
* Protocol
* Functions
* Data models of input/output objects
Note:
* General principles behind an API
* extend to different domain
## Web Service / Web API
* Web Services
* SOAP (Simple Object Access Protocol)
* XML
* Web APIs
* REST (Representational State Transfer)
* JSON
Note:
* 2 main trends
* different type of APIs/ approaches
* common pattern nowadays
## Why Web API?
* Allow to easily build new applications
* Simplify development (API is fixed, known contract)
* Allow new services to be built, used (either internally or externally)
* Same approach than traditional software development
* Components
* Composing
* Reusability
* Testing
* ...
Note:
* How the are used
* Authentication / Authorization
* https://zapier.com/learn/apis/chapter-2-protocols
## API - System architectures
Note:
* n-tier architecture, monolithic
* microservices, small services, one or a very low number of function
* increase scalability
* increased testing/deployment/replacement
## Frontend / Backend
* **Backend**
* Business logic
* API
* **Frontend**
* Visualization
* User interaction (UX)
* Decoupling eases:
* development
* testing
* deployment
Note:
* used to decouple elements in applications
* improve development and iterations
## Dive 2 - Content Delivery
Note:
* Infrastructure required for pushing/publishing content to consumers
* Download / buy music
* Stream music
* Access website with rich content / audio / video
### CDN
> Content Distribution Network
Note:
* Deliver content nearer to users
* Smaller latency
* Improved experience
* CDN synchronization
### Peer 2 Peer
> Decentralized network
Note:
* Relies on user exchanging data peer 2 peer
* Seeds from main servers
* Lower load and network activity on company's infrastructure
### Streaming
Note:
* DRM digital right management
* Adaptive quality / streaming
* Encoding/decoding
## Dive 3 - Partners
Note:
* On the business side collaboration required to acquire content, prepare
and store it for distribution and usage by client
* B2B
## Dive 4 - Data & Events Collection
Note:
* Data collection from users
* Personalized services:
* recommendation
* social features, sharing
* mailing campaign & marketing
* A/B testing, product development
* the data part, using data from users to personalize services
* buid and validate new products
### What is being collected ?
* Events generated by users
* Click / Browsing activity
* Listening activity
* Internal services
* Performances, logs
* External services
* Analytics, User engagement, tracking, ...
* Social website monitoring (Facebook/Twitter feeds, ...)
Note:
* the data part, using data from users to personalize services
### Why ?
* Marketing / Targeting
* Improve product quality
* Focus development on specific features
* Performance analysis / reliability
* Recommendation
* ...
Note:
* the data part, using data from users to personalize services
## Dive 5 - Content analysis & Enrichment
### Content analysis & Enrichment
* Metadata & content analysis
* Crowd sourcing
* Clustering & classification
* Fingerprinting
* Added content
* ...
Note:
* Metadata & content analysis = Catalog authors/albums/history/...
* Crowd sourcing = LastFM / Grooveshark / Musicbrainz / ...
* Clustering & classification = Spotify / Pandora / music genre
* Fingerprinting = Shazam / Echonest
* Added content (Lyrics, Shop, recommendations, ...) = Additional content (lyrics, link to buy content
...)
* ...
## Dive 6 - Infrastructure
Where do we run the different services?
### Where?
* Physical Data Centers
* On-premises
* DC (owned or colocation)
* Cloud Infrastructure
### Different types of approach to infrastructure
### IaaS
> IaaS - *Infrastructure As A Service*
* Servers, Storage, Network, Operating System, ...
eg: Amazon EC2 & co, Windows Azure, Google Compute Engine, VmWare, OpenStack, ...
Note:
* Access to cloud features to setup VMs, OS, some network components
### PaaS
> PaaS - *Platform As A Service*
* Managed databases, web servers, container solutions, ...
eg: AWS Elastic Beanstalk, AWS RDS, Heroku, Google App Engine, Cloud Foundry, ...
Note:
* PaaS vendors offer a development environment to application developers
* Pre-built solutions, don't require managing network, OS, storage
### SaaS
> SaaS - *Software As A Service*
* User facing software / consumption
eg: Google Apps, Office 365, Gmail, Dropbox, SalesForce, ...
Note:
### Who's there?
Note:
Not only music:
* ticket sales
* gig recommendation
* ticket exchange
* music software
## Lab
Get the Jupyter notebooks
git clone https://github.com/glinmac/scimus-2016.git
Start Jupyter
cd scimus-2016
jupyter notebook