# Science
&
# Music
-
October 2017 - Ecole Centrale de Nantes
*Guillaume Gardey*
## Planning
* Session 1
* Talk & QA: *Music & Web - Architecture & Technology Overview*
* Lab 1: Working with APIs
* Session 2
* Talk & QA: *Music & Big Data - Overview of challenges & technologies*
* Lab 2: Introduction to Data Processing - Python/Pandas
Note:
* General description and organization of the 3 sessions
## Music & Web
—
## Architecture & Technology Overview
Note:
* Objectives is to give an overview of elements related to the distribution
of digital music and the new area of IT, Research & Science used in the
context of music.
## Music transformation
Note:
marque par de nombreux changement et innovations
innovations technologiques
different modes de consomation
nouveaux types de musiques et d'instruments
musique numeriques
### Technical & Digital transformation
Vinyl > Cassette > CDs > MP3 > Streaming
Note:
* Phonographe / Vinyl ~1877 / Edisson
* Cassette ~ 1960
* Compact disc ~ 1980
* MP3 ~ 1990
* Active area of innovation & technology
* Recording and distribution of music
* Music support & services
* innovations technologiques qui vont permettre de distribuer et d'ameliorer la qualite des
enregistrements
### Consumption models
* Access
* Concert / Public Events
* Record Shop
* Subscription
* Pay As You Go
* Ownership
* Physical libraries
* Digital libraries (local/remote)
Note:
* Evolution o the consumption models
* Access has evolved and shape new ways of consuming
### Internet
* Online stores
* Cloud libraries
* New services
* Recommendation
* Discovery
* Music is social
## Components of a Music service
> Thinking of building a Music service?
Note:
* Put in place the story to present different concepts
* Deep dive into architecture of a generic music service/system
## 10,000 feet view
Note:
* Very simplistic approach
* "consumer" view
* Web service
* Different types of clients (phones, laptop, tablet, ...)
## Dive 1 - Generic architecture
Note:
* Interactions are performed through APIs
* Frontends and applications communicates to APIs, that talk to private backends/api
* Some storage in place to store information
* Next is to get more into the details of API and services
## API
**Application Programming Interface**
> A set of subroutine definitions, protocols, and tools for building software and applications
## API
They are *everywhere*!
* OS (POSIX, Windows API, iOS, Android, ...)
* Software libraries (C++, Scala, Java, Python, Javascript, ...)
* Protocols, Remote APIs (HTTP, JDBC, ...)
* Web API (SOAP, REST, ...)
Note:
* Building blocks of any software
## API
API is not an implementation, only defines the *interface*
// Compare strings - C
int strcmp(const char *s1, const char *s2);
// The strcmp() and strncmp() functions return an integer greater than, equal to,
// or less than 0, according as the string s1 is greater than, equal to, or less
// than the string s2.
* Functions, Methods, Input/Output parameters, Return types
* Protocols
* Data models of Input/Output objects
Note:
* General principles behind an API
* Extend to different domain
## API & Web
* Web Services
* REST
Note:
* 2 main trends
* different type of APIs/ approaches
* common pattern nowadays
## Web Service
* SOAP (Simple Object Access Protocol)
* XML (eXtended Markup Language)
POST /InStock HTTP/1.1
Host: www.example.org
Content-Type: application/soap+xml; charset=utf-8
Content-Length: 299
SOAPAction: "http://www.w3.org/2003/05/soap-envelope"
<soap:Envelope
xmlns:soap="http://www.w3.org/2003/05/soap-envelope"
xmlns:m="http://www.example.org/stock/Manikandan">
<soap:Header>
</soap:Header>
<soap:Body>
<m:GetStockPrice>
<m:StockName>GOOGLE</m:StockName>
</m:GetStockPrice>
</soap:Body>
</soap:Envelope>
Note:
* "oldest" abstraction
* Gain success with Java
* Structured and well identify way to build API and distributed services
## Web APIs
* REST/RESTful (Representational State Transfer)
* JSON (JavaScript Object Notation)
GET /products/1
Host: https://api.example.com
Accept: application/json
{
"id": 1,
"name": "Foo",
"price": 123,
"stock": {
"warehouse": 300,
"retail": 20
}
}
Note:
* More recent abstraction
* HTTP & JSON
## Why Web Services/API?
* Composition
* Re-usability
* Testing
* Prototyping
> Sofware Development!
## Architecture Overview
Note:
* client/server, n-tier architecture, monolithic
* microservices, small services, one or a very low number of function
* increase scalability
* increased testing/deployment/replacement
## Frontend / Backend
Note:
* Common terminology
## Frontend
> Presentation layer, software closest to the end users
* Visualization
* Client application
* User interface and interaction (UI/UX)
* Lightweight
## Backend
> Business, Data access & Data storage layer
> Not accessed directly by end users
* Business logic
* Data access
* API
## Design Principles
* **Monolithic Applications**
* Self-contained
* Independence
* **Modular Applications**
* client/server, n-tiered, microservices
* loose coupling
* modularity
* reuse
Note:
* Different approaches to build software
* Monolithic, historical, comes from the mainframes
* Used to decouple elements in applications
* improve development and iterations
## Dive 2 - Content Delivery
Note:
* Infrastructure required for pushing/publishing content to consumers
* Download / buy music
* Stream music
* Access website with rich content / audio / video
### CDN
> Content Distribution Network
Note:
* Deliver content near to the location of the users
* Smaller latency, lower cost
* Improved user experience
* CDN distribution & synchronization
### Peer 2 Peer
> Decentralized network
Note:
* Relies on user exchanging data peer 2 peer
* Seeds from a smaller number of main servers
* Lower load and network activity on company's infrastructure
* Resilience
### Streaming
Note:
* DRM digital right management
* Adaptive quality / streaming
* Encoding / decoding
## Dive 3 - Data & Events Collection
Note:
* Data collection from users
* Personalized services:
* recommendation
* social features, sharing
* mailing campaign & marketing
* A/B testing, product development
* the data part, using data from users to personalize services
* buid and validate new products
### What is being collected ?
* **User Activity**
* Clicks
* Application activity
* Listening activity
* **Monitoring**
* Logs
* Application performance
* **Related servicess**
* Analytics, User engagement, tracking, ...
* Social website monitoring (Facebook/Twitter feeds, ...)
Note:
* Applications collect different type of metrics or events
* Very important area of technology and IT: Big Data & Deep learning
### Why ?
* Sales increase: Marketing / Targeting
* Product improvements
* Prioritization on features
* Performance analysis / reliability
* Recommendation
* ...
Note:
* Different objectives: performance, features, use engagement, sales, ....
* KYC - Know Your customer
## Dive 4 - 3rd Parties
Note:
* Services provided by different companies
* Collaboration - data exchange
* Services bought from other companies
### Content analysis & Enrichment
* Metadata & content analysis
* Crowd sourcing
* Clustering & classification
* Fingerprinting
* Added content
* ...
Note:
* Metadata & content analysis = Catalog authors/albums/history/...
* Crowd sourcing = LastFM / Grooveshark / Musicbrainz / ...
* Clustering & classification = Spotify / Pandora / music genre
* Fingerprinting = Shazam / Echonest
* Added content (Lyrics, Shop, recommendations, ...) = Additional content (lyrics, link to buy content
...)
* ...
### Outsourcing & External Services
* Payments
* Analytics
* CDN
* Streaming
* ...
Note:
* Company will generally specialized in one specific area
* Other services provided by other company
## Dive 5 - Infrastructure
> Where & How do we run all of this?
Note:
* We need some computers & resources to run all of this
### Where?
* Physical Data Centers
* On-premise
* DC (owned or colocation)
* Cloud Infrastructure
Note:
* Historically, self hosting and own data center
* Then collocation and hosting provider
* THen cloud infrastructure and As A Service system
### Different types of approach to infrastructure
### Physical
* Management of servers, network, cabling ...
* Human actions
* No automation
### IaaS
> IaaS - *Infrastructure As A Service*
* Servers
* Storage
* Network
* Operating System
* ...
> Amazon EC2 / S3, Windows Azure, Google Compute Engine, VmWare, OpenStack, ...
Note:
* Access to cloud features to setup VMs, OS, some network components
### PaaS
> PaaS - *Platform As A Service*
* Managed databases
* Web servers
* Content Delivery
* Container solutions
> AWS Elastic Beanstalk, AWS RDS, Heroku, Google App Engine, Cloud Foundry, ...
Note:
* PaaS vendors offer a development environment to application developers
* Pre-built solutions, don't require managing network, OS, storage
### SaaS
> SaaS - *Software As A Service*
* Managed Softwares
> Google Apps, Office 365, Gmail, Dropbox, SalesForce, ...
Note:
### Music Services (not exhaustive!)
Note:
Not only music:
* ticket sales
* gig recommendation
* ticket exchange
* music software