Incanter bigdata jsc2012

BigData avec Incanter et ClojureJUG Summer Camp 2012

Claude Falguière @cfalguiere

goo.gl/4mmJQ1

1jeudi 13 septembre 2012

JUG Summer Camp 2012 Claude Falguiere

AU PROGRAMME

Lire et explorer des données

Faire un job MapReduce Hadoop

Générer un rapport HTML

Stocker les données dans MongoDB

Construire un projet Clojure

avec Clojure

VOTRE MISSION SI VOUS L’ACCEPTEZ ...

filtrage des erreurstransformer les donnéesregrouper par catégories

charts, distributions, corrélations

500 utilisateurs x 5h de test environ 1 000 000 relevés

rapports d’analyse de tests de performance

INCANTER

+=Math, Stats, Graps

INCANTER

manipulation de matrices et tables

statistiquescharts

modèlestest d’hypothèse

INCANTER

Motivation pour se mettre à Clojure Langage Fonctionnel

Ecosystème JavaNetlib

Parallel ColtJFreeChart

CLAUDE FALGUIERE

@cfalguiere

goo.gl/4mmJQ

http://cfalguiere.wordpress.com

ET VOUS ?

JUG Summer Camp 2012 Claude Falguiere9

Petit rappel Clojure

HELLO WORLD

(def conference "JUG Summer Camps")(println "Hello" conference)

(println "Hello JUG Summer Camp")

(count (str "Hello" conference))

-> "Hello JUG Summer Camp"

LES NOMBRES

(+ 1 2 3) -> 7

(= 2 (+ 1 1)) -> true

(+ 4 (* 2 3)) -> 10

Lire et explorer des données

SE LANCER

La voie du novice

1) downloader Incanter

2) lancer le REPL

LES EXEMPLES

https://github.com/cfalguiere/ClojureBigDataJugSummerCamp2012

PREMIER SCRIPT

LIRE DES DONNÉES

(def ds (read-dataset "readings.csv" :header true))

équivalent pour Excel

(use '(incanter core io stats charts))

(save ds "./data.csv")

LES COLLECTIONS

(def colors ["red" "magenta" "orange"])

(def colors ‘("red" "magenta" "orange"))

(def colors #{"red" "magenta" "orange"})

Clojure

LES COLLECTIONS

MatrixDataset

Incanter

LES MAPS

(def conf {:name "JUG Summer Camp" :location "La Rochelle" })

Clojure

JEU DE DONNÉES

t lb ts s rc shop srch userId ...

754 "SU01-HomePage-Page" 1330419301862 "true" 200 "MAF" "gants" "PT1022809551"

LES COLONNES

user>($ :t ds)

user> (mean ($ :t ds)) 994,75

QUANTILE 95%

(quantile ($ :t ds) :probs [0.95])

Temps maximum pour 95% des relevés

911.1758241758242

($ [:lb :t :shop] ds )

LES COLONNES

(view ($ [:lb :t :shop] ds )) ->

LES LIGNES

($where {:s "true"} ds) -> les lignes OK

($where {:s "true" ds)

LES LIGNES

($where {:s "true" :shop {:$in #{"MAF", "FAB"}} ds)

($where {:s "true" :shop {:$in #{"MAF", "FAB"}} :t {:$gt 3000 :$lt 5000}} ds)

WITH-DATA

($where {:shop "MAF"} ds)

($ :t ($where {:shop "MAF"} ds))

(view ($ :t ($where {:shop "MAF"} ds)))

(with-data ds

(view ($ :t ($where {:shop "MAF"} ))))ds

INCANTER CHARTS

Time-series

Histograms

Bar charts

Box Plot

Scatter Plot

(time-series-plot :ts :t :data ds)

(view (time-series-plot :ts :t :data ds))

(view (time-series-plot :ts :t :data ds :title "Readings" :x-label "time" :y-label "resp. time (ms)" :legend true :series-label "duration ms"))

(time-series-plot :ts :t :data ds :title "Readings" :x-label "time" :y-label "resp. time (ms)" :legend true :series-label "duration ms" )

(doto (time-series-plot :ts :t :data ds :title "Readings" :x-label "time" :y-label "resp. time (ms)" :legend true :series-label "duration ms" ) (add-lines ts (repeat 3000) :series-label "threshold") (set-stroke-color java.awt.Color/blue :series 0) (set-stroke :width 1 :series 0) (set-stroke-color java.awt.Color/red :series 1) (set-stroke :width 3 :series 1) view)

(time-series-plot :ts :t :data ds :title "Readings" :x-label "time" :y-label "resp. time (ms)" :legend true :series-label "duration ms" )

(time-series-plot...

(add-lines (time-series-plot...

(set-stroke-color (add-lines (time-series-plot...

(view (set-stroke-color (add-lines (time-series-plot...

(time-series-plot :ts :t :data ds :title "Readings" :x-label "time" :y-label "resp. time (ms)" :legend true :series-label "duration ms" ) (add-lines ts (repeat 3000) :series-label "threshold")

(time-series-plot :ts :t :data ds :title "Readings" :x-label "time" :y-label "resp. time (ms)" :legend true :series-label "duration ms" ) (add-lines ts (repeat 3000) :series-label "threshold") (set-stroke-color java.awt.Color/blue :series 0)

(time-series-plot :ts :t :data ds :title "Readings" :x-label "time" :y-label "resp. time (ms)" :legend true :series-label "duration ms" ) (add-lines ts (repeat 3000) :series-label "threshold") (set-stroke-color java.awt.Color/blue :series 0) view

(doto (time-series-plot :ts :t :data ds :title "Readings" :x-label "time" :y-label "resp. time (ms)" :legend true :series-label "duration ms" ) (add-lines ts (repeat 3000) :series-label "threshold") (set-stroke-color java.awt.Color/blue :series 0) view)

REGROUPEMENT

(def groups ($group-by :shop ds))

{ "MAF" "FAB" }{ {:shop "MAF"} {:shop "FAB"} }

(get groups {:shop "MAF"})

LIRE UN GROUPE

$ROLLUP

user> ($rollup count :t :shop ds)

Applique une fonction à chaque entrée d’un $group-by

fonction regroupement[:shop :t]["MSK" 170]["KIE" 76]["OIB" 342]["FAB" 171]...

MOYENNE

user> ($rollup mean :t :lb ds)

Moyenne par groupe

[:lb :t]["SU61-BrowseCat2-Page" 911.1758241758242]["SU43-Paiment-Page" 1770.6004566210045]["SU42-DeliveryAndBilling-Page" 732.0340136054422]["SU41-CustomerDetails-Page" 608.2222222222222]...

($rollup count :t :shop ds)

user>(view (bar-chart :shop :t :vertical false :data ($rollup count :t :shop ds)))

DIAGRAMMES EN BARRES

(histogram :t :data ds)

ENREGISTRER LES CHARTS

(save (histogram :t :data ds) "histogram.png")

REJOUER LE SCRIPT

(use '(incanter core io stats charts))(def ds (read-dataset "../data/readings.csv" :header true) )(save (histogram :ts :t :data ds) "histogram.png")(println ($rollup mean :t :lb ds))

analyse.clj

user>(load-file "analyse.clj")

ET ENSUITE ...

Les limitations du REPL seul- difficile d’écrire des commandes complexes- pas de tests unitaires - que les libs fournies dans l’uberjar incanter

lein ...

Construire un projet Clojure

L’ECOSYSTEME CLOJURE

build et gestion de dépendance lein (leiningen)

repos Mavenclojars, sonatype

tests unitairesclojure.test (built-in), midge

intégration avec les IDEswank, jark

L’ATELIER

REPL + editeur de texteou

Emacs + Slime + SwankEclipse + CounterClockWiseIDEA + La Clojure + LeiningenNetbeans + EnclojureVim + VimClojure

SE LANCER

La voie du gourou

installer leiningen

lein new myproject

configurer project.clj

lein <task>

demo classes lib project.clj src demo core.clj test demo test core.clj

(defproject demo "1.0.0-SNAPSHOT" :description "demos incanter " )

PROJECT.CLJ

(defproject demo "1.0.0-SNAPSHOT" :description "demos incanter" :dependencies [[org.clojure/clojure "1.3.0"] [incanter "1.3.0"] ] )

(defproject demo "1.0.0-SNAPSHOT" :description "demos incanter" :dependencies [[org.clojure/clojure "1.3.0"] [incanter "1.3.0"] ] :dev-dependencies [[swank-clojure "1.4.2"]])

myproject classes lib project.clj

.m2 repository

DEPENDANCES

>lein deps

Clojars

Sonatype.m2 repository incanter incanter 1.3.0 incanter-1.3.0.jar

demo classes lib incanter-1.3.0.jar project.clj

TEST UNITAIRES

(deftest test-f1 (is 6 (f1 3)))

(defn f1 [s] (* 2 s))

clojure.test (built-in)

NAMESPACES

(ns demo.test.core (:use demo.core) (:use clojure.test))

(deftest test-f1 (is 6 (f1 3)))

(ns demo.core)

(defn f1 [s] (* 2 s))

demo classes lib project.clj src demo core.clj test demo test core.clj

RECHERCHER PAR DATE

user> (view ($where {:timestamp {:$gt start-time :$lt end-time}} ds)) user> (let [ start-time end-time ] (view ($where {:timestamp {:$gt start-time :$lt end-time}} ds)))

user> (let [ start-time (date-time 2012 02 28 8 55) ] (view ($where {:timestamp {:$gt start-time :$lt end-time}} ds)))

user> (let [ start-time (date-time 2012 02 28 8 55) end-time (date-time 2012 02 28 8 57) ] (view ($where {:timestamp {:$gt start-time :$lt end-time}} ds)))

(ns demo.utils (:use [clj-time.core :only date-time]) )

TEST TIME

(deftest test-readable-timestamp (is ) )

(deftest test-readable-timestamp (is (= 2012 (year adate ))) )

(deftest test-readable-timestamp (let [ result (readable-timestamp ds) ] (is (= 2012 (year (sel result :rows 0 :cols :timestamp ))))))

(deftest test-readable-timestamp (let [ ds (dataset [:ts] [{:ts 1330418007548} {:ts 1330418007549}]) result (readable-timestamp ds) ] (is (= 2012 (year (sel result :rows 0 :cols :timestamp ))))))

MAPPER UNE COLONNE

($map fct-to-date :ts ds )

CLJ-TIME (JODA)

(coerce/from-long 1330418007548)

#<Date Tue Feb 28 09:33:27 CET 2012>

(ns demo.utils (:require [clj-time.coerce :as coerce]) )

($map :ts ds ) ($map #(coerce/from-long %) :ts ds )

MAPPER UNE COLONNE

fonction anonyme

AUGMENTER LE DATASET

($map #(coerce/from-long %) :ts ds )

(dataset [:timestamp]

($map #(coerce/from-long %) :ts ds ))

(conj-cols ds (dataset [:timestamp]

($map #(coerce/from-long %) :ts ds )))

(defn readable-timestamp [ds] (conj-cols ds (dataset [:timestamp]

($map #(coerce/from-long %) :ts ds ))))

LEIN TEST

>lein test Testing loaderdemo.test.jmeterloaderRan 3 tests containing 5 assertions.0 failures, 0 errors.Testing loaderdemo.test.xmlloaderRan 5 tests containing 13 assertions.0 failures, 0 errors.Testing loaderdemo.test.coreTesting loaderdemo.test.jmeterloaderTesting loaderdemo.test.xmlloaderRan 9 tests containing 19 assertions.0 failures, 0 errors.

SCRIPT

(ns demo.core:use [[incanter.core] [demo.utils]] ;; readable-timestamp:require [[incanter.io :as io]

[incanter.stats :as stats][incanter.charts :as charts]])

(let [ds (readable-timestamp(io/read-dataset "readings.csv"))](save (charts/time-series-plot :ts :t :data ds) "times.png")(println ($rollup mean :t :lb ds)))

demo/core.clj

(ns demo.core:use [[incanter.core] [demo.utils]] ;; readable-timestamp:require [[incanter.io :as io]

[incanter.stats :as stats][incanter.charts :as charts]])

(defn -main [& args] (let [ds (readable-timestamp(io/read-dataset "readings.csv"))] (save (charts/time-series-plot :ts :t :data ds) "times.png")(println ($rollup mean :t :lb ds)))

lein runNo :main namespace specified in project.clj.

demo/core.clj

LEIN RUN

(defproject demo "1.0.0-SNAPSHOT" :description "demos incanter" :dependencies ... :dev-dependencies ... :main demo.core)

lein runread 1738

LEIN REPL

lein replREPL started; server listening on localhost port 61426demo.core=>

LES GRIMOIRES

incanter.orgdata-sorcery.org

github.com/liebke/incanter

Incanter sur StackoverflowIncanter Google Group

Stocker les données

ARCHITECTURE

Clojure

Document

CongoMongo

Incanter.mongodb

MongoDBServer

SE CONNECTER

(ns dm (:require [somnium.congomongo :as cm] [incanter.core :as incanter] [incanter.mongodb :as im]))

(def conn (cm/make-connection :demodb))

#'demomongo/conn

(im/insert-dataset :readings ds)

MASS INSERT

(cm/with-mongo conn (im/insert-dataset :readings ds))

({:ts "1330419301862", :search__phrase "gants", :userId "PT1022809551", ...

DANS MONGO

mongoconnecting to: test> show dbsdemodb 0.203125GBlocal (empty)

{ "_id" : ObjectId("5034e71703643050fef59474"), "ts" : "1330419324275", "search__phrase" : "gants", "userId" : "PT1022809551", "hrts" : "2012-02-28T08:55:24.275Z", "rc" : NumberLong(200), "productId" : "8519304225180894762", "shop" : "MAF", "lb" : "SU12b-PickProduct-Page", "t" : NumberLong(1177), "s" : "true" }...

db.readings.find( {'lb':'SU12b-PickProduct-Page', 'shop':'MAF', 'search__phrase:‘gants’} )

(im/fetch-dataset :readings ))

(cm/fetch :readings :where {:shop "MAF"} )

({:ts "1330419301862", :search__phrase "gants", :userId "PT1022809551", :hrts "2012-02-28T08:55:01.862Z", ...

pas d’intégration Incanter JDBC

des librairies Clojureclojure.java.jdbcKormaClojureQL

Générer un rapport HTML

LE WEB ET CLOJURE

Moustache ou Compojure

Enlive ou Hiccup

LE RAPPORT

HICCUP

(ns hiccupdemo.core (:use hiccup.core) (:use hiccup.page) (:use hiccup.element)

(defn build-table [{:keys [rows]}] (html ))

FORMATER LA TABLE

(defn build-table [{:keys [rows]}] (html [:table ]))

(defn build-table [{:keys [rows]}] (html [:table (table-headers) (map table-row rows) ]))

(defn table-row [{:keys [lb count mean min max q95]}] (html [:tr [:td lb] [:td {:class "number"} count] [:td {:class "number"} mean ] ... ]]))

FORMATER LA LIGNE

(defn table-row [{:keys [lb count mean min max q95]}] (html [:tr [:td lb] [:td {:class "number"} count] [:td {:class "number"} (format "%6.2f" mean)]... ]]))

DESTRUCTURINGDE MAP

(defn build-table [statsds] (html [:table (table-headers) (map table-row (:rows statsds)) ]))

user=> (keys statsds)(:column-names :rows)

DESTRUCTURINGDE MAP

user=> (keys statsds)(:column-names :rows)

(defn build-table [{:keys [rows]}] (html [:table (table-headers) (map table-row rows) ]))

45350 relevés"Elapsed time: 24940.557 msecs" => loader"Elapsed time: 24542.737 msecs" => 1er chart"Elapsed time: 3042.975 msecs" => 2eme chart"Elapsed time: 24317.2 msecs" => stats"Elapsed time: 122.479 msecs" => rapport

OPTIONS JVM

lein runException in thread "AWT-Shutdown" java.lang.OutOfMemoryError: Java heap space! at java.util.IdentityHashMap$Values.iterator(IdentityHashMap.java:1009)! at java.util.Collections$SynchronizedCollection.iterator(Collections.java:1573)

:main perfdemo.core :jvm-opts ["-Xmx1g"] )

project.clj

PERFORMANCE

Consolesurtout ne listez pas les données :o

Mémoiredonnées accumulées dans le dataset

IO Disquesi beaucoup de graphes à sauver sur disque

Faire un job MapReduce Hadoop

MAP REDUCE

output

:input-format source format

CLOJURE-HADOOP

:map-reader fn:map fn

:reduce fn

(defjob/defjob job :map my-map :map-reader wrap/int-string-map-reader :reduce my-reduce :input-format :text)

(defn my-map [key line] (cond (= key 0) [] :else ... ))

(defn my-map [key line] (cond (= key 0) [] :else [["all" t] [ lb t] [ shop t]] )))

(defn my-map [key line] (cond (= key 0) [] :else [["all" t] [(str "lb:" lb) t] [(str "shop:" shop) t]] )))

(defn csv-line-parser [line] (map #(.trim %) (first (csv/parse-csv line))))

;; require clojure-csv

(defn my-map [key line] (cond (= key 0) [] :else (let [ cells (csv-line-parser line) ] [["all" t] [(str "lb:" lb) t] [(str "shop:" shop) t]] )))

(defn my-map [key line] (cond (= key 0) [] :else (let [ cells (csv-line-parser line) t (nth cells 0) ] [["all" t] [(str "lb:" lb) t] [(str "shop:" shop) t]] )))

(defn my-map [key line] (cond (= key 0) [] :else (let [ cells (csv-line-parser line) t (Long/parseLong (nth cells 0)) ] [["all" t] [(str "lb:" lb) t] [(str "shop:" shop) t]] )))

(defn my-map [key line] (cond (= key 0) [] :else (let [ cells (csv-line-parser line) t (Long/parseLong (nth cells 0)) [lb, shop] (map #(nth cells %) [1,6])] [["all" t] [(str "lb:" lb) t] [(str "shop:" shop) t]] )))

REDUCE

(defn my-reduce [key values-fn] [[(str key ":count") (count values-fn)] [(str key ":mean") (stats/mean values-fn)]])

LEIN JAR

lein jar

et installer déployer dans hadoop

CASCALOG

Cascading + Datalog + Clojure

user=> (?<- (stdout) [?person ?age] (age ?person ?age) (< ?age 30))

Dites, vous aimez Prolog ?

LES ÉCOLES DE MAGIE

Paris Clojure User Group

http://dev.clojure.org/display/community/Clojure+User+Groups

Clojure sur Stackoverflow

4clojure.comlabrepl

CLAUDE FALGUIERE

@cfalguiere

goo.gl/4mmJQ Merci

Incanter bigdata jsc2012

Technology

BigData selon IBM

Presentation 2013 Bigdata

Présentation du Groupe · #bigdata 5 Pluralité d’expertises pour répondre à vos besoins • Intégration de progiciels • Nearshore • Centres de services • AMOA • PMO

Analyse en temps réel de BigData

Morning Tech#1 BigData - Oxalide Academy

Rapport Veille Salon-MobileIT&BigData

Bigdata opensource

20160216 - From BigData to BigProcessing

Bigdata et datamining au service de la transition énergétique

Hadoop / Big Data - Tokidevcours.tokidev.fr/bigdata/cours/mbds_big_data_hadoop_2019... · 2019-10-28 · Exemple : Blue Gene (1999) Supercalculateur « classique ». Connecte 131072

Statistiques Datamining Bigdata Un aperçu des enjeux, des ...pierrechainais.ec-lille.fr/Centrale/Option_DAD/... · Statistiques –Datamining –Bigdata Un aperçu des enjeux, des

Petit-déjeuner MapReduce-La révolution dans l’analyse des BigData

1 BIGDATA Stella Kemgang Samuel Quettier Dzung Nguyen Cyrille Meli Encadré par : Julie Josse

Avec Hadoop, Excel et… 1€, réalisez votre premier Mobile BigData Tracker en moins de 45 min !

Baina bigdata introduction 2016

Stratégiede spécialisation intelligentede la Wallonieeconomie.wallonie.be/sites/default/files/Présentation S3... · 2019. 2. 4. · Plate-forme Bigdata INFOPOLE CLUSTER TIC Mobilité

BigData comment l'utilisateur peut-il s'en sortir ?

Taller Apache Cassandra - eventos.citius.usc.eseventos.citius.usc.es/bigdata/workshops/Cassandra.pdf · Introducción Que es Apache Cassandra 3 Apache Cassandra es un motor de bases

BigData - Semaine 2 · 2019-05-20 · BigData - Semaine 2 Types des clés et valeurs ClasseArrayWritable LetypeArrayWritable représentedestableauxdeWritable quelconques. Ilestpréférabledelasous

Réseaux, Sécurité et objets Connectés –RSC · 2020. 5. 14. · Business Intelligence/BigData Worldline, Lyon Véhicule autonome : Développement d'outils dédiés aux IMH embarqués