et quelques outils Olivier Richard (MdC Polytech’)mescal.imag.fr/seminars/matinee_thematique/08_04_10/richard.pdf · Di cile a utiliser pour faire des l’ eval de perf a grande

Plateformes pour l’experimentation a grande echelleet quelques outils

Olivier Richard (MdC Polytech’)

MescalLaboratoire d’Informatique de Grenoble

9 avril 2008

Olivier Richard (Mescal/LIG) Plateformes pour l’experimentaions 9 avril 2008 1 / 38

Sommaire

1 Problematique

2 PlanetLab

3 Grid’5000

4 Outils avances

5 Expo : vers un atelier pour l’experimentation

6 Conclusion


Problematique

Problematique

Evaluation des systemes distribues et HPC (calculs hautes-performances)

Systemes distribues / paralleles de plus en plus complexes

Tailles : du 1000 au 1 000 000

Approches : modelisation, simulation, evaluation

Evaluations sur plate-formes dediees

Incontournables

Ces plate-formes sont devenues elles-memes complexes

Defis : realisme, precision, reproductibilite


PlanetLab

PlanetLab

Some facts :

A global research network that supports the development of newnetwork services

850 nodes at 427 sites


PlanetLab

Planet-Lab : Principes

Architecture

Noeud : 2 a 4 machines reliees sur internet.

Machines virtuelles sur les noeuds (Vserver)

L’utilisateur cree un slice et y attache des machines virtuelles (=ensemble de machines virtuelles).

Un slice est alloue pour une duree donnee qui peut-etre etendue

Un service central (infos globales, liste des noeuds, operation degestion). Interface web et XMLRPC

Quelques services distribues de monitoring, diffusion de fichier...

Des outils developpes/maintenus par les experimentateurs (difficultede maintenance, evolution)


PlanetLab

Planet-Lab

Utilise comme demonstrateur

Difficile a utiliser pour faire des l’eval de perf a grande echelle

Reproductibilite difficile

Realiste par construction pour les conditions reseaux (mais biaisreseaux academiques, machine virtuelle)

Possibilite d’experiences a longue duree


Grid’5000

Grid’5000 : Platform for Large Scale Experiments

Objectifs :

The capability to adapt the platform to fit experiment requirements(deep reconfiguration).

The capability to reproduce experimental conditions (repeatexperiments on the same nodes, isolate experiments, etc.).

Example of Use :

Address critical issues of Grid system/middleware :

Programming, Scalability, Fault Tolerance, Scheduling

Address critical issues of Grid Networking

High performance transport protocols, Qos

Port and test applications

Investigate original mechanisms

P2P resources discovery, Desktop Grids


Grid’5000

Grid’5000

500

500

500

500

500

500

500

500

1000

Some facts :

9 sites

several clusters per site

one batch scheduler per site

5000 cores

Nodes : 2 to 8 coresAMD/Intel 64bits

2 GiB of memory

1Gib/s Ethernet adapter

some Infiniband and Myrinethigh speed networks

sites interconnected at10Gib/s by Renater


Grid’5000

Some Tools in Grid’5000 : a global picture


Grid’5000

Provisioning (Deploying)

Provisioning or deploying complete computing environment on computenodes is an important issue :

to administrators to simplify maintenance operations (installation,upgrade).

to users to dispose of their own environment and typically rootprivileges.

Computing Environment is all necessary softwares :

OS

Libraries

Runtime, Middleware

Applications

Data

Several tools exists : Freesbee (Emulab), Dolly+, Kadeploy, Xcat,SystemImager...


Grid’5000

Kadeploy : a low level deploying tool

Features :

It allows to quick deploy and manage multiple computingenvironments on cluster and multi-cluster platform.

It permits also the deployment of multiple Operating Systems onevery node without compromising the installed partitions.

Kadeploy makes it possible for users to concurrently deploycomputing environments on different sets of nodes on the sameclusters.

Application :

On Grid’5000 platform, Kadeploy is used by users to easily setup theirown environment to do more deep and controllable experiments. It’salso used to homogenize environment between different clusters.


Grid’5000

Kadeploy : Principles

A relational database is used to store all information aboutenvironments (except image of environment)

Scalable thanks to pipelined chain for environment diffusion

Deployment is composed of 14 steps (reboots, node checks, diffusion,pre- and post- installations...).

During process when a fault occurs concerned nodes are excluded.

10 minutes for 400 nodes

D D0 1 2 n

Physical Network


Grid’5000

Taktuk : a powerful parallel launcher (developed byMOAIS)

from manpage

TakTuk

A tool for large scale remote execution deployment.

Description

TakTuk is a tool for broadcasting the remote execution of one or morecommands to a set of one or more distant machines. TakTuk combineslocal parallelization (using concurrent deployment processes) and workdistribution (using an adaptive work-stealing algorithm) to achieve bothscalability and efficiency.


Grid’5000

Taktuk : example

taktuk -s -f nodes-file broadcast exec hostname


Grid’5000

OAR : a Job and Resource Management System

History :

Developed after the use of OpenPBS/PBSPro on cluster of 200 nodes(2003).

Need of a scalable, robust and flexible batch scheduler with shortdevelopment cycle.

Need of a RMS to test new ideas, abstractions, schedulers.

Based on high level software components :

Relational database engine (MySql) to store all states (job,resources, internal states...).

Scripting Language (mainly Perl, ) to implement the executionengine, modules.

SSH as remote command on resource.

Taktuk to execute parallel operation on resources.


Grid’5000

Key Features

Interactive and Batch modes

Advance Reservation

Suspend/Resume Job

Checkpoint support

Job Dependencies

Resources Matching

First-Fit scheduling (backfilling)

FairSharing

Resources Hierarchy

Multiple type of resources (as Licence, Switch, Opitical Network,virtual resources and so on)

Multiple types of jobs (deploy, best-effort, co-system, ...)

Moldable Jobs support


Grid’5000

OAR : Jobs and Resources Status


Grid’5000

OAR : Gantt Diagram


Grid’5000

Grid’5000

Grid’5000 c’est super, mais

Ne possede pas le realisme reseau de PlanetLab

C’est un instrument, pas un plate-forme invivoCe n’est pas vraiment une grille (une grille legere a la rigueur)

Pas adapte pour des experiences de longue duree

Plate-forme complementaire a PlanetLab


Outils avances

Outils avances

Les outils de bases ne sont pas suffisant

Outil de deploiement (kadeploy) trop complexe a utiliser

Reservation multi-sites pas simple

Est-ce que les noeuds sont propres ?

Y-a-t’il des perturbations ?

Comme reproduire un contexte experimental (charge de fond)

Des outils pour faciliter le deroulement des experiences


Outils avances

Outils avances

Besoin d’outils avances

Monitoring reseaux precis

Isolement reseau

Outils pour le controle de l’heterogeneite

Injecteurs de charges

Injecteurs de pannes

Emulation, virtualisation ...

Outils pour la calibration

Atelier pour la conduite d’experience

Kavlan, P2PLAB, EWAN, disco, FAIL, VDS, Wreakavoc, expo, GRUDU, ...


Expo : vers un atelier pour l’experimentation




Problematic : Large Scale Experiments

Large scale platforms

Study the performances of distributed applications

On a large number of nodes

Running on several geographic locations

Efficient experiments

Obtain accurate and reproducible results

Obtain a maximum of information

In a minimum of time



To Conduct Experiments is a Difficult Process

Many parameters

Nodes are not as homogeneous as they should be

Nodes status can vary between nodes and through time

Environment is ever-changing, other experimenters are working

Managing large experiments is tedious

How to ?

Account for nodes heterogeneity

Bring nodes in a known state

Check the environment

Plan and run experiments



Objectives : Methodology and Tools to Solve thisProblematic

A methodology thatguarantees :

Reproducibility ofmeasurements

Reproducibility ofmeasurements

Efficiency of the process

Automation of theprocess

Design tools that :

Helps the experimenter

Enforces the methodology

Can adapt to severalplatform

Expo ExperimentFramework



Framework to Conduct Experiments : a Global Picture

Configuration Experimental plan

execution

Experimental control

Platform status control

Observations synthesis :

− Experimental platform

− Subject measurements

Experimental plan

Measurement tool setup

Architecture configuration

Description :

− Architecture

− Protocols

− Metrics

− Corect behavior

− Softwares

Analysis

Storage

Conclusion

Synthesis

Grid5000 Architecture :

− Configurable

− Distributed

Configuration

Observation

Problem definition

QuestionSubject

Exper

imen

t en

gin

eIn

terf

ace

Use

r

Characteristics :

Experiment description

Execution engine

Resources management

Native archiving

Fine command control

Multi-platforms

Related Works :

Plush + Nebula

DART

Zenturio

Various ScientificWorkflow Engines



Expo : Experiment Execution Engine

Example of experiment onlightweight grid.

Architecture :

Client/Server

Versatile parallel launcher(Taktuk)



Expo : Architecture of Experiment Execution Engine

Characteristics :

Script Language : Ruby

Embedded Domain Specific Language to describe experiment

Operation on Resources Set

Extensible (driver for new platform, new command...)



Main Expo’s Commands

Global Commands

check resources (nodes) checking

task task execution

atask asychronous task execution

ptask parallel task execution

patask asychronous parallel task execution

barrier waiting of asychronous tasks completion

copy file copy from one node to another one

Grid’5000’s Specific Commands

oargridsub resources reservation w/ oargrid

oargridconnect connection to previous reserved resources

kadeploy low-level environment deployment thanks to kadeploy

akadeploy asychronous low-level environment deployment

kadeploy progress deployment progression monitoring



Experiment Description : Simple Example

#l o a d s p e c i f i c f u n c t i o n s f o r G r i d ’5000 p l a t f o r mr e q u i r e ’ expo g5k ’

#r e s e r v a t i o n on gdx en h e l i o s c l u s t e r so a r g r i d s u b : r e s => ” gdx : nodes =10, h e l i o s : nodes=10”

#nodes ’ c h e c k i n gcheck $ a l l#e x e c u t i o n on each nodesp t a s k $ a l l . gateway , $ a l l , ” d at e ”

#e x e c u t i o n on gdx c l u s t e rid , r e s = p t a s k $ a l l [ ” gdx ” ] . gateway , $ a l l [ ” gdx ” ] , ” s l e e p 1”p u t s ” a v e r a g e : ” + r e s . m e a n d u r a t i o n



Case Study : File Broadcast

File broadcaster on the Grid’5000 platform

20 1 n

Launching

BBroadcast

SendingNode

Simple experiment :

Parallel Launching + File Pipelining



Experiment Description :

. . .#F i l e to b r o a d c a s t c r e a t i o nt a s k $ a l l . f i r s t , ” . / c r e a t e f i l e s ”

#I t e r a t i o n on node s e t ( s e t s i z e d o u b l e a t each i t e r a t i o n )$ a l l . e a c h s l i c e p o w e r 2 do | n |#s k i p f i r s t i t e r a t i o nn e x t i f n . r e s o u r c e s . l e n g t h == 1

#copy f i l e on s e n d e rcopy n . n o d e f i l e , $ a l l . f i r s t#i t e r a t e on f i l e s i z e[ 1 0 2 4 , 2 0 4 8 , 4 0 9 6 , 8 1 9 2 ] . each do | s i z e |1 0 . t i m e s do | i | #f o r s t a t i s t i c

id , r=t a s k $ a l l . f i r s t , ” k a s t a f i o r #{n . n o d e f i l e } f i l e #{ s i z e }”p u t s ”#{n . r e s o u r c e s . l e n g t h } #{ s i z e } #{r . d u r a t i o n }”

endend



Results

100

1000

10000

100000

0 20 40 60 80 100 120 140

tem

ps d

e di

ffusi

on (

en s

ecs)

nombre de noeuds

Kastafior sur la grappe gdx (moyenne et ecart-type sur 10 tests consecutifs)

2KB4KB8KB

16KB32KB64KB

128KB256KB512KB

1024KB2048KB4096KB8192KB

16384KB32768KB65536KB

131072KB262144KB524288KB

1048576KB

2 parameters (node, file size), 10 iteration by point, 1400 measures, 25LOC for experiment description



Conclusion

Large experiments are difficult to conduct

New tools required

Expo : toward a Framework to conduct and control experiments

eDSL approach + versatile parallel launcher

Futur Works

Set of Resources Enhancements

Measures and experiments stokage and manipulation

Beta Release (http ://expo.imag.fr)

Test on more platform (PlanetLab, DSLLAB, Emulab, EGEE ?)

GUI (to launch and monitor experiment)

More experiment examples



Expo : GUI


Conclusion

Plateformes pour l’experimentation

Multiples

PlanetLab, OneLab2, Emulab, GENI

DSL-Lab, SensLab...

Complexes

4 a 5 ans pour construire la 1er version (pas trop experimentale) deGrid’5000 (plus recodage)

Construire des outils pour l’experimentation ce n’est pas rentable, pasreconnu et pas compris

Former les jeunes (stagiaires, doctorant) c’est primordial, c’est euxqui font les courbes, les expes. Enseignements M2R ? ? ?


Conclusion

Plateformes pour l’experimentation

Defis

Les outils avances

Archivage d’experience (experience facilement/vraimentreproductibles )

Passerelles entre les plateformes

Couplage : simulateurs, injecteurs

Liens : modelisation, simulation, evaluation


Documents

et quelques outils Olivier Richard (MdC Polytech’)mescal.imag.fr/seminars/matinee_thematique/08_04_10/richard.pdf · Di cile a utiliser pour faire des l’ eval de perf a grande