L E WYS : un Canevas Logiciel à Composants pour Construire des Applications de Supervision

LEWYS : un Canevas Logiciel Composants pour Construire des Applications de Supervision Emmanuel Cecchet*, Oussama Layada et Vivien Quma INRIA Rhne-Alpes, projet SARDES*Emic Networks

www.objectweb.org Journes Composants 2005 - * - 06/04/2005

Plan

Contexte et motivations

LeWYSArchitectureMise en uvre

Conclusion


ContexteApplications J2EE sur grappeBut : construire des systmes autonomesAjout/suppression dynamique de noeudsEquilibrage de chargeContrle dadmission, etc.Besoin : outils de supervision


MotivationsSystmes dobservation existantsAd-hoc : RUBiSSpcifiques une plateforme ou un domaine prcisPas rutilisables dans de nouveaux contextes

Gnriques : Supervision de ressources dans les grappes et grilles : Ganglia, NWS, JAMM, etc.Peu flexibles Nature des donnes collectesPropagation des donnes Traitement des donnesPas utilisables dans notre contexte


ObjectifsOutils de supervision multi-plateformes

FlexibleDploiement dynamique des entits dobservationConstruction des canaux de traitement et de propagation Mode danalyse des donnesEn ligne (console) Hors ligne (stockage)

Intrusivit limite

Conception base de composants: (re)configurabilit


Plan



Conclusion


Proposition : LeWYSCanevas logiciel composant pour la construction de systmes dobservation

Bibliothque de composants SondesCanaux vnementsObservateurs (consommateurs dvnements)

Implantation en FractalModle de composants hirarchiques et rflexifsOutils de dploiement : ADLOutils de contrle : fractalexplorer


ObservationSondes (probes) permettant lobservationDes ressources matrielles (CPU, mmoire, rseau, disque, etc.)Du systme (interruption, processus, etc.)Des applications: JMX, JVMPI, SNMP, etc.Etc. Kernel ProbeNetprobeCPUprobeDiskprobe JMX ProbeDiskprobeServeur MBeanNuds 1Nuds 2Nuds 3


DploiementMonitoring Pump sur chaque nudDploiement dynamique des sondes ncessairesGestion des abonnements aux probes Collecte et estampillage des observations

Monitoring PumpKernelprobeNetprobepumpthreadMonitoring PumpCPUprobeDiskprobepumpthreadMonitoring Pump JMX ProbeDiskprobepumpthread MBeanServerNuds 1Nuds 2Nuds 3


CommunicationsConstruction via des canaux DreamComposants de communication: marshallers, TCP Socket, etc. Composants de traitements: filtres, agrgateurs, etc.Composition dynamique selon les besoinsMonitoring PumppumpthreadMonitoring PumppumpthreadMonitoring Pump JMX ProbeDiskprobeMBeanserverDREAMDREAMDREAMpumpthreadDREAMAgrgationFiltragePrtraitementKernelprobeNetprobeCPUprobeDiskprobe


Observateurs (1)ObserverBoucle de contrleMonitoring Pump JMX ProbeDiskprobeMBeanserverDREAMDREAMDREAMpumpthreadDREAMStockageEquilibrage de chargeObserver


Observateurs (2)Cas particulier : Entrept (Monitoring Repository)Stockage des donnes pour analyse post-mortemParcours de lhistorique Corrlation entre vnements

Monitoring RepositoryQuerythreadsDREAMDREAMDREAMDREAMObserverObserverStoragethreadMonitoring DBEventsubscribeservice


Plan



Conclusion


Implmentation : sondesSondes matrielles: Windows, Linux, et SolarisSondes logicielles: JMX, (JVMPI, SNMP, etc.)Chaque sonde rifie diffrentes ressourcesCPU : nice, idle, user, kernel


Performances Sondes LinuxPentium IV 1.8GHz, 512 MB RAM, 40 GB IDE disk (6 partitions), Linux 2.4.20150s pour collecter toutes les ressources

SondeNombre de ressourcesTemps moyen pour collecter une ressourceTemps moyen pour collecter toutes les ressourcesCPU822.9s23.4sMemory1340.3s40.7sDisk6630.4s31.5sNetwork4825.3s27.8sKernel323.0s23.0s


Performances Sondes WindowsPentium IV 2GHz, 512 MB RAM, 40 GB IDE disk (2 partitions), Windows 200016,57ms pour collecter toutes les ressources


Implmentation : PompeComponentMonitoringPumpThreadProbeMonitoringPumpManagerChannelOutProbeManagerBindingControllerProbeProbeRepositoryProbeManagerOutputManagerComponentPullPushMultiplexerMonitoringMumpManagerOutputManagerTimeStampProbeCacheCachedProbeRMI


Implmentation : Canaux vnementsUtilisation de Dream

Dveloppement de filtresApproximation sous forme de fonction linaire par morceaux dune squence discrte de points (ti ,xi)Rduction des donnes transmises : Uniquement les segments successifs et non les points individuels> 90% de donnes filtres pour une prcision de 10% Overhead CPU quasi ngligeable < 0,01%


ConclusionLeWYSCanevas logiciel composants pour construire des systmes de supervisionSondes efficaces implantes en JavaCanaux de communication arbitraires construits avec DreamProjet ObjectWeb (http://lewys.objectweb.org)

Travaux futursDveloppement de sondes (JVMPI, SNMP)Intgration avec CLIFUtilisation pour la construction de boucles de contrle pour serveurs J2EE en grappeDveloppement dalgorithmes publish/subscribe adapts aux hypothses des clusters

Questions ?

Bonus slides


LeWYS design choicesComponent-based frameworkprobes, monitoring pump, event channelsprovides (re)configurability capabilities

Minimize intrusiveness on monitored nodes

No global clocktimestamp generated locally by pump

Information processing in DREAM channels


Centralized monitoring using a monitoring repository (2)Monitoring repositorystores monitoring informationservice to retrieve monitoring information

ProsDB allows for storing large amount of datapowerful queriescorrelate data from various probes at different locationsresynchronize clocksbrowsing history to diagnose failuresuse history for system provisioning

Consrequires a DB (heavy weight solution)


Windows hardware probes


JMX probescollect monitoring information about software applications running in J2EE environmentsclient-server architectureinstrumented applications (MBeans) JMX client generic JMX probeJMX client that accesses all MBeansstandard RMI connectorspecific probessubset of relevant MBeans


Cartography probeReify resources available in a Linux nodehardware: cpu, mem, disk, net, pci, software: rpm, kernelsnetwork connections

Reify network topologymatches switch/router information (SNMP) with node information


Linux vs WindowsWindows probes are less efficient that Linux onesJNI callsregistry accesssome Windows performance components requires a lot of processing and memorywhole data requires 95kB of memory


Related workWatchTowerWindows using PDH less efficientJMX monitoring serviceLess general (string, counter, gauge monitors)GangliaNo J2EE probes availableLess flexible communication channels (centralized)Not online-orientedXamplerComplementary


Cache filterif value is the same as the previous one, it is filteredprecision width is tunableprobe and observer must be aware of sampling interval


Linear filterdata points around a line segment


Swing filterdont use just the first 2 points to define the approximating linedynamically compute optimal orientation


Slide filterallow disconnected line segmentschooses optimal start point and line orientation


Filters overheadprocessing overhead between 0.001% and 0.007% of cpu timeslide filter is O(n) because it needs to keep data points

CacheLinearSwingSlides/data point1.492.203.367.21State complexityO(1)O(1)O(1)O(n)


Online filters performancePreliminary resultsUp to 99.75% of data points filtered10% precision 20% precisionCache4.25%3.44%Linear5.72%5.31%Non-optimized Swing1.00%0.40% Optimized Swing 0.88%0.39% Non-optimized Slide0.68%0.29%Optimized Slide0.55%0.24%

dominant cost = reading /procperform only once regardless of the number of resources to probememory cost more because it involves two different filesWindows est moins efficace que LinuxAppels JNIAccs au registreRequiert bcp de traitement et de mmoire (95kB)

Intrusivit limite : un seul thread dans la pompe dominant cost = reading /procperform only once regardless of the number of resources to probememory cost more because it involves two different filesdominant cost = reading /procperform only once regardless of the number of resources to probememory cost more because it involves two different files

Documents

L E WYS : un Canevas Logiciel à Composants pour Construire des Applications de Supervision