Opensource APM SCOUTER in practice

  • View
    779

  • Download
    4

Embed Size (px)

Text of Opensource APM SCOUTER in practice

  • gunlee01@gmail.comSCOUTERcommiter

    Gunhee Lee

    OpenSourceAPMSCOUTER

    Basics&inPractice

  • .

    LGCNSsoftwarearchitectLGCNSAPMScoutercommiterKOSSA

  • UXMNMS

    EMS

    SMS

    Profiler

    APMDatabaseMon.

    Intranet&Security

    Gateway&FirewallBrowser

    Network

    WebServerWebApp.

    ExternalOrg.

    SSO

    DB

    Systemperformancemonitoring

  • WhyAPM?

    ?

    Systemperformancemonitoring

  • Application!

    Systemperformancemonitoring

  • - -

    Systemperformancemonitoring

  • .

    - By -

    Systemperformancemonitoring

  • ~ .

    - ByScouter -

    (SCOUTER )

    Systemperformancemonitoring

  • ScouterBasics

  • APM

    - Applicationperformancemanagement

    SCOUTER?

  • ? Agent,Collector,ClientWAS

    Java Agent

    WEBapplication

    HostAgent

    ScouterCollectorServer

    ScouterClient

    SCOUTER?

  • SCOUTER?

  • Object

    Alert

    ActiveService

    CPU

    Visitor

    Throughput

    Elapsed

    Heap

    XLOG

    ActiveserviceTop

    Todaycallcount

    Activespeed

  • (Java)ApplicationProblems

    UnhandledException

    Slowquery/Slowexternalcall

    Threadhang

    Inefficientlogic

    ObjectLeak/Unclosedresource

    OutofmemorySlowtransaction

  • 1.Slowtransaction

    Logic

    Logic

    VerylongtimeDatabaseQuery

    Request

    Response

  • 1.Slowtransaction

    ActiveService

  • 1.Slowtransaction

    ActiveService

  • 1.Slowtransaction

    XLOG

  • 1.Slowtransaction

    Profile

  • 2.Threadhang

    Requests Threads

    Locked

    Veryveryveryveryslow

    Blocked

    Veryveryslow

  • 2.Threadhang

    Objectrequest>ThreadList

    Objectrequest>ThreadDump

  • 2.Threadhang

    ThreadList

  • 2.Threadhang

    ThreadDump

  • 3.Outofmemory

    Memoryleak

    ( Bigfile)

  • 3.Outofmemory

    HeapusedObjectreq.>HeapHistogramObjectreq.>HeapdumpToomanyrecordexception

    -->10,000

  • 4.Resourceleak

    GetConnection

    ConnectionPool

    Releaseconnection

    excecuteexcecuteexcecute

    Missing!

  • 4.Resourceleak

    ConnectionleakStatementleakResulteSet leak

    Hang!OOM!

  • 4.Resourceleak

    SCOUTER Options

  • 5.InefficientLogic/Logicoutsideofapp.

    lightbuttoomanyexecutionsevillibrary/framework

    checkLogging() 1ms

    10s checkLogging() 1ms

    checkLogging() 1ms

    Biz()

    Biz()

    10,000timescall

  • 5.InefficientLogic/Logicoutsideofapp.

    SFA(Stackfrequencyanalyzer)RequestProcessing

    ThreadDump

    5sec 5sec 5sec

    Topstackfrequency Top-downanalysis

  • - SOA- MSA

    6.SCOUTERforservicearchitecture

  • 6.SCOUTERforservicearchitecture

  • .

    6.SCOUTERforservicearchitecture

  • .

    Cascadingfailure

    6.SCOUTERforservicearchitecture

  • Cascadingfailure

    Ref. https://github.com/Netflix/Hystrix/wiki

    6.SCOUTERforservicearchitecture

  • Cascadingfailure

    Ref. https://github.com/Netflix/Hystrix/wiki

    6.SCOUTERforservicearchitecture

  • Cascadingfailure

    Ref. https://github.com/Netflix/Hystrix/wiki

    6.SCOUTERforservicearchitecture

  • 6.SCOUTERforservicearchitecture

  • ?

    6.SCOUTERforservicearchitecture

  • Anti-fragilepatterns CrossServiceTracing

    6.SCOUTERforservicearchitecture

  • Crossservicetracing6.SCOUTERforservicearchitecture

  • Demo#1Q&A#1

  • ScouterInPractice

  • SimpleScriptingPluginBuilt-inplugin

    ScouterPlugin

  • 1)AgentPlugin

    2) CollectorserverPlugin

    ScriptingPlugin

    HttpServicepluginHttpCallPluginCapture(parameter,returnvalue)

  • AgentPlugin ID

    JavaAgent Httpserviceplugin

    1. useridStringuserId =$req.getSessionAttribute(userId);

  • JavaAgent Httpserviceplugin

    2.XLog id $ctx.login(userId);

    AgentPlugin ID

  • JavaAgent Httpserviceplugin

    3.Profile $ctx.profile(LoginID=+userId);

    AgentPlugin ID

  • $ctx.login(userId);

    $ctx.profile(###Itsuserdefined+userId);

    AgentPlugin ID

    !

  • AgentPlugin Parameterdebugging

  • AgentPlugin Parameterdebugging

    A

  • AgentPlugin Parameterdebugging

    Captureplugin+hook_args_patterns+redefineclass

  • AgentPlugin Parameterdebugging

    !

  • FrontService1)ID Backend2)Service ID .BackendService 3) ID . 4) .

    Plugin

  • Plugin

    Service(Back)ScouterAgent

    Service(Back)ScouterAgent

    Service(Front)ScouterAgent

    Service(Back)ScouterAgent

    1) Req. ID--

    2) HttpCall ID

    3) Http ID--

    ScouterCollector

    ApacheKafka

    Legacy

    4) ID Kafka

    StOrm InfluxDB

    RDB

    Hbase

    LegacyDashboard

    Plugin

  • Scouter-Alert-Telegram

    Built-inPlugin

    Scouter-Influxdb

  • Scouter-Alert-Telegram

    https://github.com/scouter-project/scouter-plugin-server-alert-telegram

    TelegramBot Alert

  • Scouter-Alert-Telegram

  • Influxdb-plugin

    https://github.com/scouter-project/scouter-plugin-server-influxdb

    Influxdb

  • Influxdb-plugin

    Influxdb database

    Grafana

  • Influxdb-plugin

    Scouter->InfluxDB ->Grafana url

    https://goo.gl/fSd8Dd

  • Influxdb-plugin

    Demohttp://demo.scouterapm.com:3000

  • Influxdb-plugin

    Demohttp://demo.scouterapm.com:3000

  • Built-InPlugin

    1. scouter.plugin.server 2. Methodannotation3.jar ./lib .

    *Dependency- scouter.common- scouter.server

  • Built-InPlugin

  • Built-InPlugin

    Guidehttps://github.com/scouter-

    project/scouter/blob/master/scouter.document/main/Plugin-Guide.md

  • ?

  • ApplicationApplicationTuningPlugin- debugging, - ,

    !

  • ..Plugin Centralizedlogging/monitoring

    Application

    SCOUTER

  • slack-Pluginkafka-pluginelasticsearch-Plugin

    Redis-Agent

  • Nodejs agent

  • SCOUTER

    node.js(w/express.js)ApacheHTTPDRedis /memcachedNoSQL(MongoDB)Opensource RDB(Maria,Cubrid)Clientsidemonitoring(scripterror,dom rendering)

  • Contribution ~

    Plugin Agent

  • SCOUTER

    Githubhttps://github.com/scouter-project/scouter

    Facebook https://www.facebook.com/groups/scouterapm

  • Q&AorLater

    gunlee01@gmail.com