Embed Size (px)
Citation preview
Qlik Consulting Services
Qlik Application Architectural Patterns
Sajid Shaikh – Financial Services Practice Lead
Feb 2018
2
• Qlik Application & Data Architectural Patterns
• 3-Tier Framework
• Representative Case Studies
• Emerging Technologies
• FS Consulting Services
Agenda
Data Integration Design PatternsHigh Level Data & Application Integration Design Patterns
Dat
a Pr
epar
atio
nQ
lik S
ense
App
Multi-Tenant Design Pattern-2ODAG Big Data Design Pattern-1
Big Data
Dai
ly L
oad
Pro
cess Selection App
Apply User FiltersApply Section
Access
Send Formatted
Constraint to QVD Layer
Refresh / ReloadDashboard
Send Data RequestReceive Data
Dataset
App
Inte
grat
ion
Lay
er
Access/Create Client Specific
FolderStructure
Create / Access Master Template
Apps
QRS – API Create /Replace
Customer App
QRS – APIReload / Refresh
Customer App
Self Service Design Pattern-3 ETL / ELT Design Pattern-4
RolesAnalyst• Viewer (Consumer)• Contributor• Developer• Team AdminDMAP• Content Admin
• Root Admin
Create Self Service
Dashboards
In memory Data Model
Data Model, Master Items, LibrariesExtensions, Templates
Dataset
Qlik Sense Incremental
Process
Qlik Sense Rip/Replace
Process
Reload QS in-memory
Dashbaord
Dai
ly L
oad
Pro
cess
XO
R
Direct Discovery Blend Live Data
- or – Intermittent
Refreshes
Extranet Customer Apps / Mashup Per
CustomerBDI (Big Data Index) Emerging Platform <H2 2018>
Leave data where it is – No in-memory loads
One Qlik Servers per Hadoop Node
3-Tier Framework Data Ingestion
Key Artifacts
▪ A Global Data Layer Framework
▪ 3-Tier Architecture: Extract Layer, Transformed and Associative Data Model
▪ Various connectors for Databases, Big Data, SFDC, SAP, CSV, etc.
▪ Data Governance guidelines & validations rules for data profiling and exception management
▪ Star Schema or Dimensional Data Model
Data Ingestion Best Practices
▪ Build Data Integration Framework for analytics Dashboard development and
▪ Extract, Transform load processes,
▪ Source data ingestions guidelines: Daily, Event or Scheduled, incremental, Full Rip & Replace, Blended in-memory and real-time (Direct Discovery), & ODAG (Big Data)
▪ Dashboards to Binary load final data model
EDW / Big Data
Global Data Layer
Dat
a In
tegr
atio
n L
ayer
Data Governance• Data Extraction• Transformation• Normalization• Association• Metrics
Preparation
Ops Systems
• Star Schema• SCD Dimensions• Generate QVDs,• QVX, CSV, etc
Facts
Measures
Product Customer
Calendar
Entity
Hierarchy
Source Data Extract Layer Transformation Layer Associative Data Model Dashboards
Files Data Refreshes• Time / Event Based• Intraday Refreshes• ODAG - On-Demand App• Custom Streaming APIs• BDI – Big Data Index
Risk / Analytics, Performance,
Compliance, etc. Dashboards
Business Benefits
Working with Qlik to simplify and enhance data management practices will help transform the usability and agility
of data. The following are just some of the core benefits delivered through the 3-Tier Framework;
✓ Greater trust and confidence in data quality through governed operational analytics
✓ The ability to validate data assumptions through analytics
✓ A single source of truth for analytical dashboard development
✓ Increased agility and speed when interpreting data and generating reports
VLDB Handling Big Data (ODAG)High level Design Approach for Large Datasets
Op
t-1
: Sel
ecti
on
Ap
p +
Do
c C
hai
n (
De
tail)
Op
t-2
: Seg
me
nta
tio
n b
y C
oun
try
Op
t-3
: OD
AG
Qlik
Sen
se
Phase
Data Mart
Dat
a In
tegr
atio
n La
yer
Detail Transaction Data for all Countries
Entitlements SA
AggrDim.QVD
Selection AppApply User Filters
Apply Section Access
Country DetailMex_YYYYMM.
QVD
Generate Dimensional Level Aggr for each Country
Detail App All Countries
Document Chain Detail
Data MartD
ata
Inte
grat
ion
Laye
r
Detail Transaction Data for all Countries
Section Access
AggrDim.QVD
Selection AppApply User Filters
Apply Section Access
Generate Dimensional
Level Aggr for each Country
Detail App a Specific Country
Document Chain Detail
Loop & Reduce by
Country
QVD Layer
QVD Layer
QVD Layer
Data Mart
Dat
a In
tegr
atio
n La
yer
Detail Transaction Data for all Countries
Section Access
AggrDim.QVD
Selection AppApply User Filters
Apply Section Access
Generate Dimensional
Level Aggr for each Country
Reload Country Data into Target
ODAG App
Send Formatted
Constraint to QVD Layer
QVD Layer
Country DetailMex_YYYYMM.
QVD
QVD Layer
Country DetailMex_YYYYMM.
QVD
QVD Layer
Pros (Design Approach) Cons
Available functionality in View System overhead
Separation of Apps/workload App per country
Targeted Performance Improvement non-consolidated Apps
View functionality only
Pros (Sense vs. View) Cons for View
Flexible User Driven Selection Complex ODAG Macro code
Can Pull data directly from Mart limited self-service
Improved Self Service Visualizations
Use Sense ODAG built in-functionality
Pros (Design Approach) Cons
Available functionality in View System overhead
Separation of Apps/workload App per country
View functionality only
Financial Analysis,
Ad hoc Data and
Report generations &
Big Data Analysis
ODAG Built-In Navigation
Qlik Sense
Server
Selections and
aggregations
Details
1. User logs into the Selection App which is
populated with dimensional data and high level
aggregated charts on a schedule. The user
has been granted the relevant security rules
2. User selects dimensional criteria in the
selection app from filter boxes/charts and the
map. After the governed limit is reached the
navigation button on the toolbar becomes
active with a green indicator. The user can
then choose to generate a new application by
dynamically reloading data from the source
databases or choose an existing app.
3. Analysis app updated in the chosen stream
with details of the energy consumption over
time.Other sources
Big Data
Source
ODBC or other
connectivity
Navigation Option
Securitised
Governed
Consumer/creator options
ODBC or other
connectivity
Qlik and Big Data
With distributed indexing and calculation, Qlik scales out
to make all the data available even for high data volumes
and velocity while leaving the source data in place
Aggregates to make selections Details for each selection
With On-Demand App Generation Qlik surfaces
aggregate data for a selection of a slice of detail
records to be loaded on-demand from the source system
Setup at Qonnections
• 8 x i3.xlarge AWS EC2 instances with 16 vCPUs
• Data distributed along 1424 workers in multiple
geographies
• Memory footprint about 22GB per EC2 instance
Full details for all records available all of the time
Now: ODAG Under development: BDI
Qlik Financial Services Consulting
Poor
AdoptionLimited
ROI
Poor
insightsToo reliant
on IT Too many
risks
(e.g. skills)
Why does Consulting help improve Customer Success?
Qlik Consulting Project Portfolio
Investment Banks, Commercial Banks, Wealth & Asset Mgt, Capital Markets, Fund Managers, Insurance
Insurance Claims tracking and cyber index tracking
Client Reporting exposure positions, and lot level detail reporting
Loan portfolio analysis, production vs. pipeline, Relationship Manager score card
Trade Reporting, FINRA OATS, TRACE, MiFID 2, Reg NMS, and Post Trade Analysis
Regulatory reporting, Basel 3, CCAR, Volcker Rule, PRA, FFIEC & FR Y8/9C for Fed Reporting
Global Institutional Sales Commissions & Revenue Reporting Dashboards and N Printing enablement
Fixed Income Analytics and revenue reporting including pricing and Duration based models comparisons
Wealth Management in-flows / outflows, Client & Product insight, NNA, CBV & FTE Analysis, P&L vs. Cost
Asset Management Benchmark & Portfolio Risk/Returns, Factor based smoothing, Morningstar peer group
Embedded analytics & Mashups OEM Loan Application (SaaS) provider in a Multi-tenant environment
Benchmark Index Plotter, TCA, Embedded Analytics, APIs, integration of legacy + new EDW data
Sales Pipeline & Production reporting, track coverage, contact, campaign, expenses, P&L & RWA
External Client Facing Reporting Dashboard. Supported Custody, FX, Account Services, etc.
Financial Analysts Reporting, a self-service guided Dashboard reporting for field reps
PEP (Politically Exposed People) tracking and KYC (Know Your Customer) analysis
Legacy Migration of OBIEE, Cognos, BO users to Qlik, SAP / BW Integration
Portfolio
Cloud
Qlik Project Methodology (QPM)
Services
Packages
Start-up
Packages
Advanced
Packages
Partner
Packages
Business
Analytical Services
Advisory
Services
Solution
Services
Benefits Realization
Center of
Excellence
Deployment
Services
Industry Solution
Design
Solution
Accelerators
Solution Workflow
Services
On-Premise Hybrid
Technical
Analytical Services
Platform &
Infrastructure
Services
Data
Services
Server Architecture
Services
Performance
Services
Data Integration &
Modeling
Data Quality &
Management
Security Set-upBig Data Services
Visual
Analytic
Services
Application
Development
Web & Mobile
Development
Advanced UI/UX
Design
13
Americas Tech Services Disciplines
Cloud
• Qlik Cloud
• AWS
• Private Cloud
Performance
• Scalability Testing
• Platform Tuning
• Automated Testing
Security & Integration
• Custom Security
• Enterprise Security Integration
Platform Development
• OEM Solutions
• Deployment Automation
• Platform Management
Architectural Patterns
• Hybrid Cloud Model
• Global Deployments
• Geo Analytics, Nprinting
Geo Analytics
• Geo Coding
• Geographic Analysis
• Geometric analysis
Nprinting
• Template Development
• Reporting on Demand
• Data Strategies for Reports
UX Design
• Information Design
• Visual Design
• Application Prototyping
Web Development
• Custom Web Application
• Advanced Visualizations
• Embedded Analytics
Predictive Analytics
• R, Python, MATLAB
• Server Side Extensions
• Machine Learning
Dev Methodology
• Standardization
• Leading Practices
• Frameworks (SEEQ)
Big Data
• On Demand App Gen
• App on Demand (Session)
• Direct Query
Governance
• Foundational Security Rules
• Data Governance
• Platform Governance
Data Services Disciplines
Visual Analytics Services Disciplines
Platform & Infrastructure Services Disciplines
14
Industry (Vertical) Trends
Manufacturing & High Tech
• Demand Forecasting
• Quality & Machine Breakdown
• Warranty Analytics
Healthcare
• Patient Admission Forecasting
• Readmission Analysis
• Clinical Outcomes
Communications
• Customer Churn
• Quality/Process Control
• Next Best Offer/Action
Financial Services
• Credit Risk Analysis
• Loan Default
• Fraud Detection
Public Sector
• Crime Prevention
• Transportation Optimization
• Recidivism
Retail & Consumer
• Market Basket Analysis• Retail Store Selection• Product Recommendation
Energy & Utilities
• Customer Churn
• Proactive Failure
• Peak Demand
Life Sciences
• Territory Alignment
• Campaign Management
• Drug Discovery & Effectiveness
Technical Architect DeveloperData Architect Project Manager
• Data Center Capacity
• Shared data layer
• Source connectivity
• Data certification
• Governance
• Wireframes
• Low Fidelity Prototyping
• Responsive Designs
• Graphic Assets and
Libraries
• User Interaction Design
• Platform and security
• Long range capacity
planning
• Performance and
Availability
• VM/Hardware
Management
• Data Extraction
• Complex Expressions
• Data Modeling
• Workflow
• App Certification
• Information Architecture
• Onsite or Near-shore
• Manages
development and
delivery of the
Solutions to agreed
scope, timelines and
budget
• First point of contact
UI / UX Designer
Qlik Consulting Roles
The following roles are potentially relevant to assist in achieving business objectives.
Qlik Advisory
• Deployment
Strategy and
Change
Management
• Benefits
Realization,
Business Case,
Adoption Plans
Engagement Options
Qlik Leads
Implementation
Collaboration
Enablement
Customer Leads
Coaching
• Jointly determine vision,
goals and success criteria
• Qlik defines infrastructure,
data & application foundation
• Qlik installs & configures Qlik
components
• Qlik lead agile development
of initial app(s)
• Co-development of
additional app(s)
• Qlik knowledge transfer and
enablement
• Joint enterprise scalability
planning
• Define an information
architecture plan
• Customer lead development,
with Qlik assistance
• Qlik provides architecture
and development assistance
• Collaborative planning for
QlikView competency center
• Qlik provides periodic app
and architecture reviews
Qlik Consulting’s world class team ensures rapid time to value and return
on investment, combined with knowledge transfer and enablement
Thank you
