60
05/19/2016 Etourneau Gwenn @The_shinji62 Concourse Overview

ConcourseCi overview

Embed Size (px)

Citation preview

Page 1: ConcourseCi  overview

05/19/2016 Etourneau Gwenn

@The_shinji62

Concourse Overview

Page 2: ConcourseCi  overview

About meEtourneau Gwenn

Sr Solution Architect

Bef.: Platform Architect

https://github.com/shinji62

https://twitter.com/the_shinji62

Page 3: ConcourseCi  overview

Pivotal

• Spring Framework • RabbitMQ • Concourse

• Cloud Foundry • Pivotal Tracker • Gemfire

Page 4: ConcourseCi  overview

Agenda

• Yet Another CI / CD • Architecture • Usage • Demo • Q&A

Page 5: ConcourseCi  overview

Yet Another CI / CD

Page 6: ConcourseCi  overview
Page 7: ConcourseCi  overview

But

Page 8: ConcourseCi  overview

Why not Jenkins ? (v1)

• Snowflakes • No strict isolation • Too much dependency • well java …. ( ) • V2 really ?! pipeline yes ! but Groovy ..

Page 9: ConcourseCi  overview

Why not Jenkins ? (v1)

• Snowflakes • No strict isolation • Too much dependency • well java …. • V2 really ?! pipeline yes ! but Groovy ..

Page 10: ConcourseCi  overview

Why not ?• No pipeline • Difficult to debug, commit to debug

• Difficult to use / configure .. • Learning curve is high • Difficult to see everything

Page 11: ConcourseCi  overview

WE ARE ENGINEER

Page 12: ConcourseCi  overview

Concourse• Pluggable Resource Interface • Running builds in Containers Natively • Zero Snowflake-able Configuration • Submitting builds from the local file system up to run in CI • Pipeline pipeline pipeline !!! • Reproducible

Page 14: ConcourseCi  overview

Concourse• Pluggable Resource Interface • Running builds in Containers Natively • Zero Snowflake-able Configuration • Submitting builds from the local file system up to run in CI • Pipeline pipeline pipeline !!! • Reproducible

Page 15: ConcourseCi  overview

Yes yaml ….

Concourse

Page 16: ConcourseCi  overview

Concourse

Page 17: ConcourseCi  overview

Download Yaml

Set-pipeline in concourse

Concourse

Page 18: ConcourseCi  overview

Concourse• Pluggable Resource Interface • Running builds in Containers Natively • Zero Snowflake-able Configuration • Submitting builds from the local file system up to run in CI • Pipeline pipeline pipeline !!! • Reproducible

Page 19: ConcourseCi  overview

Architecture

Page 20: ConcourseCi  overview
Page 21: ConcourseCi  overview

Architecture

ATC

TSA

Workers

Page 22: ConcourseCi  overview

ArchitectureATC Air Traffic Controller

• Brain • Web UI (ELM) • Responsible for Pipeline scheduling

ELM

Page 23: ConcourseCi  overview

ArchitectureTSA Travel Security Agency

• Register workers against ATC • Unregister workers against ATC

Page 24: ConcourseCi  overview

ArchitectureWorkers

• No State • Run the task • Container inside Garden container • Register themselves to the TSA

Page 25: ConcourseCi  overview

Architecture Workers

Garden

image_resource: type: docker-image source: repository: ubuntu/….

Garden

image_resource: type: docker-image source: repository: ubuntu/….

Garden

image_resource: type: docker-image source: repository: ubuntu/….

Garden

image_resource: type: docker-image source: repository: ubuntu/….

VM Operating System

Page 26: ConcourseCi  overview

Concourse• Pluggable Resource Interface • Running builds in Containers Natively • Zero Snowflake-able Configuration • Submitting builds from the local file system up to run in CI • Pipeline pipeline pipeline !!! • Reproducible

Page 27: ConcourseCi  overview

Concept

Page 28: ConcourseCi  overview

Three main concept

• Jobs • Resource } PIPELINE

• Task

Page 29: ConcourseCi  overview

Task• Smallest configurable unit in a Concourse pipeline • Always behave same ways if inputs are the same result

should be the same • Basically run things into containers • Input and output • Exit 0 ==> Success • Exit != 0 ==> failed

Page 30: ConcourseCi  overview

Task

platform

image_resource

inputs

run

Worker type

Docker image used to run the task

input used by the task

what to do

unit-testing.yml

Page 31: ConcourseCi  overview

Task unit-testing

Page 32: ConcourseCi  overview

Concourse• Pluggable Resource Interface • Running builds in Containers Natively • Zero Snowflake-able Configuration • Submitting builds from the local file system up to run in CI • Pipeline pipeline pipeline !!! • Reproducible

Page 33: ConcourseCi  overview

Pipeline

Page 34: ConcourseCi  overview

Pipeline

Master

Develop

Page 35: ConcourseCi  overview

Pipeline

Page 36: ConcourseCi  overview

Resource

Resource Resource Resource

Page 37: ConcourseCi  overview

Resource• Objects used for jobs in the pipeline. • Base on the resource type, ex : git, s3, github-release,

semver, slack…. even your own • As well as input or output

• Git • Git-release • Docker Image • S3 • Semver • Etc ..

• Slack • Twitter • Perforce • Bintray • Flowdoc • Etc …

Official Community

Page 38: ConcourseCi  overview

name

type

Source, etc…Name of resource

resource type in this case git resource

depend of the resource

Resource

Page 39: ConcourseCi  overview

Resource (own resource type)

Can import new resource type directly into the pipeline

Page 40: ConcourseCi  overview

Concourse• Pluggable Resource Interface • Running builds in Containers Natively • Zero Snowflake-able Configuration • Submitting builds from the local file system up to run in CI • Pipeline pipeline pipeline !!! • Reproducible

Page 41: ConcourseCi  overview

Job

Job Job

Job

Page 42: ConcourseCi  overview

Job• Determine “actions” of the pipeline • Plan, which resource, which task and in which order • Basically

• What I need to do ? ==> Task • What I need to use ? ==> Resource • What I need to produce ? ==> Resource

Page 43: ConcourseCi  overview

Job

name

public

serial

plan

Name of jobs

Visibility

input used by the task

builds_plan

pipeline.yml

Page 44: ConcourseCi  overview

Get

Job Build-plans

Input resource trigger the job new version of the resource

1 Job = 1 build-plan

TaskRun the task unit-testing

pipeline.yml

Page 45: ConcourseCi  overview

VM Operating System

Garden

Basic Rootfs base on busybox

git clone myrepos ./concourse-meetup-goserve-ci

Task unit testing . is mounted

./concourse-meetup-goserve-ci/ci/unit-testin/unit-testing.yml

Page 46: ConcourseCi  overview

Usage

Page 47: ConcourseCi  overview

InstallationVagrant for local deployment

Bosh for Cluster deployment, aws, azure, gce

vagrant init concourse/lite vagrant up

Configure your deployment file Deploy your cluster

docker, chef etc… you can use the concourse binary

Page 48: ConcourseCi  overview

InstallationConcourse Binary

You can use the concourse binary with your favorite tool. Or no tool at all

https://github.com/concourse/bin

Page 49: ConcourseCi  overview

First Step

$ fly -t blite sync

To be sure your client is on the same version of your Concourse deployment

sync

$ fly -t blite login --concourse-url https://ci.example.com

login and create target to be use for all command

login

Page 50: ConcourseCi  overview

One off-Task

$ fly -t blite execute -c my-task.yml --input my-input-res=.

Run one-off task from the local directoryexecute

I want to execute the task my-task.yml using my local directory as input my-input-res

Page 51: ConcourseCi  overview

Pipeline

$ fly -t blite set-pipeline -p pipeline-name -c ci/pipeline.yml --load-vars-from=/secret/credentials.yml

set-pipeline Set or update the pipeline

I want to setup the pipeline defined in ci/pipeline.yml and I will use the file credentials.ml for variable

Page 52: ConcourseCi  overview

Demo

Page 53: ConcourseCi  overview

One off-Task

$fly -t clocal execute -c ci/unit-testing/unit-testing.yml --input concourse-meetup-goserv-ci=.

Running unit-testing task

Page 54: ConcourseCi  overview

Pipeline

$ fly -t clocal set-pipeline -p -p concourse-meetup-c ci/pipeline.yml

set-pipeline Set or update the pipeline

Page 55: ConcourseCi  overview

Online Resource

Page 56: ConcourseCi  overview

Documentation

Tutorials• Official documentation http://concourse.ci

• S&B https://github.com/starkandwayne/concourse-tutorial • Flight-School http://concourse.ci/flight-school.html

Page 57: ConcourseCi  overview

Public Pipeline• Buildpack

• http://buildpacks-ci.cfapps.io/ • CF Release

• https://runtime.ci.cf-app.com/pipelines/cf-release • Bosh

• https://main.bosh-ci.cf-app.com/ • Concourse

• https://ci.concourse.ci/

Page 58: ConcourseCi  overview

Slack

http://concourseci.slack.com/

Page 59: ConcourseCi  overview

Q & A

Page 60: ConcourseCi  overview

Thank You !!