Eduardo Silvaeduardo@treasuredata.com

@edsiper

Fluent Bit

September 08, 2015Treasure Hunter School

About MeEduardo Silva

● Github & Twitter @edsiper● Personal Blog http://edsiper.linuxchile.cl

Treasure Data

● Open Source Engineer● Fluentd / Fluent Bit http://github.com/fluent

Projects● Monkey HTTP Server http://monkey-project.com● Duda I/O http://duda.io

Embedded World

Embedded WorldIt's not new...

● Embedded systems have been around for decades: automotive, medical devices, cameras, etc.

● Hardware resources are very limited.

● Embedded is emerging within a new concept: IoT

Internet of Things

Internet of ThingsFacts

● IoT will grow to many billions of devices over the next decade.

● Now it's about device to device connectivity.

● Different frameworks and protocols are emerging.

● It needs Logging.

Internet of ThingsAlliances

Vendors formed alliances to join forces and develop generic software layers for their products:

Internet of ThingsSolutions provided

Alliance Framework

→

→

IoT and Big DataAnalytics

IoT requires a generic solution to collect events and data from different sources for further analysis.

Data can come from a specific framework, radio device, sensor and others. How do we collect and unify data properly ?

Internet of ThingsMarket needs

The IoT market is growing and the device to device connectivity is a must, for hence a strong and fair software platform is required for a reliable system.

IoT Software LayersInteroperability

Hardware vendors are aware that interoperability withthird party products is a success key. A generic software layer across products is the answer.

In a galaxy not so far away...

@fluentbit

Fluent Bit is an open sourcedata collector

It let's you collect data from IoT/Embedded devices and transport It to third party services.

Fluent BitTargets

● Services

● Sensors / Signals / Radios

● Operating System information

● Automotive / Telematics

Fluent BitRequirements

IoT and Embedded environment requires special handling, specifically on performance and resources:

● Lightweight

● Written in C Language

● Customizable, pluggable architecture

● Full integration with Fluentd

Fluent BitI/O

Fluent BitIntegration with Fluentd

Fluent BitDirect Output

Fluent Bitand Fluentd

Fluentd Fluent Bit

Scope Servers Embedded & IoT devices

Language C & Ruby C

Performance High Performance High Performance

Dependencies Built as a Ruby Gem, it requires a certain number of gems.

Zero dependencies, unless some special plugin requires them.

Plugins More than 300 plugins available

Less than 10 plugins available

Architecture

ArchitectureGlobal Overview

ArchitectureCore Engine

● Everything related to initialization.

● Interface plugins.

● Abstract network operations.

● Make the magic happens.

ArchitecturePlugins

● Abstraction level between the Engineand data handlers.

● Provide registration mechanism.

● Expose callbacks used in the Engineevent loop.

ArchitectureInput Plugins

● Collect data.

● Behave as a network service, built-inmetric or generator.

● It runs at intervals of time (triggeredby the Engine) or upon file descriptorevents.

ArchitectureInput Plugins (as of 0.2 dev)xbee get messages from XBee radio devices.

cpu metric plugin, gather and calculate CPU usage.

mem metric plugin, gather system memory usage.

kmsg get messages from the Linux Kernel Log.

serial get messages from the Serial input interface.

stdin get messages from the standard input interface.

mqtt MQTT TCP server, get publish messages.

ArchitectureOutput Plugins

● Take buffered data and enqueue it for delivery.

● Handle plugins

● Provides the main event loop.

●

ArchitectureOutput Plugins

fluentd Forward messages to Fluentd over TCP.

td Upload data to Treasure Data service.

stdout Print messages to the standard output.

ArchitectureStack Providers

Fluent Bit have three dependencieswhich are distributed in the source code and are linked statically.

These components are helpers for the Engine and Plugins Manager.

ArchitectureMessagePack

MessagePack is an efficient binary serialization format. This componentis heavily used to pack JSON into a binary version.

ArchitecturembedTLS

mbedTLS provides a Transport Layer Security (TLS) for network operations, for short, It provides TLSand SSL encryption for network operations that require secure channels.

ArchitectureMonkey

Monkey is a web server that providesan embeddable HTTP stack and someextra routines to handle configurationfiles, memory handling, event loops,string manipulation within others.

ArchitectureLibrary Mode

ArchitectureLibrary Mode

ArchitectureConfiguration Schema

Fluent BitDevelopment

Contributors Takeshi Hasegawa, Masaya Yamamoto, Naoya Kaneko within others

Fluent BitDocumentation

Fluent Bit

Thank you!

● http://fluentbit.io

● http://github.com/fluent/fluentd

● http://twitter.com/fluentbit