Fluentd Meetup 2016 - ServerEngine Integration & Windows support

download Fluentd Meetup 2016 - ServerEngine Integration & Windows support

If you can't read please download the document

  • date post

    09-Jan-2017
  • Category

    Software

  • view

    3.102
  • download

    1

Embed Size (px)

Transcript of Fluentd Meetup 2016 - ServerEngine Integration & Windows support

  • ServerEngine Integration & Windows support

    Ritta Narita

    Fluentd Meetup 2016

  • About me.

    Ritta Narita

    joined Treasure Data after internship in 2015

    2

    @naritta

    @narittan

  • Fluentd v0.14

    ServerEngine is integrated

  • Whats ServerEngine?

    A framework to implement robust multiprocess servers like Unicorn

    worker

    worker

    workersupervisor Server

    Dynamic reconfiglive restart

    Heartbeat via pipeauto restart

    multi process/thread

  • how to use ServerEngine

    workersupervisor Server

    module MyServer def before_run @sock = TCPServer.new(

    config[:bind], config[:port]) end

    attr_reader :sockend

    module MyWorker def run c = server.sock.accept c.write "Awesome work!" end

    def stop c.close endend

    1. write server module and worker module

  • how to use ServerEngine

    workersupervisor Server

    se = ServerEngine.create(MyServer, MyWorker, { daemonize: true, log: 'myserver.log', pid_path: 'myserver.pid', worker_type: 'process', workers: 4, bind: '0.0.0.0', port: 9071,})

    se.run

    2. write configuration and run

  • module MyWorker def spawn(process_manager) process_manager.spawn(env, config[:spawn_cmd]) endend

    ServerEngine.create(MyServer, MyWorker, { worker_type: spawn, spawn_cmd: cmd})

    ServerEngine.create(MyServer, MyWorker, { worker_type: thread'})

    workerServer

    workerServer

    workerServerspawn

    thread

    processServerEngine.create(MyServer, MyWorker, { worker_type: process'})

    worker type

  • Whats the advantages for integrating serverengine into fluentd?

  • serverengine advantages

    Auto restartlive restart

    socket managersignal handlerlog rotation

  • Auto restart live restart

    socket managersignal handlerlog rotation

    serverengine advantages

  • if worker died unexpectedly

    worker

    workersupervisor Server

    worker

  • if worker died unexpectedly

    worker

    worker

    workersupervisor Server

  • if worker died unexpectedly

    worker

    workersupervisor Server

    Its detected by heartbeat and server launch worker again.

    worker

  • Auto restartlive restart

    socket managersignal handlerlog rotation

    serverengine advantages

  • workersupervisor Server

    tcp tcp

    send SIGHUP

    live restart

  • supervisor Server

    worker

    tcp

    kill the worker, leaving servers condition

    live restart

  • worker

    supervisor Server

    worker

    tcp

    start new worker and read config again

    live restart

  • worker

    supervisor Server

    worker

    tcp

    tcpshare the TCP with living zero downtime

    live restart

  • Signal and RPC server

    api/config.reloadlive restart/config reload HUP

    /api/processes.interruptWorkers

    INT

    /api/processes.killWorkers

    /api/plugins.flushBuffers

    system config reload USR2

    TERM

    USR1flush buffers

    shutdown

  • workersupervisor Server

    you can reload system config without restart

    system config reload

    send USR2

    check mtime and previous load time,if there is new one, reload!

    serverengineconfig cash

    new config

  • Auto restartlive restart

    socket manager signal handlerlog rotation

    serverengine advantages

  • Socket Manager

    workerSocket

    Managerclient

    SocketManagerserver

    Server

    26

    socket listen requestfind or create listening socket

    share the socket FD(unix)/duplicated socket info(windows)

    UDS(unix)/TCP(windows)

    get socket and accept in worker

  • how to use Socket Manager

    workerSocket

    Managerclient

    SocketManagerserver

    Server

    27

    #generate path for UDS/TCP for server connection socket_manager_path = ServerEngine::SocketManager::Server.generate_path

    #open socket manager server socket_manager_server = ServerEngine::SocketManager::Server.open(socket_manager_path)

    In server

  • workerSocket

    Managerclient

    SocketManagerserver

    Server

    28

    #get socket manager client from socket manager path socket_manager = ServerEngine::SocketManager::Client.new(socket_manager_path)

    #get listening socket from socket manager lsock = socket_manager.listen_tcp(bind, port)

    In worker

    how to use Socket Manager

  • Whats the merit of Socket Manager?

    worker

    worker

    worker

    SocketManager

    client

    SocketManager

    client

    SocketManager

    client

    SocketManagerserver

    supervisor

    Server

    29we can use multicore easily in future

  • easy to use multicore power fully 30

    with SocketManagerwith in_multiprocess plugin

    type multiprocess cmdline -c /etc/td-agent/td-agent-child1.conf cmdline -c /etc/td-agent/td-agent-child2.conf

    #/etc/td-agent/td-agent-child1.conf

    type forward port 24224

    #/etc/td-agent/td-agent-child2.conf

    type forward port 24225

    type forward port 24224

    setting when using 2 core

  • Auto restartlive restart

    socket managersignal handler

    log rotation

    serverengine advantages

  • Queue based signal handling

    When user sent USR1 and QUIT

    signal handler thread

    userUSR1

    send USR1

  • Queue based signal handling

    When user sent USR1 and QUIT

    signal handler thread

    userUSR1

    USR1

  • Queue based signal handling

    When user sent USR1 and QUIT

    signal handler thread

    userQUIT

    USR1

    send QUIT

  • Queue based signal handling

    When user sent USR1 and QUIT

    signal handler thread

    user

    USR1

    QUIT

  • Queue based signal handling

    When user sent USR1 and QUIT

    signal handler thread

    user

    USR1

    QUIT

    run!

  • Queue based signal handling

    When user sent USR1 and QUIT

    signal handler thread

    user

    QUIT

  • Queue based signal handling

    When user sent USR1 and QUIT

    signal handler thread

    user

    QUIT run!

  • Auto restartlive restart

    socket managersignal handlerlog rotation

    serverengine advantages

  • support new features(These are already ported to ruby core.)

    multiprocess aware log rotationreopening of log file'trace' level, which is lower level than debug'

    ServerEngine.create(MyServer, MyWorker, { log: 'myserver.log', log_level: 'debug', log_rotate_age: 5, log_rotate_size: 1*1024*1024,})

  • summary

    windowssignal rpc handlerno socket FD and UDSfork based spawn based

    live & auto restartmulticore with socket managersophisticated signal handlereasier log rotation

  • Thank you!

  • appendix

  • unixfork based damonwindowsspawn based damon

    damonize

    fluentdserverengine

    set windows_daemon_cmdlinein configuration

    run windows_daemon_cmdlinewhen damonize

    windows_daemon_cmdlineruns daemon.rb

    run_server with servermodule and workermodule