nginx_internals

8/3/2019 nginx_internals

1/61

Nginx Internals

Joshua Zhu

09/19/2009


2/61

Agenda

Source code layout Key concepts and infrastructure

The event-driven architecture HTTP request handling Mail proxying process

Nginx module development Misc. topics


3/61

Source Code Layout

Files $ find . -name "*\.[hc]" -print | wc l

234 $ ls src

core event http mail misc os

Lines of code

$ find . -name "*\.[hc]" -print | xargs wc -l | tail-n1110953 total


4/61

Code Organization

core/ The backbone and infrastructure

event/ The event-driven engine and modules

http/ The HTTP server and modules

mail/ The Mail proxy server and modules

misc/ C++ compatibility test and the Google perftools module

os/ OS dependent implementation files


5/61

Nginx Architecture

Non-blocking Event driven

Single threaded

[*]

One master process and several workerprocesses

Resource efficient Highly modular


6/61

The Big Picture


7/61

Agenda

Source code layout Key concepts and infrastructure The event-driven architecture HTTP request handling Mail proxying process



8/61

Memory Pool

Avoid memory fragmentation Avoid memory leak Allocation and deallocation can be very

fast Lifetime and pool size

Cycle Connection Request


9/61

Memory Pool (contd)

ngx_pool_t Small blocks Large blocks

Free chain list Cleanup handler list

API ngx_palloc

memory aligned ngx_pnalloc ngx_pcalloc


10/61

Memory Pool Example (1 Chunk)


11/61

Memory Pool Example (2 Chunks)


12/61

Buffer Management

Buffer Pointers

memory start/pos/last/end

file file_pos/file_last/file

Flags last_buf last_in_chain flush in_file memory


13/61

Buffer Management (contd)

Buffer chain Singly-linked list of buffers

Output chain Context

in/free/busy chains

Output filter

Chain writer Writer context


14/61

String Utilities

ngx_str_t data len sizeof() - 1

Memory related String formatting String comparison String search Base64 encoding/decoding

URI escaping/unescaping UTF-8 decoding String-to-number conversion


15/61

Data Structures

Abstract data types Array List Queue

Hash table Red black tree Radix tree

Characteristic

Set object values after added keep interfaces clean Chunked memory (part)

efficient


16/61

Logging

Error log Level Debug

Access log Multiple logs Log format

variables Per location

Rotation


17/61

Configuration File

Directive name type set

conf offset post

Parsing ngx_conf_parse

Values init merge


18/61

Configuration File (contd)

Block events http server upstream location if

Variables Buildins Other types

http_ sent_http_ upstream_http_ cookie_ arg_


19/61

Agenda




20/61

Master and Workers

Master Monitor workers, respawn when a worker dies Handle signals and notify workers

exit

reconfiguration update log rotation

Worker

Process client requests handle connections Get cmd from master


21/61

Master Process Cycle


22/61

Worker Process Cycle


23/61

Inter-process Communication

Signals Channel

socketpair

command Shared memory

Connection counter Stat Atomic & spinlock Mutex


24/61

Event

ngx_event_t Read Write Timeout

Callbacks Handlers

ngx_event_accept ngx_process_events_and_timers ngx_handle_read_event ngx_handle_write_event

Posted events Posted accept events queue Posted events queue


25/61

Time Cache

The overhead of gettimeofday() Time cache variables

ngx_cached_time ngx_current_msec Time strings

ngx_cached_err_log_time ngx_cached_http_time ngx_cached_http_log_time

Timer resolution Interval timer

setitimer()


26/61


27/61

Timer Management

Actions Add a timer Delete a timer Get the minimum timer

Red black tree[*]

O(log n) complexity


28/61

Accept Mutex

Thundering herd Serialize accept() Lock/unlock Listening sockets Delay


29/61

I/O

Multiplexing kqueue/epoll

NGX_USE_CLEAR_EVENT (edge triggered) select/poll/dev/poll

NGX_USE_LEVEL_EVENT (level triggered)

Advanced I/O

sendfile() writev() direct I/O

mmap() AIO TCP/IP options

TCP_CORK/TCP_NODELAY/TCP_DEFER_ACCEPT


30/61

Agenda




31/61

Important Structures

Connection ngx_connection_t

HTTP connection

ngx_http_connection_t HTTP request

ngx_http_request_t headers_in headers_out


32/61

Virtual Servers

Address Port Server names Core server conf


33/61

Locations

Location tree Static Regex

= ^~ ~ ~*

Per-location configuration Value

inheritance override

Handler Named location

try_files/post_action/error_page


34/61

HTTP Contexts

Types main_conf srv_conf loc_conf

Request ngx_http_get_module_main_conf ngx_http_get_module_srv_conf ngx_http_get_module_loc_conf

Parse conf file ngx_http_conf_get_module_main_conf ngx_http_conf_get_module_srv_conf ngx_http_conf_get_module_loc_conf

Module context ngx_http_get_module_ctx ngx_http_set_ctx


35/61

HTTP Handling

Receive data Parse the request Find the virtual server

Find the location Run phase handlers Generate the response Filter response headers Filter the response body Send out the output to the client


36/61

Request Parsing

Request line Headers Interesting tricks

Finite state machine ngx_strX_cmp


37/61

Phases and Handlers

Phases POST_READ SERVER_REWRITE FIND_CONFIG REWRITE POST_REWRITE PREACCESS ACCESS POST_ACCESS TRY_FILES CONTENT LOG

Phase handler Checker Handler Next


38/61

Phases and Handlers (contd)

Phase engine Handlers server_rewrite_index location_rewrite_index r->phase_handler

Default checkers ngx_http_core_generic_phase ngx_http_core_find_config_phase ngx_http_core_post_rewrite_phase

ngx_http_core_access_phase ngx_http_core_post_access_phase ngx_http_core_try_files_phase ngx_http_core_content_phase


39/61

Phases and Handlers (contd)

logLOG

autoindex, dav, gzip, index,random_index, static

CONTENT

access, auth_basicACCESS

limit_req, limit_zone, realipPREACCESS

rewriteREWRITE

rewriteSERVER_REWRITE

realipPOST_READ

modulesphase


40/61

Filter Chain

Singly-linked list like (CoR) Filter response only

Header filter Body filter

Send out the response ngx_http_send_header

top_header_filter ngx_http_output_filter

ngx_http_top_body_filter ngx_http_header_filter

ngx_http_copy_filter ngx_http_write_filter

Process order


41/61

Filter Chain Example


42/61

HTTP Handling Example

curl -i http://localhost/


43/61

HTTP Keep-Alive

Request memory reuse Connection memory shrink Keep-alive timeout Request count


44/61

Subrequest

Filters Addition filter SSI filter

Maximum subrequests


45/61

Internal Redirect

Return a different URL than originallyrequested

Examples try_files index/random_index post_action

send_error_page upstream_process_headers


46/61

Upstream

Hooks input_filter_init input_filter create_request reinit_request process_header abort_request finalize_request rewrite_redirect

Modules FastCGI Proxy Memcached

Event pipe Load balancer


47/61

Agenda

Source code layout Key concepts and infrastructure The event-driven architecture HTTP request handling Mail proxying process Nginx module development Misc. topics


48/61

Mail Proxy

Sequence diagram


49/61

Mail Proxy (contd)

Mail session Command parsing Packets relay

Things you can do Load balancing Authentication rewriting

Black lists/white lists


50/61

Agenda



51/61

General Module Interface

Context index & ctx_index

Directives Type

core/event/http/mail

Hooks init_master called at master process initialization

init_module called when the module is loaded

init_process called at worker process initialization

exit_process called at worker process termination

exit_master called at master process termination


52/61

Core Module Interface

Name Hooks

create_conf

init_conf Examples

Core Events Log HTTP


53/61

Event Module Interface

Name Hooks

create_conf init_conf event_actions

add del enable disable add_conn del_conn

process_changes process_events init done


54/61

Mail Module Interface

Protocol type init_session init_protocol parse_command auth_state

create_main_conf init_main_conf create_srv_conf merge_srv_conf


55/61

HTTP Module Interface

Hooks preconfiguration postconfiguration

create_main_conf init_main_conf create_srv_conf merge_srv_conf create_loc_conf merge_loc_conf


56/61

A Hello World HTTP Module

Creating a hello world! module Files

ngx_http_hello_module.c

config Build

./configure add-module=/path/to/hello/module

Configuration location & directive
http://blog.zhuzhaoyuan.com/2009/08/creating-a-hello-world-nginx-module/http://blog.zhuzhaoyuan.com/2009/08/creating-a-hello-world-nginx-module/


57/61

Agenda



58/61

Auto Scripts

Handle the differences OS Compiler

Data types Libraries

Module enable/disable

Modules order


59/61

Reconfiguration


60/61

Hot Code Swapping


61/61

Thank You!

My site: http://www.zhuzhaoyuan.com My blog: http://blog.zhuzhaoyuan.com
http://www.zhuzhaoyuan.com/http://blog.zhuzhaoyuan.com/http://blog.zhuzhaoyuan.com/http://blog.zhuzhaoyuan.com/http://www.zhuzhaoyuan.com/

nginx_internals

Documents

Transcript of nginx_internals