© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 11
Writing Apache 2.0 Modules andporting 1.3 modules to 2.0
Dirk-Willem van GulikVP of Research, Covalent Technologies
O'Reilly Open Source Conference 2002v1.09
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 22
Overview
What are modules– Apache fundamentals
Module anatomy 2.0 - what is new 1.3 -> 2.0 checklist Writing an authentication module (PAM) Sample 1.3 -> 2.0 Migration Acronym Module - filter sample (time permitting) Q&A
http://www.apache.org/~dirkx/oscon2002/
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 33
Fundamentals
Apache webserver– NCSA roots; The 'Shambala' rewrite– Modular architecture
Other Apache projects– Apr, Apr-utils, Proxy, docs– Tomcat,– Xml, java, Tcl, …
http://www.apache.org/~dirkx/oscon2002/
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 44
History
Apache 0.9 Apache 1.2 Apache 1.3.29
shambalaApache 2.0
APRAPR Utils
Apache 2.0New Proxy
patches
ncsa
….
modulesmodules
?
PerlPHPJava
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 55
Apache: Basic facts
Webserver is written in ‘C’ Runs on virtually all platforms
– Unix, QNX, BeOS, Win, OS/390 Very open modular infrastructure
– Sequence of handlers act on each stage of the (http) request
Server core handles ‘child care’ and ensures protocol compliance.
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 66
Changes in 2.0
Protocol engine abstracted out– http, https– POP3, Commercial FTP module
Filters and daisy chaining now possible
Abstracted out ‘child’ management– (pre)forked, threaded– Or a hybrid
More infrastructure for portability
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 77
Modules tasks:
Modules are small and simple:– Auth: verify a username and
password– Access: verify if a username is on a list
or an IP address matches– Rewrite: change a URL– Logging: log to a file or backend– Headers: Modify or add a header– Content: Simple content substitution
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 88
When - or when NOT
Simple specific task
Task maps to a single handler
Relatively light weight task for each request
Security close to the wire needed
Complex interaction with other sources required
Complex to secure.
Complex rules
Large footprint
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 99
Module Environment
Modules have– One known entry point– Set of registered callbacks– Set of known directives it can handle
Modules live a shielded live– Memory management– Log files, UID, socket access– Private memory for
– Configuration information– Per request data
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 1010
Anatomy - the short version
At startup– Load the modules– Set up basic infrastructure (logs, uids)– Read the configuration file
– Hand each module its registered configuration directives
– Ask each module to initialize Runtime
– Wait for requests to process– (Re)Initialize new children
Shutdown– Ask each module to shut down
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 1111
Anatomy - Requests
Request comes in– File name translation– Check user id– Check access permissions– Check type– Last change fixups– Log when all is done
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 1212
The API
Module block– Master router
Command block– Registry of configuration commands
Hooks– Registry of functions called at each stage
Handlers– Registry of functions called on condition
APR, APR utils– The backside; system abstraction
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 1313
Configuration file
Apache configuration file– Httpd.conf, srm.conf, access.conf– .htaccess– Include 'file.conf'
Three types of directives– Processing and core
– Includes, directory– UID, LoadModule, errorlog– Hierarchy
– Module specific– Everything else
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 1414
Anatomy - Configuration
Configuration file is read– Hierarchy is build
– <Directory, <Server, ..– Modules are instructed to create/merge
matching configurations– Server– Directory
Directive is read– Registered module is called with
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 1515
Anatomy - your inputs
Pointer to your private memory segment.
Specific Arguments for– Parsed Directive details– Request you are expected to handle– I/O and Log streams
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 1616
Anatomy - your outputs
Very few– DECLINED: 'aint me gov'– OK: 'Yup - I've done my job'– SERVER_ERROR: 'Scream murder'– Protocol/http status codes
But..– You can modify data in the structures
passed on– url, mime type, headers
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 1717
Modules: 1.3 versus 2.0
No fundamental Changes
More APR and APR utils support– Module authors can rely on a portability layer;
existing code often reduced or simplified significantly.
Extra features– Filters; layered IO– More hooks, fewer pointer structs
Process management– Threaded, Process or hybrid
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 1818
2.0 Checklist
No global variables– Your environment may be Threading– Use thread-safe libraries
Use Request record for passing state– No assumptions as to which child,
process or thread will handle successive requests (from the client) or phase (within a request).
Potential longer lifetimes– Do your cleanups.
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 1919
Examples
Mod_auth_pam– Popular existing 1.3 module– Authenticates against PAM
– Pluggable authentication layer– Solaris, FreeBSD, Linux, others– /etc/passwd, SecurID, ldap, others
Mod_acronym (time permitting)– Contrived example– Uses filters
http://www.apache.org/~dirkx/oscon2002/
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 2020
Writing the module
Don’t write modules !– You sure it is not already in the base
install ? Don't write modules !!
– You sure there is not a module already on modules.apache.org ?
Don’t' write modules !!!– There must be a module already which
comes close and which you can simply adapt to suit your purpose.
http://www.apache.org/~dirkx/oscon2002/
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 2121
Essentials
ANSI -C
(binary) version of apache
C compiler (gcc), make, linker
Documentation and Samples:– http://dev.apache.org/apidoc/– http://modules.apache.org/
http://www.apache.org/~dirkx/oscon2002/
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 2222
Initial Setup
Fetch a copy of apache– http://httpd.apache.org/dist
Compile and install– ./configure --enable-maintainer-mode– make && make install
And build your own module– cd ~/myModule– apxs -c mod_auth_pam2.c
http://www.apache.org/~dirkx/oscon2002/
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 2323
Initial testing
Install the module– apxs -I mod_auth_pam2.c– Check httpd.conf
– Add directives as needed (Re)start the server
– apachectl stop– apachectl start
Always keep an eye on the log !– tail -f ../logs/errorlog
http://www.apache.org/~dirkx/oscon2002/
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 2424
Common gotcha's
The 'apxs' in your path is -not- the one of the installed apache
'--enable-maintainer-mode' or -O3, -Wall not present.
You should restart your server each time. gdb ./httpd
– run -X (1.2.x and above)– run -D ONE_PROCESS (2.0.x)– Use forking MPM if possible
http://www.apache.org/~dirkx/oscon2002/
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 2525
Mod PAM - Requirements
Requirements:– Use 'Basic Auth'– Authenticate a user against PAM– Access control on username– Access control on group membership
PAM API provides the basics– Pam_start(), Pam_authenticate()..– In: Username, password– Out: valid, has account– Thread safe.
http://www.apache.org/~dirkx/oscon2002/
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 2626
MaP: Outline
Configuration:– Accept Configuration directives– Store our configuration
Authenticate a username and a password– Respond with a yes/no/duh– Flag errors as appropriate.
http://www.apache.org/~dirkx/oscon2002/
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 2727
MaP: Headers
#include "ap_config.h"
#include "httpd.h"
#include "http_config.h"
#include "http_core.h"
#include "http_log.h"
#include "http_protocol.h"
#include "http_request.h"
#include <security/pam_appl.h>
@144
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 2828
MaP: Configuration.
Pick your directives with care– Auth: we authenticate– PAM: against the PAM system
AuthPAM_Enabled <yes|no>– General on/off switch
AuthPAM_FailDelay <int - mSec>– PAM specific timeout
AuthPAM_FallThrough <yes|no>– Am I authoritative ?
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 2929
MaP: configuration (cont)
Your Private Configuration
typedef struct {
int fail_delay;
int fall_through;
int enabled;
} auth_pam_dir_config;
@191
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 3030
MaP: Initialize
Few modules actually need this.
Note: static and different pools.
static int auth_pam_init(
apr_pool_t *p, *plog,*ptemp,
server_rec *s
) {
ap_log_error(APLOG_MARK, APLOG_INFO, 0, s,"PAM: mod_auth_pam/" VERSION);
return OK;
}
@201
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 3131
MaP: Real initialize 1/4
Happens piece by piece– Within the hierarchy– Passed the correct pool– Return is opaque pointer.
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 3232
MaP: Real Initialize 2/4
static
void* create_auth_pam_dir_config(apr_pool_t *p, char *dummy)
{
auth_pam_dir_config *new =
(auth_pam_dir_config*) apr_palloc (p,
sizeof(auth_pam_dir_config));
new->fail_delay = 0; /* 0 ms */
new->fall_through = 0; /* off */
new->enabled = 1; /* on */
return new;
}
@214
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 3333
MaP: Real initialize 3/4
Create your own memory segment.
Use the right pool!
Do not even think to use malloc()
auth_pam_dir_config *new =
(auth_pam_dir_config*) apr_palloc (p, sizeof(auth_pam_dir_config));
@217
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 3434
MaP: Real initialize 4/4
Always initialize– -O5– predictable stale pools – Security assumptions.
new->fail_delay = 0; /* 0 ms */
new->fall_through = 0; /* off */
new->enabled = 1; /* on */
And return the initialized memory segment:
return new;
@220
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 3535
MaP: Populating 1/4
static command_rec auth_pam_cmds[] =
{
AP_INIT_TAKE1("AuthPAM_FailDelay", …
AP_INIT_FLAG("AuthPAM_FallThrough", ..
AP_INIT_FLAG("AuthPAM_Enabled", ..
{NULL}
}
@245
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 3636
MaP: Populating 2/4
Calls 'parse_function(args)' when directive is seen.
Lots of pre defined parse functions Be sensibly strict Always tell the default.
AP_INIT_TAKE1("MyDirective",
parse_function, extra_argument,
allowed,
"helptext shown with httpd -V" ),
@226
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 3737
MaP: Populating 3/4
AP_INIT_TAKE1("AuthPAM_FailDelay",
ap_set_int_slot,
(void *)APR_OFFSETOF(
auth_pam_dir_config, fail_delay),
OR_AUTHCFG,
"number of micro seconds to wait after failed authentication attempt. (default is 0.)"
),
@247
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 3838
MaP: Populating: 4/4
AP_INIT_FLAG("AuthPAM_Enabled",
ap_set_flag_slot,
(void *)APR_OFFSETOF(auth_pam_dir_config, enabled),
OR_AUTHCFG,
"on|off - determines if PAM authentication is enabled; "
"(default is on.)"
),
@259
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 3939
MaP: recap
Configuration:
<directory /www/internaldocs>AuthType BasicAuthName "Your Intranet
passwd"AuthPAM_Enable yesrequire valid-user
<directory>
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 4040
MaP: Check User 1/3
On entry– We get our configuration back.– The protocol (http) and core can supply more
information.
static int pam_auth_basic_user (request_rec *r)
{auth_pam_dir_config *conf =
ap_get_module_config(r->per_dir_config,&pam_auth_module);
@341
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 4141
MaP: Check User 2/3
We're always called. Make sure we are configured (This is why initialization was so important):if (!conf->enabled)
return DECLINED;
And find out if a username and password was supplied: if ((res = ap_get_basic_auth_pw (r,
(const char**)&(userinfo.pw))))
return res;
userinfo.name = r->user;
@357
@361
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 4242
MaP: recap
At this point– We have access to the right
configuration at the right place in the configuration hierarchy.
– The username and password are available to us.
– As is everything else in 'r'.
Decision time !
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 4343
MaP: Check User 3/3
Log!
ap_log_rerror(APLOG_MARK, APLOG_ERR, 0, r,
"PAM: user '%s' - not authenticated: %s",r->user,compat_pam_strerror(pamh, res));
On Error
return HTTP_INTERNAL_SERVER_ERROR;
On success
return OK;
On unkown
if (fall_through)
return DECLINED;
Otherwise
ap_note_basic_auth_failure(r);
return HTTP_UNAUTHORIZED;
@374
@377
@423
@404
@410
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 4444
MaP: Wiring it all up 1/3
Advertising– Our configuration entry points– Our initialization callbacks– Registration of our hooks
– Post_config (init message)– Check_user (verification)A
Apache 1.3/2.0 - biggest difference.
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 4545
MaP: Wiring it all up 2/3
Apache 1.3 /2.0 quite similar
module AP_MODULE_DECLARE_DATA
pam_auth_module = {
STANDARD20_MODULE_STUFF,
create_auth_pam_dir_config, /* dir config creater */
NULL, /* dir merger */
NULL, /* server config */
NULL, /* merge server config */
auth_pam_cmds, /* command table */
……
@517
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 4646
MaP: Wiring it all up 3a/3
The Apache 1.3 method ….
auth_pam_init, /* initialize */
….
NULL, /* handlers */
NULL, /* filename translation */
pam_auth_basic_user, /* check_user_id */
pam_check_auth, /* check auth */
NULL, /* check access */
NULL, /* type_checker */
NULL, /* fixups */
NULL /* logger */
};
@519
@525
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 4747
MaP: Wiring it all up 3b/3
OR The Apache 2.0 way:
…
pam_register_hooks, /* register hooks */
};
static void pam_register_hooks(apr_pool_t *p)
{
ap_hook_post_config(auth_pam_init,
NULL, NULL, APR_HOOK_MIDDLE);
ap_hook_check_user_id(pam_auth_basic_user,
NULL,NULL,APR_HOOK_MIDDLE);
}
@525
@511
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 4848
MaP: 1.3 versus 2.0 1/3
Easy– Thread safe, no global variables
#include "ap_config.h" (114) Closed security niggle in _init. (209) Removed 'legacy' (2.0: good excuse) Removed 'require' checking. (430). Changed: ap_palloc() -> apr_palloc(). All directive functions removed (226)
– Stock string/int/boolean parsers used. Rewrote command_rec with macro's. (245)
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 4949
MaP: 1.3 to 2.0 changes 2/3
server_rec; (336)– r->connection->user changed location to r-
>user Logging functions (374)
– 'r' variant (security)– PAM: prefic for clarity.– Use of log levels.
AUTH_REQUIRED became (419)– HTTP_UNAUTHORIZED
Module callback list shorter (511)– And callbacks actively registered instead.
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 5050
MaP: Conclusion
Very typical module
Change to 2.0– Made the module simpler.– Made it more portable– Was simple and we could borrow from
'modules/aaa/mod_auth_*.c
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 5151
Example deficiencies
PAM library assumed to be thread safe– If not - needs to be 'mutex'-ed.
Use of /etc/group rather than a AuthGroupFile– Use mod_access_etc_group.c
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 5252
Note:
APXS is not the only way to build a module. Do not use it if:– You need 'autotconf'.– Complex linking or dynamic loading.
Other methods– ./configure pointing and Config.m4– --with-module=type:name
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 5353
Fine !
Questions
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 5454
Acronym Expansion - filters
Filters– The true cool feature of 2.0– Daisy chain modules– On Input and Output
– Some header/body understanding– Not carved in stone
– Buckets and Brigades
http://www.apache.org/~dirkx/oscon2002/
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 5555
Acronyms
Outline and requirements– Add a footnote to each page containing
acronyms (e.g. ASF*).– Detects acronyms a˜utomatically– Be reasonable fast
Solution:– An Output filter.– Filter each APR‡ buckets and replace or
insert where needed
*) ASF: Apache Software Foundation
‡) APR: Apache Portable Runtime
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 5656
Rough Design
Configuration– Enable/Disable per directory/file or
similar resource blocks.– Standard unix dictionary format.
– E.g. /usr/share/misc/airports.txt– Not use too much memory.
– Needs to work on only the HTML files.
Sample– Lots of deficiencies.
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 5757
Outline 1/2
Configuration– Per directory– For the server as a whole
Read in a dictionary – And store in a hash
Plug the filter into the stream– When enabled and for html only
http://www.apache.org/~dirkx/oscon2002/
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 5858
Outline 2/2
Output Filter– Detect an acronym– Add a ‘dagger’ and a link.– Add some in situ javascript/ALT– At the bottom of the page;
– add the expansion.
http://www.apache.org/~dirkx/oscon2002/
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 5959
Bucket Brigades
Brigade
Bucket Bucket Bucket
Brigade
Bucket Bucket
EOS
EOS
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 6060
“Hello World of Apache”
Brigade
“Hell” “o Worl” “d ”
Brigade
“of Ap” “ache”
EOS
EOS
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 6161
Configuration
In memory structures
Create/Store in our private context.
Configure Directives
Read in the specified acronym file(s).
@103
@126
@492
@413
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 6262
Parsing (1/8)
Let’s ignore the header - focus on body.
File read or generated.– Verified to be HTML, enabled.– And chained into the stream output
stream set up to the client.
I am flying out of SFO to night.
@147
@168
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 6363
Parsing (2/8)
Passed in several brigades– Each brigade is a single call
I am flying out of SFO to night.
@171
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 6464
Parsing (3/8)
Passed in several brigades– Consisting in several buckets!
I am flying out of SFO to night.
@517
@517
@517
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 6565
Parsing (4/8)
Prepare a brigade with the output
First brigade can be parsed until the end
and passed down the chain.
I am flying out of SFO to night.
I am flying
@200
@201
@397
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 6666
Parsing (5/8)
Second brigade– Buckets parsed until we hit the acronym– Pass ‘S’ state on to the next brigade
I am flying out of SFO to night.
I am flying out ofS
@307
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 6767
Parsing (6/8)
Using ‘S’ passed– parse until end of Acronym
Add <html> around acronym.
I am flying out of SFO to night.
I am flying out of SFO‡SFO‡
@319
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 6868
Parsing (7/8)
And parse out the final part of the string
I am flying out of SFO to night.
I am flying out of SFO‡ To night.
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 6969
Parsing (8/8)
Once done - add the footnote with the acronyms.
I am flying out of SFO to night.
I am flying out of SFO‡ To night.
SFO: San Francisco International.
@352,208
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 7070
Demo
<h2><hr>Travel Plan<hr></h2>
Travel for next week:
<ul>
<li> Tuesday: From IAD to SFO
<li> Thursday: back from IAD to SLC
<lI> Sunday: out from SLC to OAK.
</ul>
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 7171
In the browser:
QuickTime™ and aTIFF (LZW) decompressorare needed to see this picture.
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 7272
Summary
Filtering module– Brigades with buckets– Be prepared– Keep state
Simple and Fast– One specific function
Imagination– Combine with anything apache does !
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 7373
Reminder
No need to write modules!– It is probably there in core.
No need to write modules!– modules.apache.org may have it.
No need to write modules– Something must be close enough to be
subverted for your purpose. But if you do
– Let the world know !
© 2001 Covalent Technologies – Commercial in Confidence – 29 January 2001 - 7474
Fine !
Questions
http://www.apache.org/~dirkx/oscon2002/
Top Related