NetOps Coding 201 - NANOG Archive · NetOps Coding 201 auto remediation for your network! NANOG 66...
132
NetOps Coding 201 auto remediation for your network! NANOG 66 02.10.2016 david swafford
Transcript of NetOps Coding 201 - NANOG Archive · NetOps Coding 201 auto remediation for your network! NANOG 66...
NetOps Coding 201auto remediation for your network!
NANOG 66
02.10.2016
david swafford
continuing our theme
2
automating
3
the remediation of network faults
4
based on parsing of syslog messages
5
focusing today
6
on building the system
7
8
goals• organized to support 100s of
remediations
• fast enough to react to 100s of events at at time
• simple enough to manage without a CS background
how it works
9
10
device B
device A
Ethernet
Ethernet
11
device
device link down!X device B
device A
12
syslog server
sendsmessage
1
device
device X
%ETHPORT-5-IF_DOWN_LINK_FAILURE:InterfaceEthernet5/1isdown(Linkfailure)
13
syslog server
sendsmessage
parser
tails messages
1
2
device
device X
%ETHPORT-5-IF_DOWN_LINK_FAILURE:InterfaceEthernet5/1isdown(Linkfailure)
14
syslog server
sendsmessage
parser
tails messages
1
2
device
device X
regex convertsmessage into data
15
syslog server
sendsmessage
parser dbcreates
event
tails messages
1
23
device
device X
16
syslog server
sendsmessage
parser dbcreates
event
event processor
gets
event
tails messages
1
23 4
device
device X
17
syslog server
sendsmessage
parser dbcreates
event
event processor
gets
event
tails messages
1
23 4
remediation
runs5
device
device X
18
syslog server
sendsmessage
parser dbcreates
event
event processor
gets
event
tails messages
1
23 4
remediation
runs5
remediates6device
device X
19
syslog server
sendsmessage
parser dbcreates
event
event processor
gets
event
tails messages
1
23 4
remediation
runs5
remediates6
saves result
7
device
device X
remediates
20
syslog server
sendsmessage
parsercreates
event
tails messages
1
23
6device
device Xtoday's focus
dbevent
processorgets
event
4
remediation
runs5
saves result
7
demo!
21
22
syslog generator
23
syslog generator
24
syslog generator
25
syslog generator
26
syslog parser
27
syslog parser
28
syslog parser
29
event processor
30
event processor
31
event processor
32
event processor
33
event processor
the lab environment
34
a virtual machine of Ubuntu desktop
35
36
customized with
iPython! and extras
Python 2.7.11 & 3.5.1 MySQL server & Python libs
37
importing the virtual appliance
39
40
41
42
43
TERMINAL
log-in details
user:demopass:demo
44
keyboard shortcuts to break out of the VM
f enter or leave full-screen
getting started
45
organizing our data
46
47
the event as a tuple
48
the event as a tuple
49
what's the problem?
50
what's the problem?new field!
data formats will change
51
the event as a dictionary
52
the event as an object
link
53
accessing attributes of an object
link
which to choose?
54
55
the advantage of an object
enforcement of structure
56
the advantage of an object
input validation (not shown)
57
where to find it?
:Event()
• a database (MySQL)
• an events table
• a library for managing our data
• installation of "netfbar"(a Python package)
58
what's already staged?
59
database setup
60
netfbar package
the parser
61
62
where to find it?
remediations, refactored.
63
64
a few problems
• duplication
• no structure
• limited debug-ability
organizing our remediations
65
66
"sparse is better than dense"
1 remediation, 1 file.
...remediations, v2.
67
code that repeats in each remediation
68
moved to a base remediation
remediations, v2.
69
from base.BaseRemediation
remediations, v2.
connecting remediations to error codes
70
71a dictionary of "error_code" to "module"
connecting remediations to error codes
72
connecting remediations to error codes
73
where to find it?
questions?
74
building the event processor
75
starting simple
76
77
building it live
78
building it live
79
building it live
80
building it live
81
building it live
82
no events?
83
building it live
84
building it live
85
building it live
86
building it live
87
building it live
88
building it live
89
building it live
90
building it live
91
building it live
92
building it live
(no logging in the default remediation)
93
building it live
create a function for handling one event
94
building it live
iterating through events
95
building it live
so slow!
questions?
96
making this a little faster
97
• threading • multiprocessing • asyncio • gevent • go...
98
so many choices!
starting simple!
99
threading & multiprocessing
100
101
threading's pros
https://docs.python.org/3/library/threading.html
• simple API
• lightweight
• shared memory with the parent
102
threading's cons
https://docs.python.org/3/library/threading.html
• not true parallelism
• all threads are limited to a single CPU - the parent's
103
threads are evil! ....?!
https://docs.python.org/3/library/threading.html
"In CPython, due to the [GIL], only one thread can execute Python code at once"
...
"However, threading is still an appropriate model if you want to run multiple
I/O-bound tasks simultaneously."
104
multiprocessing's pros
https://docs.python.org/3/library/threading.html
• simple API
• true parallelism using all CPUs
105
multiprocessing's cons
https://docs.python.org/3/library/threading.html
• based on forking (cloning/copying the current process)
• heavier memory footprint
• stale memory / state
106
multiprocessing's cons
https://docs.python.org/3/library/threading.html
• no shared memory with parent
working with threads
107
108
working with threads
https://docs.python.org/3/library/threading.html
109
working with threads
https://docs.python.org/3/library/threading.html
target - the method to run inside the thread
110
working with threads
https://docs.python.org/3/library/threading.html
args - the values passed as input to process_event()
111
working with threads
https://docs.python.org/3/library/threading.html
.start() - spins up the thread in the background (the for loop continues)
112
working with threads
https://docs.python.org/3/library/threading.html
caution! this is dangerous! (starting an unknown number of threads)
113
adding visibility to process_event()
114
starting our threads again (with visibility)
https://docs.python.org/3/library/threading.html
115
starting our threads again (with visibility)
https://docs.python.org/3/library/threading.html
oh my! ...that just started 1020 threads!
creating a thread pool
116
117
creating a thread pool
the work to perform inside a thread
118
creating a thread pool
starting the background threads
119
creating a thread pool
looking at active threads
120
creating a thread pool
looking at active threads
121
creating a thread pool
fetching events that we'll send to the threads
122
creating a thread pool
123
creating a thread pool
https://docs.python.org/3/library/threading.html
creating a queue to pass events to threads
124
creating a thread pool
putting the pieces together
125
creating a thread pool
putting the pieces together
...
126
creating a thread pool
now, continuously!
127
creating a thread pool
questions?
128
closing out
129
130
syslog server
sendsmessage
parser dbcreates
event
event processor
gets
event
tails messages
1
23 4
remediation
runs5
remediates6
saves result
7
device
device X
131
goals• organized to support 100s of
remediations
• fast enough to react to 100s of events at at time
• simple enough to manage without a CS background
NANOG 66
02.10.2016
david swafford
#netengcode
