Luis & Yves
▪ Luís Marques
▪ Oracle ACE
▪ Lisbon
▪ http://lcmarques.com / @drune / [email protected]
▪ Yves Colin
▪ Oracle ACE Associate
▪ Paris
▪ http://ycolin.wordpress.com / @ycolin / [email protected]
ksplice - A piece of history (before Oracle)
▪ Created by Jeffrey Arnold, Tim Abbott, Waseem Daher, and Anders Kaseorg at MIT
▪ Released as open source (GPL v2 license) - Latest open source version: 0.9.9
▪ Ksplice (as company) started in 2008 and got 700 customers
▪ Support for RHEL, CentOS, Fedora, Debian, Ubuntu, CloudLinux and Scientific Linux
▪ Announced in LKML as “A system for rebootless kernel security updates” – (23 April 2008) (http://lwn.net/Articles/279378/)
ksplice - A piece of history (after Oracle)
▪ Oracle acquired Ksplice in 2011.
▪
▪ Offers feature only to Oracle Linux Premier Support customers
▪ Support for Oracle Linux (UEK, RH compatible kernels) via ULN – (also supports RHEL as legacy customers(?))
▪ Support for Ubuntu and Fedora “Desktop” versions as “free community protection”
▪ Strategic move for Oracle: mission critical systems run Linux (eg: Databases, Exadata, ODA)
ksplice – Why use it?
Allows to install security and reliability patches without reboot and without downtime
Downtime (and other) dilemmas:
▪ Huge security Linux kernel hole discovered and few Oracle single instance without DR?
▪ Factory Control Systems, Energy Management System Critical Applications
▪ Rollback an update without another emergency reboot?
▪ SLA or Security?
ksplice – Installation
Kernel Patching – Manual (Recommended)
# wget -N https://www.ksplice.com/uptrack/install-uptrack
# sh install-uptrack ACCESS_KEY
# uptrack-upgrade –y
Kernel Patching - Automatic Mode
# sh install-uptrack ACCESS_KEY –autoinstall
or edit /etc/uptrack/uptrack.conf and change autoinstall = yes
ksplice – Your first live demo with ksplice
Install and view all kernel updates available
# uptrack-show –-available # uptrack-show
# uptrack-install <ksplice_id> -y
# uptrack-upgrade -y
# uptrack-uname –a
ksplice – Your first live demo with ksplice
You can patch your system without any reboot but remove them rebootless as well
# uptrack-remove <ksplice_id>
# uptrack-remove –all
# uptrack-uname –a
ksplice and Oracle Database
▪ Ksplice doesn’t patch any Oracle shared libraries or binaries
▪ You want to patch your Database that is running of top of ext4 (?)
▪ Oracle instrumentation doesn’t catch it (?)
▪ Nothing in “Wait event” interface
▪ A few more cpu cycle are recorded during and after the patch, but is negligible.
ext4_file_open needs to be
patched but it is in use by
Oracle and your OS
The thread that can run the
function is suspended for
<1ms – No CPU can execute
it
Execution is resumed<1ms
Ksplice ,Exadata and AWS
▪ Email from Ksplice support about Exadata:
▪ Check against OVM on EXADATA version 12.1.2.2.1.160119
▪ Tested under Redhat 7.3 EC2 image
▪ Possible on AWS AMI customized XEN kernel (“Supported for legacy customers”)
Kplice – How patch is created? (1/2)
Source Code of Running Kernel
Slice
Source Code of Patch
Ksplice Build Environment:
• Pre Object Code
• Post Object Code Ksplice-id.koKsplice-id-vmlinux-
new.ko
Ksplice-id-vmlinux-
old.ko
Identify modified functions
Object Level Code difference
Patch Created
Slice
Kplice – How patch is created? (2/2)
▪ Ksplice compares code at object code level
▪ In this way Ksplice patch creation could be completely automated(in most of the cases)
▪ Ksplice compiles code(patched/non patched) with special gcc options -ffunction-sections -fdata-sections
Kplice – How patch is applied?
Connect to the
repository
Download the
Splice (tar.gz)
Files in
/var/cache/uptrack/
Linux/$arch/$kernel
Ksplice-id.ko core module
interface
(userland – kernel land)
Ksplice-id-vmlinux-new.ko Ksplice-id-vmlinux-old.ko
Interaction via
/sys/modules/ksplice-id/ksplice
Inspect thread – patched functions
in execution? (safety check)
Execute stop_machine() to
suspend thread
Patched code loaded in
memory
Patch applied
Thread
execution
resumed
Kplice – How patch is applied (in deep) ?
Compare Running Code in
memory to pre-code(safety check)
Resolve Undefined Symbol in Post
Code/Patched code (where are
they located in running kernel?) pre
matching phase
Find the patched function in
running kernel
Modified Old Function to
insert a JMP instruction
Patch
applied
stop_machine() : disable CPU
interruption and set the
patching process in High
priority scheduling mode
Patched Function
Check kernel stack thread
Ksplice - Alternatives
▪ Kpatch (RedHat) / kGraft (Suse):
▪ Uses ftrace to do the patching
▪ Kpatch uses stop_machine() to load new code (like Ksplice)
▪ Kgraft works on thread level, as soon as the threads reaches a point it switches to a new function
▪ KernelCare (CloudLinux)
▪ Proprietary software
Kpatch
▪ Kpatch > LivePatch (RedHat)
April 2015
Kernel 4.0
release –
Foundations for
LIVE PATCHING
May 2014
Submitted
Linux Kernel
Mainline
February 2014
Public release
GPLv2
June 2014
RHEL 7 – Preview
Canonical has a service called LivePatch that takes advantage of Kernel 4.x LIVE PATCHING
Ksplice – Web Management tool
▪ Webinterface available at: http://uptrack.ksplice.com or https://status-ksplice.oracle.com
▪
Ksplice – Web Management tool
▪ Webinterface available at: http://uptrack.ksplice.com or https://status-ksplice.oracle.com
Ksplice - Uptrack offline client
▪ Remove the need for servers to connect to “public” Ksplice server
▪ Create a local YUM mirror of Ksplice mirror and allow your servers to connect to it
▪ Or, Oracle bundles all available Ksplice patches for each supported kernel version to one RPM package and install in every server using a pen drive
▪ A new patch is available, a new RPM is available
▪ You cannot use the Web management tool.
▪ Details for offline client: https://docs.oracle.com/cd/E37670_01/E37355/html/ol_olcsetup_ksplice.html
Ksplice - Ksplice SNMP Plugin
▪ Use Oracle Enterprise Manager 12c/13c(?) to monitor status of Ksplice on your systems.
▪ Query the status of Ksplice using SNMP command snmpwalk:
[root@phoenix ksplice-46gkktym]# $ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceStatusKSPLICE-UPTRACK-MIB::kspliceStatus.0 = STRING: outofdate
▪ [root@phoenix ksplice-46gkktym]# $ snmpwalk -v 1 -c public -O e localhost KSPLICE-UPTRACK-MIB::kspliceAvailTable
▪ KSPLICE-UPTRACK-MIB::kspliceavailName.12 = STRING: [ur5of4nd] KSPLICE-UPTRACK-MIB::kspliceavailName.13 = STRING: [ue4dtk2k] KSPLICE-UPTRACK-MIB::kspliceavailName.14 = STRING: [wy52x339]
Details: http://docs.oracle.com/cd/E37670_01/E39380/html/ch01s11s01.html
Top Related