Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?
-
Upload
anne-nicolas -
Category
Software
-
view
2.259 -
download
0
Transcript of Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?
![Page 2: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/2.jpg)
3
Outline
• Basic architecture of Linux IO subsystem
• Tools
• Examples
![Page 3: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/3.jpg)
Linux Kernel IO Subsystem
![Page 4: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/4.jpg)
5
Linux Kernel IO Architecture
Task1 Task2 ... Userspace
KernelPage cache VFS
mmap
Driver
Filesystems
Block layer
Disk Hardware
Buffered IO
Direct IO, metadataBuffered data
![Page 5: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/5.jpg)
6
Linux Kernel IO Architecture
Task1 Task2 ... Userspace
KernelPage cache VFS
mmap
Driver
Filesystems
Block layer
Disk Hardware
Buffered IO
Direct IO, metadataBuffered data
mmap
![Page 6: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/6.jpg)
7
Block Layer Basics
• Works with IO requests– Starting sector, length, read / write / special
– Can have hints (SYNC) and other flags (FUA, FLUSH)
• Life of a request– Created in block layer when IO submitted by a filesystem
– Can be delayed, merged (IO scheduler, multiqueue handling)
– Dispatched into a device driver
– Completed when IO is finished
![Page 7: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/7.jpg)
8
Submission Handling in Block Layer
bio
Per-process plugging
IO scheduler
Dispatch queue
Multiqueue deviceNormal device
DQ1 DQN...
Queue mapping
Device driver
![Page 8: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/8.jpg)
9
IO Schedulers
• Decide when and in which order IO requests are submitted– NOOP – just pass requests into dispatch queue
– Deadline
• Prefers reads over writes
• Sorts waiting requests to reduce seeking
• Aims to dispatch each request at latest after its deadline has expired
– CFQ
• Prefers sync requests over async
• Tries to achieve fairness among tasks
• Support for IO priorities, cgroups, sync request idling, ...
![Page 9: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/9.jpg)
10
Multiqueue Device Handling
• Used for fast devices
• Limited plugging
• No IO scheduling
• Support for multiple hardware IO queues
• Lightweight
![Page 10: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/10.jpg)
IO Performance Analysis
![Page 11: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/11.jpg)
12
Iostatusually packaged in sysstat package
• Shows basic statistics about IO
• Very lightweight
• Run: iostat -dxk 1
Dev: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrqsz avgqusz await svctm sda 182.00 0.00 353.00 0.00 2152.00 0.00 12.19 0.89 2.53 2.50 dm0 0.00 0.00 536.00 0.00 2144.00 0.00 8.00 1.42 2.65 1.74
Dev: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrqsz avgqusz await svctmsda 235.00 0.00 300.00 0.00 2160.00 0.00 14.40 0.87 2.89 2.86 dm0 0.00 0.00 540.00 0.00 2160.00 0.00 8.00 2.01 3.72 1.79
![Page 12: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/12.jpg)
13
Example: Storage attached via Xen
• Sequential writes slow when writing through Xen blkfront driver to multipathed device
• Direct: 112 MB/s, throught Xen: 46 MB/s
• Iostat numbers:Dev: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrqsz avgqusz await svctmSdb 0.00 0.00 0.00 354.00 0.00 176128.00 995.07 31.97 91.79 2.84
Dev: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrqsz avgqusz await svctmSdd 0.00 0.00 0.00 1377.00 0.00 59988.00 87.13 30.98 22.38 0.73
After plugging fix (104 MB/s):Dev: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrqsz avgqusz await svctmSdb 0.00 0.00 0.00 571.00 0.00 145920.00 511.10 31.66 55.68 1.76
IO through Xen
![Page 13: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/13.jpg)
14
Takeaway 1
• Small requests hurt throughput– Overhead in kernel
– Overhead in the device itself
• Holds for any storage device
![Page 14: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/14.jpg)
15
Iowatcherhttp://masoncoding.com/iowatcher/
• Consumes traces from blktrace, mpstat output, fio bandwidth logs
• Plots various graphs (throughput, IO latency, IO location, …)
• Movies
![Page 15: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/15.jpg)
16
Iowatcher Graphs
Device IO
Of
set
(MB
)
22929
36276
49623
62970
76317
0 10 20 30 40 50 60 70 80 90 100 120
Time (seconds)
WritesReads
![Page 16: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/16.jpg)
17
Iowatcher Graphs
Throughput
KB
/s
0
16090
32180
48270
64360
0 10 20 30 40 50 60 70 80 90 100 120
Time (seconds)
WritesReads
![Page 17: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/17.jpg)
18
Iowatcher Graphs
IO Latency
late
ncy (
us)
0
1122
2244
3366
4489
0 10 20 30 40 50 60 70 80 90 100 120
Time (seconds)
![Page 18: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/18.jpg)
19
Iowatcher Graphs
IOPs
IO/s
0
715
1430
2145
2862
0 10 20 30 40 50 60 70 80 90 100 120
Time (seconds)
![Page 19: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/19.jpg)
20
Iowatcher Graphs
Queue Depth
Pen
din
g I
O
0
2
4
6
10
0 10 20 30 40 50 60 70 80 90 100 120
Time (seconds)
![Page 20: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/20.jpg)
21
Iowatcher Movie
• Movie time!
![Page 21: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/21.jpg)
22
Iowatcher Use Example
• Enabling ext4 dir_index feature slows down deleting of a directory tree.
# Without dir_indexleet:~ # time rm r /mnt/linux2.6.32.orig/real 0m4.862suser 0m0.032ssys 0m2.388s
# With dir_indexleet:~ # time rm r /mnt/linux2.6.32.orig/real 0m8.100suser 0m0.040ssys 0m2.588s
![Page 22: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/22.jpg)
23
Tree Deletion without Dir_index
Device IO
Of
set
(MB
)
22000
22724
23448
24172
24898 WritesReads
Throughput
KB
/s
0
10956
21912
32868
43824
0.00 0.50 1.00 1.50 2.00 2.50 3.00 3.50 4.00 4.50 5.00 5.50 6.00
Time (seconds)
WritesReads
![Page 23: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/23.jpg)
24
Tree Deletion with Dir_index
Device IO
Of
set
(MB
)
21000
21615
22230
22845
23460 WritesReads
Throughput
KB
/s
0
1050
2100
3150
4200
0 1 2 3 4 5 6 7 8 9 10 11 12
Time (seconds)
WritesReads
![Page 24: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/24.jpg)
25
Takeaway 2
• Seeking matters– Smaller chances of merging IO requests
– Seek time for rotational storage
![Page 25: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/25.jpg)
26
Blktrace
• Detailed information about IO requests processing
• Relatively large amount of data– May store over network
• Can handle multiple devices in parallel
• Gather as:
blktrace d /dev/sda d /dev/sdb d /dev/dm0
• Further processing using iowatcher, blkparse, btt
![Page 26: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/26.jpg)
27
Blkparse
8,0 4 498 0.536245624 5072 A RM 46664392 + 8 < (8,1) 466623448,0 4 499 0.536248072 5072 Q RM 46664392 + 8 [gcc]8,0 4 0 0.536262021 0 m N cfq5072S / alloced8,0 4 500 0.536262739 5072 G RM 46664392 + 8 [gcc]8,0 4 501 0.536266614 5072 I RM 46664392 + 8 [gcc]8,0 4 0 0.536268520 0 m N cfq5072S / insert_request8,0 4 0 0.536270374 0 m N cfq5072S / add_to_rr8,0 4 0 0.536276200 0 m N cfq workload slice:758,0 4 0 0.536278314 0 m N cfq5072S / set_active wl_prio:0 wl_type:28,0 4 0 0.536280939 0 m N cfq5072S / fifo=(null)8,0 4 0 0.536282276 0 m N cfq5072S / dispatch_insert8,0 4 0 0.536285224 0 m N cfq5072S / dispatched a request8,0 4 0 0.536286509 0 m N cfq5072S / activate rq, drv=18,0 4 502 0.536286919 5072 D RM 46664392 + 8 [gcc]8,0 4 503 0.556455119 0 C RM 46664392 + 8 [0]8,0 4 0 0.556469202 0 m N cfq5072S / complete rqnoidle 08,0 4 0 0.556471881 0 m N cfq5072S / set_slice=258,0 4 0 0.556475942 0 m N cfq5072S / arm_idle: 2 group_idle: 08,0 4 0 0.556476510 0 m N cfq schedule dispatch
![Page 27: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/27.jpg)
28
Blkparse (cont)8,0 0 1 0.556659272 5073 A R 47002176 + 32 < (8,1) 470001288,0 0 2 0.556661415 5073 Q R 47002176 + 32 [gcc]8,0 0 0 0.556674617 0 m N cfq5073S / alloced8,0 0 3 0.556675354 5073 G R 47002176 + 32 [gcc]8,0 0 4 0.556677319 5073 P N [gcc]8,0 0 5 0.556680380 5073 I R 47002176 + 32 [gcc]8,0 0 0 0.556682649 0 m N cfq5073S / insert_request8,0 0 0 0.556684273 0 m N cfq5073S / add_to_rr8,0 0 6 0.556688402 5073 U N [gcc] 18,0 4 0 0.564839523 0 m N cfq idle timer fired8,0 4 0 0.564842003 0 m N cfq5072S / slice expired t=08,0 4 0 0.564844568 0 m N / served: vt=9817282560 min_vt=98172784648,0 4 0 0.564847483 0 m N cfq5072S / sl_used=2 disp=1 charge=2 iops=0 sect=88,0 4 0 0.564849177 0 m N cfq5072S / del_from_rr8,0 4 0 0.564850534 0 m N cfq schedule dispatch8,0 4 0 0.564869775 0 m N cfq5073S / set_active wl_prio:0 wl_type:28,0 4 0 0.564871692 0 m N cfq5073S / fifo=(null)8,0 4 0 0.564872827 0 m N cfq5073S / dispatch_insert8,0 4 0 0.564875317 0 m N cfq5073S / dispatched a request8,0 4 0 0.564876421 0 m N cfq5073S / activate rq, drv=18,0 4 504 0.564876894 2743 D R 47002176 + 32 [kworker/4:2]8,0 2 86 0.570193124 0 C R 47002176 + 32 [0]
![Page 28: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/28.jpg)
29
Btt
• Uses binary event dump of blktrace events
• Produces various statistics– Q2C, Q2D, D2C latencies
– Current device queue depth
– Seeks per second
– Per process activity
• Useful to check what to look for in blkparse output
![Page 29: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/29.jpg)
30
Ftrace
• General kernel tracing framework
• Controlled via /sys/kernel/debug/tracing
• Documentation/trace/ftrace.txt
• Static trace points
– Syscalls
– Various events in filesystems, writeback, …
• Dynamic trace points on almost every kernel function
![Page 30: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/30.jpg)
Complex Problem Analysis
![Page 31: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/31.jpg)
32
Problem
• When customer launches a large tarball creation, HA monitors of postgress DB occasionally time out ⇒ service failover– Service timeout 10 seconds
• Used HW raid for storage, deadline IO scheduler, ext3 filesystem
• 8 GB of memory free, disk is loaded with writes
![Page 32: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/32.jpg)
33
Analysis 1/7
• Iostat pretty normal:
• Blktrace output large (~900 MB parsed)
• Use btt to show latency outliers
– Watch out for lost blktrace event
sort k 2 n r q2clat.dat | head 30
127.842616 10.341348
127.842619 10.341346
127.842621 10.341344
…
Dev rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrqsz avgquszSda 0.00 13726.00 1.00 174.00 2.00 43876.00 501.46 142.12
![Page 33: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/33.jpg)
34
Analysis 2/7
• Match back to blktrace events using timestamps8,0 118.779433534 638 A WS 11705240 + 8 < (8,2) 11395984
8,0 118.779433858 638 Q WS 11705240 + 8 [kjournald]
8,0 118.779435324 638 G WS 11705240 + 8 [kjournald]
8,0 118.779436253 638 I WS 11705240 + 8 [kjournald]
…
8,0 123.784506489 0 D WS 11705240 + 8 [swapper]
…
8,0 125.870800714 0 C WS 11705240 + 8 [0]
• Ok, so some IOs really take long
• Deadline IO scheduler delays outlaying IO
![Page 34: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/34.jpg)
35
Analysis 3/7
• Switched IO scheduler to NOOP
• Max latency reduced 3.8s and generally better
• Service time outs increased !?!
• Need more insight into why they time out
• Use ftrace to trace syscallsecho 1 >/sys/kernel/debug/tracing/events/syscalls/enable
cat /.../tracing/trace_pipe | gzip c >syscalltrace.gz
![Page 35: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/35.jpg)
36
Analysis 4/7
• Found large latencies in fsyncpostgres17461 [001] 3559.059091: sys_fsync(fd: 4)
postgres17461 [008] 3570.848573: sys_fsync > 0x0
syslogng3008 [005] 3433.451593: sys_fsync(fd: 7)
syslogng3008 [005] 3449.854534: sys_fsync > 0x0
…
• Partly caused by heavy logging from sysrq-w
• Partly design limitation of data=ordered mode of ext3
![Page 36: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/36.jpg)
37
Analysis 5/7
• Removed sysrq-w, switched ext3 to data=writeback
• Time outs less frequent but still occur
• Another syscall trace analysiscrm_master20388 [000] 355206.448764: sys_read(...)
...
crm_node20389 [006] 355207.654087: sys_mmap(...)
crm_node20389 [006] 355207.654091: sys_mmap > ...
crm_node20389 [006] 355208.889691: sys_close(fd: 3)
crm_node20389 [006] 355208.889693: sys_close > 0x0
...
crm_master20388 [001] 355220.880237: sys_read > 0xa
![Page 37: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/37.jpg)
38
Analysis 6/7
• Correlate inactivity periods with blktrace8,0 283.784399307 1867 A R 9373112 + 8 < (8,2) 9063856
8,0 283.784399608 1867 Q R 9373112 + 8 [crm_node]
8,0 283.784400643 1867 G R 9373112 + 8 [crm_node]
8,0 283.784401175 1867 P N [crm_node]
8,0 283.784401701 1867 I R 9373112 + 8 [crm_node]
8,0 283.784402232 1867 U N [crm_node] 1
…
8,0 284.987422579 0 D R 9373112 + 8 [swapper]
…
8,0 284.995404698 0 C R 9373112 + 8 [0]
![Page 38: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/38.jpg)
39
Analysis 7/7
• Slow reads of shared libs sum up to time out
• Reads behind writes
• Switch back to deadline IO scheduler, tune deadlines and fifo_batch much lower
• Finally services run reliably
![Page 39: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/39.jpg)
Conclusion
![Page 40: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/40.jpg)
41
Conclusion
• Complex interactions between storage, block layer, filesystems
• Watch out for unexpected small requests, seeks
• Dependent reads vs async writes
– Hard to guarantee latency under load
![Page 41: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/41.jpg)
Thank You
![Page 42: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/42.jpg)
43
![Page 43: Kernel Recipes 2015: Linux Kernel IO subsystem - How it works and how can I see what is it doing?](https://reader031.fdocuments.us/reader031/viewer/2022030305/5872ddab1a28ab74188b78c9/html5/thumbnails/43.jpg)
Unpublished Work of SUSE LLC. All Rights Reserved.This work is an unpublished work and contains confidential, proprietary and trade secret information of SUSE LLC. Access to this work is restricted to SUSE employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of SUSE. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.
General DisclaimerThis document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. SUSE makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for SUSE products remains at the sole discretion of SUSE. Further, SUSE reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All SUSE marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.