SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux...

18
SUSE SolidDriver Program ©2018 SUSE SolidDriver Program ABSTRACT Installing third party kernel drivers on SUSE Linux Enterprise products gives customers the opportunity to enable the latest hardware and software technologies. But doing so can be a complicated and uneasy effort. This document covers the goals, standards and techniques provided by the SUSE SolidDriver Program that simplify the task of deploying kernel drivers and provide assurance to customers that do. Version: 1.3 | June 13, 2018 Contents Introduction 2 Structure of This Document ......... 2 Kernel Modules 2 What are Kernel Modules ........... 2 Kernel Modules vs Kernel Drivers ....... 3 Power and Privilege of Kernel Modules ... 3 The Kernel Module Programming Interface 4 Kernel Module Compatibility ......... 4 Kernel Modules in SUSE Linux Enterprise 5 Third Party Kernel Modules 5 Challenges with Deploying Third Party Kernel Modules ..................... 5 Traditional 3rd Party Kernel Module Installation 5 Supportability .................. 6 Mitigating Complications ........... 6 Why Not Deliver All Required Kernel Modules with the SUSE Kernels? ............. 6 Additional Expense of Quality Assurance . 7 Additional Risk .................. 7 Additional Delay ................. 7 Licensing Restrictions ............. 7 Embracing Third Party Kernel Modules ... 7 Goals of the SUSE SolidDriver Program 8 Easy to Deploy .................. 8 Compatible with SUSE Linux Enterprise .. 8 Supportable by SUSE and 3rd Party Vendor 8 Trusted by the End User ............ 9 Usable with SUSE YES Certifications ..... 9 How It Works 9 Kernel Module Packages 9 Automatic Installation via modaliases ... 10 Kernel ABI Compatiblity ........... 10 Putting the Pieces Together .......... 11 SUSE Kernel ABI Stability 12 What does it mean for vendors? ...... 12 What does it mean for users? ....... 12 Why not maintain compatibility across service packs? ...................... 13 Does SUSE provide a kernel ABI symbol whitelist? .................... 13 Compatibility With User Space Applications 13 Joint Support Agreements 13 Process Agreement .............. 13 Without a Joint Support Agreement ..... 14 Kernel Support Taint Flag ........... 14 Kernel Taint States and Support Levels Assistance from SUSE Partners ...... 15 1

Transcript of SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux...

Page 1: SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux Enterprise ThekerneldeliveredwithSUSELinuxEnter-priseproductsiscompiledwithmanyofthe

SUSE SolidDriver Program

©2018

SUSE SolidDriver Program

ABSTRACTInstalling third party kernel drivers on SUSE Linux Enterprise products gives customers the opportunity to enable the

latest hardware and software technologies. But doing so can be a complicated and uneasy effort. This document

covers the goals, standards and techniques provided by the SUSE SolidDriver Program that simplify the task of

deploying kernel drivers and provide assurance to customers that do.

Version: 1.3 | June 13, 2018

Contents

Introduction 2

Structure of This Document . . . . . . . . . 2

Kernel Modules 2

What are Kernel Modules . . . . . . . . . . . 2

Kernel Modules vs Kernel Drivers . . . . . . . 3

Power and Privilege of Kernel Modules . . . 3

The Kernel Module Programming Interface 4

Kernel Module Compatibility . . . . . . . . . 4

Kernel Modules in SUSE Linux Enterprise 5

Third Party Kernel Modules 5

Challenges with Deploying Third Party Kernel

Modules . . . . . . . . . . . . . . . . . . . . . 5

Traditional 3rd Party Kernel Module Installation

5

Supportability . . . . . . . . . . . . . . . . . . 6

Mitigating Complications . . . . . . . . . . . 6

Why Not Deliver All Required Kernel Modules

with the SUSE Kernels? . . . . . . . . . . . . . 6

Additional Expense of Quality Assurance . 7

Additional Risk . . . . . . . . . . . . . . . . . . 7

Additional Delay . . . . . . . . . . . . . . . . . 7

Licensing Restrictions . . . . . . . . . . . . . 7

Embracing Third Party Kernel Modules . . . 7

Goals of the SUSE SolidDriver Program 8

Easy to Deploy . . . . . . . . . . . . . . . . . . 8

Compatible with SUSE Linux Enterprise . . 8

Supportable by SUSE and 3rd Party Vendor 8

Trusted by the End User . . . . . . . . . . . . 9

Usable with SUSE YES Certifications . . . . . 9

How It Works 9

Kernel Module Packages 9

Automatic Installation viamodaliases . . . 10

Kernel ABI Compatiblity . . . . . . . . . . . 10

Putting the Pieces Together . . . . . . . . . . 11

SUSE Kernel ABI Stability 12

What does it mean for vendors? . . . . . . 12

What does it mean for users? . . . . . . . 12

Why not maintain compatibility across service

packs? . . . . . . . . . . . . . . . . . . . . . . 13

Does SUSE provide a kernel ABI symbol

whitelist? . . . . . . . . . . . . . . . . . . . . 13

Compatibility With User Space Applications 13

Joint Support Agreements 13

Process Agreement . . . . . . . . . . . . . . 13

Without a Joint Support Agreement . . . . . 14

Kernel Support Taint Flag . . . . . . . . . . . 14

Kernel Taint States and Support Levels

Assistance from SUSE Partners . . . . . . 15

1

Page 2: SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux Enterprise ThekerneldeliveredwithSUSELinuxEnter-priseproductsiscompiledwithmanyofthe

Driver Kits 15

Driver Kit Add-ons . . . . . . . . . . . . . . 15

Driver Kits Target Products . . . . . . . . . 15

Bootable Driver Kits . . . . . . . . . . . . . 16

Installation with Bootable Driver Kits 16

Traditional Optical Media Installation • Network

Based Installation • PXE Boot Installation • Installing

directly from drivers.suse.com

Summary 18

Introduction

In today’s fast-paced world of technology,organizations are constantly deploying newhardware into their data centers. Usingnew hardware often requires installing new,vendor-delivered kernel drivers, which causesanxiety for many customers.

Whether it’s uncertainty around having theright driver for the right installation, under-standing preconditions and knowledge are re-quired for proper installation, ensuring back-ing by the hardware and OS vendors whensupport is required, or trust in deployingsensitive kernel code into a business criticalenvironment, there are good reasons for cus-tomers to be concerned about installing thirdparty delivered kernel modules.

At SUSE we understand the need to leveragenew technology, as well as the requirementsto install third party delivered kernel driversto do so. At the same time we can identifywith customer concerns when installing whatare essentially components of the OS kernelprovided by vendors other than SUSE.

SUSE has a long history of providing anenterprise level operating system softwareand has over time established many practicesthat make the installation and maintenanceof the OS comfortable and dependable forcustomers.

The SUSE SolidDriver Program leveragesthat expertise to create a set of standardsthat facilitate third parties to provide kerneldrivers in a uniform, consistent, proven andcompatible manner that gives end users easeand confidence in identifying, installing, andusing them in their SUSE Linux Enterpriseenvironments.

This document describes in detail the goalsand methods provided by the SUSE Solid-Driver Program that benefit SUSE partnersand customers alike.

Structure of This Document

This document begins by giving a quickoverview or kernel modules (drivers), howthey are used, and the way they work.Following that, a brief overview of thekernel modules delivered with SUSE LinuxEnterprise kernels is given just beforedelving into the topic of third party kernelmodules. At this point the benefits and risksof installing kernel drivers should be clearand sets us up for a description of the SUSESolidDriver Program.

The later sections of the document introducesthe goals of the SolidDriver Program, out-lines some of the standards it establishes, andfinally goes into detail about how all of thisworks.

Kernel Modules

The SUSE SolidDriver Program is focusedon the tasks of packaging, delivering, deploy-ing, and supporting kernel modules to beinstalled on SUSE Linux Enterprise OS’s.Before looking at the details of how kernelmodules are provided, it’s important to havea basic understanding of what Linux kernelmodules are and how they work, and thissection will give a quick overview.

If you’re familiar with Linux kernel modules,you can skip to the section Third Party Ker-nel Modules.

What are Kernel Modules

Kernel modules pieces of kernel object codethat can dynamically loaded into a runninginstance of a Linux kernel. Not only can themodules be loaded when needed, they canalso be unloaded and reloaded1. The abilityto load sections of kernel code as neededallows for great flexibility when using Linuxkernels. Because of the loadable nature they

1The reliability of unloading and reloading kernelmodules depends on the design of the module codeand possibly the hardware itself. Unloading shouldnot be a common task in a production environmentand when needed, should be done with care.

Page 3: SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux Enterprise ThekerneldeliveredwithSUSELinuxEnter-priseproductsiscompiledwithmanyofthe

SUSE SolidDriver Program — 3/18

are sometimes referred to as Loadable KernelModules (LKM).

• The Linux kernel is an extensiveprogram that supports a considerableamount features and functionalitiesfrom file-systems, to vast diversitiesof hardware components. The abilityto load only those sections of kernelcode that are required for a giveninstallation or workload, allows thekernel to run with a minimal memoryfootprint.

• When new features, or technologies areto be used, new kernel modules can beloaded as needed without rebooting thesystem. This is a key part of device hot-plugging support in the Linux kernelwhere device drivers (kernel modules)are loaded as new devices are discov-ered.

• Kernel modules can be built separatelyfrom the base kernel code and load-ed/tested dynamically and easily onexisting kernel installations. This re-duces the expensive process of buildingand deploying complete new kernelswith each iteration.

KernelModules vs Kernel Drivers

The terms kernel module and kernel driverare essentially synonymous, and in this doc-ument both terms will be used interchange-ably.

To be more precise, one can think of ker-nel module is a more general term for codethat is dynamically loaded into a runningkernel instance, and that kernel drivers arekernel modules built with the task of drivingor otherwise providing interfaces to entitiesexternal to the kernel itself whether it’s filesystems, hardware devices, or software appli-cations.

Power and Privilege of Kernel Modules

As explained in Anatomy of the Linux kernel2the GNU/Linux operating system consists

2M. Tim Jones, Anatomy of the Linux kernel:History and architectural decomposition, IBM devel-operWorks, http://www.ibm.com/developerworks/linux/library/l-linux-kernel/ (2007).

Figure 1. Kernel Modules

of two major parts, the user space and thekernel. The kernel provides resource manage-ment functions to the user space, like deviceaccess and process and memory management.Everything running in the kernel has full ac-cess to the hardware of the entire machine;there is no protection between the differentparts. The lack of protection in the kernelmakes the code fast, but at the same timedangerous: a bug in one part can bring downthe entire system.

The illustration above shows the differentlayers of a running computer system with thehardware resources shown in the center, theuser space on the outer ring with the kernelspace in-between. The gray slices labeled .korepresent kernel modules. As can be seen inthe diagram, there is no separation or layerof protection between kernel modules andthe kernel or hardware resources.

The user space builds upon the abstractionsprovided by the kernel. Everything in theuser space is running in the context of aprocess, and different processes are protectedfrom each other. The main communicationlayer between the user space and the kernelis the system call interface.

In this document, we focus on the kernelspace and the abstractions and mechanismsprovided to extend the kerneli’s functionalityby adding support for new hardware devicesor software features.

www.suse.com

Page 4: SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux Enterprise ThekerneldeliveredwithSUSELinuxEnter-priseproductsiscompiledwithmanyofthe

SUSE SolidDriver Program — 4/18

Support for different kinds of hardware livesin drivers, which can be compiled directlyinto the main kernel image or into kernelmodules, which can be loaded at run-time.Likewise, features like file systems or networkprotocols can be part of the main kernel im-age or they can be implemented in modules.When a module is loaded, it becomes a partof the kernel and gains all the powers thatkernel code has.

The Kernel Module Programming Interface

The programming interface between the ker-nel and kernel modules is similar to that ofshared libraries3 in user space: a shared li-brary provides (“exports”) a set of functionsand variables. A user-space program thatuses those functions and variables loads theshared library into its address space. Thelibrary loading code makes sure that the li-brary is properly initialized (“constructed”),and that all references to functions and vari-ables implemented in the library are properlyresolved so that the program can use them.Similarly, the kernel exports functions andvariables to kernel modules. When a kernelmodule is loaded, the kernel resolves all refer-ences to functions and variables implementedin the kernel so that the module can accessthem, and then it initializes the module. (Inthis analogy, the kernel module is in the roleof the user-space program.)

For both shared libraries and kernel modules,the term symbol refers to a function or vari-able, and the term exported symbol refersto a symbol that a shared library or the ker-nel provides. Just as a shared library candepend on other shared libraries (meaningit uses symbols exported by those other li-braries), kernel modules can depend on otherkernel modules.

When discussing programming interfaces,the term Application Programming Interface(API) is often used to refer to an interface atsource code level, and the term ApplicationBinary Interface (ABI) is used to refer tothe binary interface of an application. Thesetwo terms are also used in conjunction withkernel modules; there, the abbreviations

3David A. Wheeler, Programming LibraryHOWTO, The Linux Documentation Project, Sec-tion 3: Shared Libraries, http://www.tldp.org/HOWTO/Program-Library-HOWTO/ (2003).

kAPI and kABI (k standing for kernel) arealso encountered.

Kernel Module Compatibility

When loading kernel modules, it is essentialto ensure that the kernel and modules arecompatible. We can ensure this by buildingthe kernel and all modules in the same buildwhich insures all data structures involved areequal. When building the kernel modulesseparate from the core kernel build, anothermechanism is needed to ensure that all thedata structures involved are the same.

Historically, the kernel and its kernel mod-ules were all built at once. Back then, it wassufficient to ensure that the version strings ofthe kernel and of all modules were the same;we would just need to ensure that the versionstring changed whenever an exported symbolbecame incompatible. When people startedbuilding their own kernel modules separatelyfrom the rest of the kernel, this weak formof compatibility checking often resulted inbreakages. Modules would become incom-patible and still load but crash the kernelupon first use of the module, corrupt otherparts of the kernel and cause crashes there,or worse, even cause data corruption. Kernelmodules built separately from the rest of thekernel could not be handled in a reasonableway with kernel version string comparisons.

To solve this problem, a mechanism calledmodversions was invented, which assigns achecksum to each exported symbol. Thechecksums are computed based on how asymbol is defined in the source code; theyinclude the definitions of all data types reach-able from each symbol. Instead of makingsure that the kernel and module have thesame version, the kernel makes sure thatthe checksums of all the symbols that themodule uses match. Modules with checksummismatches are rejected.

This ensures that a symbol or anything reach-able from that symbol changes. However, itmight also generate false positives, because achecksum might change even when a moduleis not actually affected by the change causingthe checksum change.

www.suse.com

Page 5: SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux Enterprise ThekerneldeliveredwithSUSELinuxEnter-priseproductsiscompiledwithmanyofthe

SUSE SolidDriver Program — 5/18

Kernel Modules in SUSE Linux

Enterprise

The kernel delivered with SUSE Linux Enter-prise products is compiled with many of thecomponents built as modules. This allowsfor the flexibility as mentioned in the previ-ous section. Today, the SUSE kernels con-tain around 1800 modules in total, but sincethese modules are loaded as required basedon functional needs of the end user, only asmall portion of the total will be loaded atany given time. A typical server installationwill have around 100 or even fewer modulesloaded.

The ability to load parts of the kernel asneeded allows SUSE Linux Enterprise to sup-port a wide range of systems from notebooksto supercomputers. It also allows to sup-port a range of file systems and I/O stackswhich provides for suitability with diverseworkloads.

The fact that SUSE provides kernels splitinto modules permits particular modules tobe updated when needed to support new tech-nology as is typically done when supportingnew hardware releases. These driver updatesare typically developed and provided by thehardware vendors that produce and sell thehardware products themselves. It’s the ex-istence, and use of drivers from these thirdparty vendors that is the primary motivationfor the SUSE SolidDriver Program which willbe covered in the following sections.

Third Party Kernel Modules

As has been pointed out in the previous sec-tion, there is the option for third party ven-dors to deliver their own kernel modules thatare built for and can be loaded with SUSEKernels.

The ability to deliver custom or updatedkernel drivers allows hardware and softwarevendors to bring compatibility of their prod-ucts to SUSE Linux Enterprise OS’s, butthis capability doesn’t come without somechallenges. Those challenges are covered inthis following sections.

Challenges with Deploying Third Party Ker-

nel Modules

As mentioned in section Power and Privilegeof Kernel Modules, kernel modules are firstclass kernel code that have the same privi-leges and access to system resources as therest of the kernel. Kernel modules deployedwithout care can introduce great risk to theintegrity and stability of the complete sys-tem. Loading modules of unknown origin orsupport commitment immediately leads toa running system that is, strictly speaking,unsupported by SUSE.

In addition to the risk of installing kernelmodules of unknown origin or support sta-tus, kernel modules are tricky to install. Be-fore the SUSE SolidDriver Program, vendorswere delivering drivers in many differing andincompatible manners which increased thecomplexities and likelihood of user error.

Traditional 3rd Party Kernel Module Instal-

lation

When open source drivers are involved, atypical method of delivering kernel modulesto end users is to simply provide the sourcecode along with build and installation in-structions. This requires users to have botha level of software building expertise as wellas the proper kernel development environ-ment installed. The common way of buildingand installing kernel modules as updates tothose delivered with the base OS is to over-write the original modules with the updates.This works fine until the next kernel updateis installed which will happily overwrite thepreviously driver update with the originalbase version. Many times this leads to anun-bootable system.

A second typical delivery method is to pro-vide kernel drivers in a pre-compiled bun-dle (some form of archive like tar or evenrpm) along with an installation script to helpmake the deployment process easy for endusers. Some of these scripts are quite exten-sive and try to manage all the subtleties ofproper kernel module installation (and hope-fully removal as well if the user ever decidesto un-install). Unfortunately vendors tendto work in isolation and have developed vary-ing methods (many of which make unsafeassumptions) to install drivers. The end user

www.suse.com

Page 6: SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux Enterprise ThekerneldeliveredwithSUSELinuxEnter-priseproductsiscompiledwithmanyofthe

SUSE SolidDriver Program — 6/18

is then encountered with multitudes of dif-ferent, vendor-specific installation processes.Most of the methods work fine with manualinstallation on single, locally accessed sys-tems, but come short when customers needto perform mass installations, or automatedinstallations of various drivers on mixes ofhardware.

Supportability

An additional challenge with deploying thirdparty kernel modules comes with requiringsupport from SUSE. If the kernel is mis-behaving while a third party kernel moduleis installed, it’s important that SUSE engi-neers can work closely with the vendor ofthe driver in order to remedy the situation.This is not only true of third party delivereddrivers but also true of many of the in-boxdrivers delivered with SUSE Linux EnterpriseKernels where kernel drivers are supported inpartnership between SUSE and the hardwarevendors.

Just as we have established partnerships andjoint support processes setup to help sup-port our in-box kernel drivers, it’s importantthat any third party delivered drivers areaccompanied with a proper technical rela-tionship and support process between SUSEand the vendor providing the kernel driver.The SUSE SolidDriver Program provides pro-cesses and resources specifically for thosetasks. See the Joint Support Agreementssection later in this document for more de-tails.

Mitigating Complications

The SUSE SolidDriver Program was estab-lished and developed to address the chal-lenges identified with delivering, deploying,and supporting third party kernel modules.Before moving on to describe the programin detail let’s take a moment to address acommon question…

Why Not Deliver All Required Kernel Mod-

ules with the SUSE Kernels?

With all the risks and challenges associatedwith using third party delivered kernel mod-ules, why not avoid the situation all togetherby having SUSE deliver all needed driverswith the SUSE Linux Enterprise kernels?

Figure 2. Delivering new kernel functionality

That would indeed take care of many of thechallenges. The drivers would simply be in-stalled with our official kernels and kernel up-dates thereby removing the installation com-plications, support issues, and issues withtrust. It sounds like an ideal solution, butit’s not reasonable given the following rea-sons:

• Additional quality assurance expense• Additional risk to users that receive no

benefit• Additional delays in delivering the

needed support• Licensing restrictions

Before going into detail on each of thesepoints lets briefly look at the figure below.

The top portion represents standard kernelupdates that go to all SUSE customers whilethe bottom portion represents an individualkernel module update going to those cus-tomers requiring the new functionality. Inthe former case, the exposure of the updateis much greater than the latter. The cus-tomers with the gray color are customersthat would receive no benefit from the up-dated, new functionality. Only the greencustomers get the benefit, and further more,once the green customers are up and runningwith their new feature support they becomegray customers when the next new featuresor functional support are introduced withkernel module updates. Keep this image inmind while reading the following sections.

www.suse.com

Page 7: SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux Enterprise ThekerneldeliveredwithSUSELinuxEnter-priseproductsiscompiledwithmanyofthe

SUSE SolidDriver Program — 7/18

Additional Expense of Quality Assurance

SUSE Linux Enterprise kernels go througha great amount of testing and quality as-surance before being delivered to customers.The development of a service pack updateentails around 6 months of testing and hard-ening by SUSE and SUSE’s partners. Thecombined testing efforts span many compa-nies and customers and having the widestand most thorough testing coverage possiblein the available time frame. Anyone famil-iar with the efforts and costs of any kind ofproduct testing can appreciate the amountof expense this calls for. It wouldn’t makemuch sense to invest so much into testingthe initial release of a product or a productservice pack and not put the same level of at-tention to subsequent kernel updates. Indeed,SUSE kernel updates are done with care andtesting and the amount of code changed forany update is kept to the bare minimum inorder to achieve the specific objectives of theupdate while keeping risk of recessions andrequired testing to a minimum. These mini-mal, specific updates require weeks of testingbefore release.

Of course, kernel driver updates would notrequire the months of testing that a full ser-vice pack update encompasses, but couldrequire weeks of testing by SUSE and SUSEpartners in order to ensure no regressionsare introduced for the existing install base(the gray customers). The frequency of newtechnology introductions and required driverupdates would quickly create a very expen-sive process for all involved only to ensurethat those customers, who otherwise receiveno additional benefit, don’t encounter issueswhen deploying our standard updates. Itshould be clear that the cost vs risk wouldbe a questionable investment.

Additional Risk

Delivering any kind of update introducesrisk of regression. As outlined above, ex-tensive quality assurance is required to re-duce the risk of regression to a comfortablelevel. There is never 100% coverage and somelevel of risk is always present. Many regres-sions will unfortunately be initially identifiedby customers when they deploy the updates.While SUSE commits to fixing such regres-sions with the highest priority, we would

rather avoid them all together. Thereforeit’s better to restrict kernel module updatesto those specific customers that require themand not expose the rest of the customers tounnecessary risk.

Additional Delay

We could avoid the additional QA and riskassociated with delivering kernel module up-dates by integrating the updates into the ser-vice pack releases. That is exactly what hap-pens with the standard upstream drivers thatare included with the SUSE kernels. Typi-cally all commonly used drivers are updatedto the latest available version at time of codefreeze for a new service pack. The challengewith this approach is that the time betweencode freeze, and first customer ship spansmany months. Given the speed of develop-ment in the IT industry, it’s not uncommonthat at time of first customer ship of a newSUSE Linux Enterprise major release or ser-vice pack, many newly released hardwarecomponents are not supported by the newlyreleased SUSE product.It’s of value to have, without delay, supportin SUSE Linux Enterprise for the latest tech-nologies as they are released to the market.For this reason, use of third party kernelmodules is equally valuable and importantfor our customers.

Licensing Restrictions

Last, but not least, licensing restrictionsmake it difficult if not impossible for SUSEto deliver some pieces of software includingsome kernel modules.

Embracing Third Party Kernel Modules

As we have learned so far, the flexibility thatloadable kernel modules bring to a Linuxbased system is a valuable asset while thesame time safely applying third party kernelmodules to production systems requires careand attention to details. SUSE embraces thedelivery and use of third parties with SUSELinux Enterprise products when done respon-sibly. The SUSE SolidDriver Program hasbeen designed to address the complicationsand risks associated with deploying such mod-ule updates and provide customers a wellintegrated, supported, and dependable expe-rience in the process.

www.suse.com

Page 8: SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux Enterprise ThekerneldeliveredwithSUSELinuxEnter-priseproductsiscompiledwithmanyofthe

SUSE SolidDriver Program — 8/18

Goals of the SUSE SolidDriver

Program

The fundamental goal of the SUSE Solid-Driver Program can be summed up in thefollowing sentence:

Ensure customers can deploy nec-essary 3rd party kernel driverswith ease and confidence.

The program achieves this by establishingstandards to be followed by vendors whopackaging and delivering kernel drivers thatare used with SUSE Linux Enterprise OS’s.These standards are set out to ensure thatkernel drivers are:

• Easy to deploy• Compatible with SUSE Linux Enter-

prise products• Supported by SUSE and the driver ven-

dor• Trusted by the end user• Usable with SUSE YES certifications

Easy to Deploy

Given the impact to system functionalitythat many kernel drivers bring, it’s impor-tant that they are installed properly. Thefirst step in improving the reliability of kernelmodule installations is to remove complexityin the operation. We can remove complexityby removing the need to compile kernel mod-ules, as well as using standard methods andtools to install them.

The SolidDriver program recommendsthat modules are provided to customers instandard RPM packages that contain pre-compiled kernel module binaries which arecompatible with the SUSE Linux Enterprisekernels. The program utilizes a specifickernel module package RPM packagingstandard that provides for automatic andcorrect installation relieving the end userof detailed knowledge around building andinstalling kernel drivers. This packagingstandard will be covered in more depth inthe section Kernel Module Packages later inthis document.

Compatible with SUSE Linux Enterprise

There are two aspects of compatibility thatthe SolidDriver program is focused on. Thefirst is related to kernel module updates prop-erly co-existing with standard system files.This means that kernel module installationsmust preserve system files and system pack-age installations must preserve kernel modulefiles. As was described in the section on thirdparty kernel modules, traditional installa-tion involves installing the module object filein the location reserved for standard kernelmodules - sometimes overwriting the defaultsystem files in the process. This leads tocomplications when installing SUSE kernelupdates which will may remove or overwritethe files installed by the third party kernelmodule. In such cases, the third party kernelmodule will need to be re-installed after thekernel update before reboot in order to en-sure the proper driver is available for systemfunctionality. It’s up to the system adminto remember this crucial step - failing to doso can result in an un-bootable system thatcan be tricky to restore. The kernel modulepackage standard defined by the SUSE Solid-Driver program places driver updates in apre-defined, SUSE standard location that isreserved for externally provided kernel mod-ules. This allows for third party kernel mod-ules to be installed in parallel with SUSEkernels without the complications of file col-lisions.

The second aspect of compatibility is con-cerned with the kABI (see The Kernel Mod-ule Programming Interface ). If the kABIrequired by the kernel module does not matchthe kABI of the running kernel, the modulewill not load. Once again the kernel modulepackage standard gives the solution by han-dling kernel ABI requirements at the packagedependency level. This protects users frominstalling incompatible kernel modules thatcould lead to systems not working. How thisis managed is discussed in detail in the sec-tion [Kernel ABI Compatibility] later in thisdocument.

Supportable by SUSE and 3rd Party Vendor

By using kernel module packages, the pro-cess of installation of kernel modules is fullysupported by SUSE. In addition SUSE Solid-Driver compliant drivers ensure that joint

www.suse.com

Page 9: SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux Enterprise ThekerneldeliveredwithSUSELinuxEnter-priseproductsiscompiledwithmanyofthe

SUSE SolidDriver Program — 9/18

support agreements are established betweenSUSE and the third party vendor so thatproper support can be given to customershaving any issues related to the SUSE ker-nels. See Joint Support Agreements for moreinformation.

Trusted by the End User

Kernel modules are proper pieces of kernelcode and have the same access and privilegesas the rest of the kernel. There is no addedprotection between the kernel module andthe hardware, data, and complete runtimeenvironment to protect against unwanted be-havior.

Because of that, users need to have a suffi-cient level of trust in any kernel drivers thatthey install on their systems. How does theuser establish that trust?

The SUSE SolidDriver Program helps theuser establish trust by using a combinationof proper package meta data (like packagedescriptions and vendor identification) alongwith digital signatures that the user can ver-ify with the vendors he or she trusts. Theverified digital signature ensures that the con-tents of the package indeed originate from theidentified vendor and has not been tamperedwith by external forces.

Usable with SUSE YES Certifications

Hardware vendors will want to utilize theirkernel drivers when running SUSE YES Cer-tification tests. Doing so requires that theenvironment under test is available to endusers in the exact configuration as has beencertified. This means that any kernel driversused must be available to the customers inthe exact binary form as used for testing andshould remain available for the life of thehardware product or SUSE Linux EnterpriseOS that the certification is based on.

To help vendors maintain this availability,the SUSE SolidDriver Program offers theopportunity to have their drivers hosted atdrivers.suse.com. SUSE will make sure thedrivers used in certifications remain availablefor the required amount of time.

How It Works

Up to this point, the complications relatedto installing third party kernel modules havebeen identified, and the goal of the SUSESolidDriver Program to reduce these com-plexities and set standards to help do so havebeen illustrated.

The following sections will go into some detailon technical implementations that meet thestandards and help achieve the goal.

• Kernel Module Packages The heart ofthe SUSE SolidDriver Program stan-dards. This packaging standard allowsus to install kernel modules with easeand confidence.

• SUSE Kernel ABI Stability To helpkeep users running with installed thirdparty kernel modules, SUSE has a pol-icy to keep the kernel ABI stable withkernel updates provided to customers.

• Joint Support Agreements In order toeffectively support customers that arerunning SUSE Linux Enterprise kernelswith third party kernel modules, we setinto place cooperative support proceseswith the third party vendors.

• Driver Kits Driver kits allow a vendorto bundle all software required to en-able a product into one location that iseasily integrated and installed on SUSELinux Enterprise operating systems.

Kernel Module Packages

SUSE Linux Enterprise products use theRPM software packaging format for softwaredelivery, installation, and maintenance Thepackage installation stack within SUSE LinuxEnterprise is designed to consume and pro-cess RPM packages for installation.

Because of this, it’s only natural that theSUSE SolidDriver program standardizes onRPM as the recommended way to packagekernel drivers. In addition to simply utiliz-ing the RPM format, SUSE has developedan approach specifically designed for packag-ing kernel modules. This packaging schemeis known as the Kernel Module Package orKMP.

The KMP defines the location where kernelmodule object files are installed on the sys-tem. This insures compatibility at the file

www.suse.com

Page 10: SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux Enterprise ThekerneldeliveredwithSUSELinuxEnter-priseproductsiscompiledwithmanyofthe

SUSE SolidDriver Program — 10/18

system level between the KMP and otherinstalled packages - specifically the kernelpackages.

The SUSE Linux Enterprise products comewith a variety of kernels to choose from de-pending on use case. These kernels are di-vided by architecture and flavor. The archi-tecture would be the platform CPU archi-tecture (e.g x86_64, i586, ppc64, ia64 etc.)while the flavor matches a specific kernel con-figuration like pae, xen, trace or the mostcommon, default. Each of these architectureand flavor combinations provides a different,and incompatible kABI which requires sepa-rate kernel module object files to be compiledfor each target kernel that a driver is devel-oped to work with.

The KMP standard will build, from a singlesource, kernel modules for each kernel. Thebuild process handles the details behind thescenes which makes the procedure easy for de-velopers. The result is separate sub-packages- one for each kernel type.

In addition to the proper location of theinstalled files, and the intrinsic handling ofdifferent kernel flavors, the KMP brings twoother capabilities unique to kernel modules:

• modalias hints• kABI dependency hints

The former allows for automated installationof the proper KMPs that matches the under-lying hardware while the later ensures thatKMPs and kernels are compatible with eachother before they are installed.

Automatic Installation viamodaliases

Understanding exactly which kernel driversare required for a given hardware productcan be a difficult and baffling routine. Kerneldrivers have arcane and sometimes outdatednomenclature like: tg3, bnx2x, ixgbe, qla2xxxetc. Kernel module packages usually carrythe same naming as the kernel modules them-selves and since many of them are used gener-ically across different releases and models ofhardware, there is no easy way to indicatewhich exact driver and version is tested andsupported with specific hardware products.To complicate matter further, KMPs comein sets of RPM sub-packages - one for each

Figure 3. Auto KMP selection

possible kernel configuration. A customertypically only requires one or two of the sub-packages and knowing which one might notbe easily known.

When kernel drivers are delivered in properlybuilt KMPs, the SUSE installer can pickan choose the right packages to install on agiven system. The installer will query theunderlying hardware for a list of components,which are identified using unique PCI or USBIDs. The ID strings are stored in what arecalled modalias strings and are the same onesused by the system to determine which kernelmodules to load upon hardware detection.

Each KMP in turn provides a list of IDS thatit supplements. The SUSE installer will queryall enabled package repositories for packagessupplementing the underlying hardware andwhen a match is found, the package(s) areselected for installation. The installer alsouses information provided by the KMP toinstall the one built for the installed kernel.

Properly built kernel module packagesstreamline the kernel driver deploymentby allowing customers to simply place thepackages in repositories accessed by theSUSE installer. The intelligence built intothe installer handles the rest.

Kernel ABI Compatiblity

It’s important that structures kernel modulesto interface with the kernel are compatiblewith the installed kernel. This interface iscalled the kernel ABI or kABI. See the section“Kernel Modules” at the beginning of thisdocument for more information on kABI andkernel module compatibility.

Kernel modules that do not use a compatible

www.suse.com

Page 11: SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux Enterprise ThekerneldeliveredwithSUSELinuxEnter-priseproductsiscompiledwithmanyofthe

SUSE SolidDriver Program — 11/18

Figure 4. kABI Checks

kABI will fail to load into the running kernel.Installing a kernel module update that failsto load can result in missing functionalitycrucial for system operation and can evenlead to a non-bootable system. Therefore it’simportant to ensure that only kernel modulesthat are compatible with the system kernelare installed. Equally important is to onlyinstall kernel updates that are compatiblewith installed, third party kernel modules.

By leveraging the package inter-dependencymechanisms of the KMP standard, the SUSESolidDriver program helps protect users fromunknowingly creating a state of incapabilitywhen installing kernel drivers or kernel up-dates.

The SUSE Linux Enterprise kernel packagesare built containing a list of kABI symbolsets that they provide. In turn, KMPs arebuilt containing a list of symbol sets that theyrequire. During installation of either kernelor kernel module packages, the SUSE pack-age installer will verify that the symbol setsmatch. If not, a package dependency error isthrown, and the package is not installed. Inthis way, we ensure that incompatible kernelsor kernel modules are not installed and savethe user from potential complications withsystem operation.

Putting the Pieces Together

The KMP Pictorial View illustration aboveshows how all the pieces of the kernel modulepackage fit together. On the left hand sideare the components of the package includingthe actual package files (in rounded boxes)and package meta data (dog-eared rectan-gles). In the middle is the KMP itself whereall the components are packaged. To the

Figure 5. KMP Pictorial View

right of the KMP extend arrows to ellipsesdesignating the different modes of installingthe KMP onto the target system at the farright of the figure.

Below is a brief description of each of theKMP components:

Module This is the actual kernel module ob-ject file or files.

Docs Package documentation.Utils Optional, supplemental utilities pro-

grams and scripts to be used in con-junction with the kernel module.

Description Package text that brieflydescribes the contents of the package.This includes both the RPM Summaryand RPM Description.

Vendor String indicating the vendor produc-ing and supporting the package.

Signature The public part of a digital signa-ture of the package that ensures thatthe package has not been altered byanyone other than that owning the pri-vate key of the signature.

kABI The kABI symbol set dependencies ofthe kernel module.

modalias The modalias list linking this pack-age to the specific hardware that thekernel module supports.

These pieces are packaged into the KMPwhich is installable onto the target systemusing YaST, AutoYaST, or zypper (SUSE’snative package installation tools). In addi-tion KMPs can be integrated into appliancesusing SUSE’s KIWI tool set. Lastly, since

www.suse.com

Page 12: SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux Enterprise ThekerneldeliveredwithSUSELinuxEnter-priseproductsiscompiledwithmanyofthe

SUSE SolidDriver Program — 12/18

KMPs are standard RPM packages, they canbe installed using the plain rpm command.The rpm bubble is shown in lightly subduedbecause it’s considered by SUSE to be a lowlevel tool that does not provide the richnessof features that the SUSE installers provide.

SUSE Kernel ABI Stability

It would be great if OS kernels had supportfor all features now and in the future, hadno bugs, and no known security holes. Thenwe could deploy our systems and not be con-cerned with updates and potential regres-sions The fact is, kernels do have bugs andsecurity holes that must be addressed. Inaddition, new features and functionality areconstantly being developed, enhanced andoptimized. This is true for the SUSE LinuxEnterprise kernels, and customers need todeal with the updates that come their way.

These updates, which contain kernel codechanges, can lead to incompatibilities withother parts of the system if precautions arenot taken. In this document we are con-cerned with the compatibility at the kernelABI (kABI) level. This compatibility is some-times termed kABI Stability and only appliesto kernel level code (i.e. kernel modules). Formore information on the kernel programinginterfaces see the The Kernel Module Pro-gramming Interface section earlier in thisdocument.

When updating kernels, SUSE does take pre-cautions with regard to the stability of thekernel ABI and has the following policy forSUSE Linux Enterprise products:

• Keep the kernel ABI stable for the life-time of each service pack. Kernel mod-ule packages built for a specific servicepack will remain compatible with all up-date kernels to that service pack. Thesame policy holds for kernels updatesbetween the initial release of the prod-uct and the first service pack.

• With service pack or major version up-dates, there is no guarantee of kernelABI compatibility with the previous re-leases4. Kernel module packages main-tained for several service pack releases

4Later service packs (e.g. SUSE Linux Enterprise10 SP4) tend to have smaller changes to the kernel,

will need to be rebuilt for each servicepack.

Given the life of general support of an indi-vidual service pack being around two yearsand the option to extend that by an additionthree years using Long Term Service PackSupport5, the policy provides for a fairly longstable period with compatibility changes hap-pening at predefined and well planned inter-vals. That allows for the introduction ofsubstantial improvements over the life of theproduct, resulting in an overall better of-fering to our customers without sacrificinglonger term stability and compatibility.

What does it mean for vendors?

If a vendor provides modules that are main-tained in the upstream kernel and supportedin the SUSE Linux Enterprise product, mostlikely kernel ABI compatibility will not bean issue. It’s important to work with SUSEto ensure the module is considered supportedin the SUSE products, and gets updated tothe latest upstream version at each servicepack release.

If the modules are not part of the upstreamkernel, they will need to be rebuilt for eachservice pack and possibly require source codechanges to match the updated kernel inter-faces. SUSE offers a beta program allowingfor partners to test and prepare for any ker-nel module adjustments well in advance ofthe service pack release.

For further information on the SUSE betaprogram or ensuring that your upstreamdrivers are up to date and supported inSUSE Linux Enterprise products, contactyour PartnerNet representative6.

What does it mean for users?

The most prominent use of third party ker-nel modules is updating the standard driversdelivered with SUSE Linux Enterprise prod-ucts to enable new products or features. Suchupdates are typically maintained upstream,

and in these cases we strive to keep the kernel ABIcompatible with the previous service pack releasewhere possible.

5https://www.suse.com/support/programs/long-term-service-pack-support.html

6https://www.suse.com/partners/

www.suse.com

Page 13: SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux Enterprise ThekerneldeliveredwithSUSELinuxEnter-priseproductsiscompiledwithmanyofthe

SUSE SolidDriver Program — 13/18

and as a result are included in the next ser-vice pack release removing the need for theupdates going forward. In those cases, thekernel ABI changes coming with service packupdates will not affect end users.

If the modules are not maintained in the up-stream kernel, then the user will need to takecare when deploying service pack updates toensure that new, compatible versions of thekernel modules are also applied. Users areadvised to contact the vendor of the kernelmodules for assistance with updating to newservice packs.

Why not maintain compatibility across ser-

vice packs?

SUSE Linux Enterprise is based on the com-munity developed Linux kernel, and by that,leverages the advancements and code qualitygained from the community development pro-cess7. To help customers best take advantageof the ongoing community work, we strive tokeep the SUSE Linux Enterprise kernels asclose as possible to the upstream communitycode base. Since the ABI of the communitykernel is continually changing, maintainingcompatibility over time would require diverg-ing more and more from the community. Asthis divergence widens, the benefits that weand our customers realize by being close toand part of the general kernel communitydiminish.

So, while keeping the kernel ABI compatiblefor the life of a SUSE product is preferred byvendors that deliver their own drivers, theresulting divergence from upstream is not thebest option for maintaining code quality overthe long term. On the other hand, breakingthe kernel ABI compatibility several times ayear is neither desirable nor imperative.

SUSE’s policy for kernel ABI stability strikesa balance between compatibility and advance-ment that allows us and our customers totake advantage of new upstream developmentduring the life of SUSE Linux Enterprisewhile minimizing divergence from general ker-nel community.

7Ibrahim Haddad (Ph.D.) and Brian Warner,Upstreaming: Strengthening Open SourceDevelopment, The Linux Foundation, Sec-tion 3: Benefits of Upstreaming, http://www.linuxfoundation.org/publications/linux-foundation/upstreaming-strengthening-open-source-development

Does SUSE provide a kernel ABI symbol

whitelist?

One tactic to maintaining a stable kernelABI over a long period of time is to definea subset of the complete kernel ABI guaran-teed to remain compatible. This subset isoften termed a “whitelist” referring to onlythe symbols that are “safe” to use in externalkernel modules. SUSE does not follow thisroute and instead keep virtually all8 symbolsunchanged, but for a shorter period of time.We have found this policy to be better suitedto most vendors maintaining kernel modulesas it provides freedom and flexibility in lever-aging the kernel interfaces they require tosupport their products.

CompatibilityWithUser SpaceApplications

We have focused on the interfaces at the ker-nel level only (i.e. between the kernel andkernel modules); the topic of user space in-terfaces is beyond the scope of this document.It will simply be stated that the formal pro-gramming interfaces between kernel and userspace are kept compatible for the life of a ma-jor release of SUSE Linux Enterprise Serverand Desktop products. Indeed, most of theinterfaces are compatible across major re-leases as well.

Joint Support Agreements

A running kernel that has had third party ker-nel module loaded can not be supported bySUSE engineering without cooperation of thevendor of the kernel module code. In orderto ensure supportability to SUSE Linux En-terprise customers, SUSE requires a supportprocess agreement to be established betweenvendor’s of third party kernel modules andSUSE.

Process Agreement

The joint support agreement establishes aprocess for handling customer support ofSUSE Linux Enterprise kernels running ker-nel modules supplied by partners. Withthe agreement, SUSE technical support en-gineers, can handle initial triage and data

8We do allow rare exceptions for a few symbolsthat clearly make no sense for an external kernelmodule to use.

www.suse.com

Page 14: SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux Enterprise ThekerneldeliveredwithSUSELinuxEnter-priseproductsiscompiledwithmanyofthe

SUSE SolidDriver Program — 14/18

collection from the customer. Once it is de-termined that the partner’s kernel moduleis suspect to the customer’s problem, or ifthere is no way to rule out the partner’s ker-nel module as root cause, SUSE will involvethe partner’s support contacts as outlined inthe support process agreement.

Through the joint support agreement, SUSEand the partner can work together on resolv-ing the issue while maintaining a consistentinterface to the end customer.

Without a Joint Support Agreement

Without a joint support agreement in place,SUSE can’t do root-cause analysis of kernelissues when third party kernel modules havebeen loaded. Customers will be instructed tocontact the vendor of the third party kernelmodule for additional support and analysis,in which case the vendor will need to eitherfix the issue in the kernel module code, orroot cause and provide evidence of a validbug in the SUSE Linux Enterprise kernel forSUSE for remedy.

Kernel Support Taint Flag

Modules can be built with a special flag thatindicates the support status of the module.The module can be queried for the modeof support flag using the modinfo command.There are two possible modes:

• Supported: yes = SUSE supported

• Supported: external = supported bydriver vendor and SUSE.

If neither of these modes are indicated, thesupportability of the kernel module, and com-plete kernel, is unknown and should be as-sumed as unsupportable.

When loading modules The SUSE Linux En-terprise kernels will identify the supporta-bility of the module code and set the ap-propriate kernel taint flag if an unsupportedor externally supported module is loaded.The kernel taint flags are one mechanism tohelp customers and SUSE support techni-cians identify the supportability of a runningkernel environment.

In default and recommended configuration,SUSE Linux Enterprise Server will refuse to

load kernel modules that cannot be identifiedas supported by SUSE or an external party.Running unknown or unsupported code inkernel space invalidates any support commit-ments between SUSE and the end user.

Kernel Taint States and Support Levels

When the kernel issues an error message itwill indicate the taint state of the kernelat that time. The information is encodedthrough single-character flags in the stringfollowing “Tainted:” in a kernel error mes-sage.

The descriptions below only apply to environ-ments running official SUSE Linux Enter-prise kernels.

Untainted kernel = SUSE SupportedWhen a kernel is not tainted with an‘X’ or ‘N’ flag, all loaded kernel code isfully supported by SUSE and SUSEengineering has expertise to debug andfix the code. Only code that has beenbuilt, tested, and shipped by SUSEloads without tainting the kernel.

‘X’ tainted kernel = e’X’ternally SupportedSome code shipped with SUSE LinuxEnterprise kernels, and all codebuilt and shipped by SUSE partners,requires the partner’s assistance insupporting the code. If any such codeis loaded by the kernel, the ‘X’ taintflag will be set indicating that partnerassistance is necessary to providesupport. Only official partners whohave support agreements establishedwith SUSE are allowed to identify theirkernel module binaries as externallysupported.

‘N’ tainted kernel = u’N’supported If atany time, a running kernel instanceloads a kernel module that cannot beidentified as supported by SUSE orsupported by a SUSE partner, the ‘N’taint flag will be set. Such an envi-ronment is not considered supportedby SUSE and looses guarantees of anyefforts by SUSE towards resolutionto customer issues encountered. Torestore the SUSE support guarantees,the issue must be reproduced in anenvironment without the ‘N’ kerneltaint flag set.

www.suse.com

Page 15: SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux Enterprise ThekerneldeliveredwithSUSELinuxEnter-priseproductsiscompiledwithmanyofthe

SUSE SolidDriver Program — 15/18

The kernel maintains list of taint flags inaddition to the ones related to support.For the full list and descriptions, refer to/usr/src/linux/Documentation/sysctl/kernel.txt

Assistance from SUSE Partners

As stated above, in order to fully resolvecustomer issues, code built and shipped bySUSE partners as well as some code shippedwith SUSE Linux Enterprise products mightrequire assistance by SUSE partners. SUSEprovides support commitments to such codebased on strong partnerships and establishedagreements between between partners andSUSE for the specific code in question. Forthe SUSE Linux Enterprise customer, thesupport process will be transparent and allpartner assistance handled behind the scenesbetween SUSE and the partners. In mostcases, the customer will not notice the dif-ference between support coming solely fromSUSE or with the cooperation of SUSE part-ners.

Driver Kits

When enabling products such as server ordesktop systems, vendors commonly provideto customers a handful of “packages” to be in-stalled on the target OS. These packages canbe anything from firmware updates, devicedrivers, and documentation that directly en-able the hardware, to utilities and programsthat enhance the user experience.

When providing packages to enable hardwareproducts for use with SUSE Linux Enterpriseproducts, the SUSE SolidDriver Program rec-ommends delivering all packages for a givenproduct, or even product family on a singlemedium so that customers have the opportu-nity to obtain from a single source everythingthat they require.

Driver Kit Add-ons

The medium of choice when providing bun-dled software is the add-On Product as de-scribed in the SUSE Linux Enterprise docu-mentation:

“Add-on products are system extensions. Youcan install a third party add-on product ora special system extension of SUSE LinuxEnterprise Server (for example, a CD with

Figure 6. Driver Kits Enable Products

support for additional languages or a CDwith binary drivers). To install a new add-on, start YaST and select Software+Add-OnProducts. You can select various types ofproduct media, like CD, FTP, USB massstorage devices (such as USB flash drives ordisks) or a local directory. You can work alsodirectly with ISO files. To add an add-on asISO file media, select Local ISO Image thenenter the Path to ISO Image. The RepositoryName is arbitrary.”9

A Driver Kit is an add-on that contains ker-nel drivers to be installed on a given releaseof SUSE Linux Enterprise product. Thedriver kit add-on can be installed during theinitial installation of the SUSE Enterpriseproduct by checking Include Add-On Prod-ucts from Separate Media on the InstallationMode screen or installed afterwards usingthe YaST2 Add-on Products module. Forfurther information refer to the SUSE LinuxEnterprise Server Deployment Guide

Driver Kits Target Products

Proper preparation and documentation ofdriver kits is important for customer usability.The customer should be expected to knowtwo things:

• Hardware product to be enabled• SUSE Linux Enterprise product that is

to be used with the hardware

Therefore, a driver kit should be documentedand delivered based on the products it en-ables, rather than on the drivers it contains.It’s also important that a driver kit indicatesthe SUSE Linux Enterprise product that it’sdesigned to be installed on.

A driver kit that is documented as9“Installing Add-On Products” in SUSE Linux

Enterprise Server Deployment Guide.

www.suse.com

Page 16: SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux Enterprise ThekerneldeliveredwithSUSELinuxEnter-priseproductsiscompiledwithmanyofthe

SUSE SolidDriver Program — 16/18

Enables ACME ProServer S3000server with SUSE Linux Enter-prise Server 11

is much more useful to the end user than adriver kit described as

Provides updates to the igb andmpt2sas drivers

The names of drivers should not be of pri-mary concern to the end user and the usershould not need to figure out if these driversapply to his product. In addition to properdescription of the products that a driver kitenables, a list of drivers or other packagescontained in the driver kit is useful as sup-plementary information.

It should assumed that the driver kit hasbeen tested by the vendor for the productsit’s indicated to work with.

These details go a long way toward usabilityand confidence for the end user and is anotheraspect of how the SUSE SolidDriver Programhelps ensure the best user experience whenusing third party kernel modules with SUSELinux Enterprise products.

Bootable Driver Kits

There are times when kernel drivers are re-quired to boot a system in order to installthe OS. There are even times when an up-dated kernel is required to boot a system.Since the bits contained in the SUSE LinuxEnterprise product are frozen when releasedto customers, and do not get updated untilthe next service pack, the Bootable DriverKit was developed by the SUSE SolidDriverProgram to easily enable bringing up systemsusing updated kernels or kernel modules sothat installation of the OS can be achieved.

A Bootable Driver Kit is an add-on productimage that contains a boot loader section inaddition to the standard driver kit add-onrepository. The boot loader contains isolinuxand UEFI loaders as well as a kernel and ini-trd containing the linuxrc10 program whichin turn initiates the SUSE Linux Enterpriseinstallation process. Only the componentsof this first stage of the installation boot

10http://en.opensuse.org/SDB:Linuxrc

Figure 7. Installing with Optical Media

process are contained on the bootable driverkit, which is used to kick-off the installationthat then continues using the standard SUSELinux Enterprise product media or reposito-ries.

One of the other benefits that the bootabledriver kit brings over the standard driver kitis that it primes the installation process byautomatically adding the driver kit add-onto the selection of repositories used for instal-lation. It can be designed to automaticallyinstall the additional packages that it pro-vides or even provide user choice via patterns.This results in eliminating any extra stepsthat the end user for to do during the instal-lation process to ensure that the packagesprovided by the driver kit get installed to-gether with the OS. The installation of thedriver it is seamless.

Installation with Bootable Driver

Kits

To illustrate the flexibility and ease of usingbootable driver kits, we will briefly describethe workflow of four installation methods:

• Installing with standard DVD/CDs• Installing over network• Installing using PXE boot• Installing directly from repository on

the Internet

Traditional Optical Media Installation

First we will show how the bootable driverkit is used to install SUSE Linux Enterpriseusing optical media (DVDs).

Workflow

1. Burn the driver kit and SUSE productsto DVD media

www.suse.com

Page 17: SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux Enterprise ThekerneldeliveredwithSUSELinuxEnter-priseproductsiscompiledwithmanyofthe

SUSE SolidDriver Program — 17/18

Figure 8. Network based installation

2. Boot machine using driver kit media3. First stage installer instructs user to

install the SUSE Linux Enterprise prod-uct media

4. Installation continues as normal

Except for the fact that the driver kit mediais used to boot the machine, and the SUSEproduct media is to be inserted as a sec-ondary step, there is virtually no differencein to the installation experience compared tothe default SUSE Linux Enterprise process.The initial boot loader menu can be used toadd installer command line options or eveninstruct the system to install from networkwhich we will illustrate next.

Network Based Installation

Installing a system from a network installa-tion server is covered next. In this methodthe driver kit media is still used to boot thesystem before initiating the network basedinstall.

Workflow

1. Burn driver kit image to DVD media2. Boot machine using driver kit media3. Using the install= command line option,

instruct the installer to pull the SUSELinux Enterprise product from a serverover the network

4. Installation continues as normal

As can be seen, installing from a networkserver is just as easy as installing completelyfrom optical media. In this example opticalmedia was still utilized to boot the system.To avoid that, and initial complete remotebased installations, a PXE boot environmentis commonly utilized.

Figure 9. PXE Boot Network installation

PXE Boot Installation

Setting up a PXE boot environment for doingfull network based installs requires pullingthe kernel and initrd images from the basemedia and hosting them on the PXE server.Leveraging a bootable driver kit in the pro-cess is basically the same.

NOTE: the instructions below are only forreference and leave out many details For com-prehensive instructions on setting up a PXEenvironment refer to the deployment guideof the SUSE Linux Enterprise product.

Workflow

1. Host SUSE Linux Enterprise productas well as the driver kit on an installa-tion server.

2. Setup the PXE boot server to usethe kernel and initrd images from thebootable driver kit.

3. Add to the pxelinux config an install=

boot option that points to the URLof the SUSE Linux Enterprise productrepository

4. Add an addon= option the pxelinux configfile that points to the URL of the driverkit repository

5. Boot system using the systems networkPXE boot source

6. Continue installation as usual

Of the steps listed above, only the fourth stepis unique compared to standard PXE instal-lation of SUSE Linux Enterprise products.Since the driver kit is an additional softwarerepository, we need a way to instruct theinstaller to use it during installation. That is

www.suse.com

Page 18: SUSE SolidDriver Program · 2018-06-13 · SUSESolidDriverProgram—5/18 KernelModulesinSUSELinux Enterprise ThekerneldeliveredwithSUSELinuxEnter-priseproductsiscompiledwithmanyofthe

SUSE SolidDriver Program — 18/18

Figure 10. Installing from drivers.suse.com

accomplished using the addon= command lineoption.

Installing directly from drivers.suse.com

In the PXE Boot example above, we learnedhow the addon= command line option allows usto point the installer at a network installationsource to pull in an extra, add-on repository.If the system to be installed has Internetconnectivity, the driver kit can be installeddirectly from a vendor hosted repository. Inthis example we illustrate pulling the driverkit directly from drivers.suse.com during in-stallation. The workflow is identical to thePXE Boot workflow described above - onlythe add-on URL is changed.

Summary

Kernel drivers are required to enable hard-ware or software products and features andnew products often requires new or updateddrivers to function properly. Hardware andsoftware vendors alike are able to deliver theneeded kernel drivers for enabling their prod-ucts with SUSE Linux Enterprise OSs andthis is usually the quickest way to get newsupport to the customers. In the end, manySUSE customers will be confronted with thetask of deploying and utilizing third partydelivered kernel modules in their IT environ-ments. Care must be taken as any form ofkernel code, including kernel modules, canimpact the integrity and stability of the com-plete system.

The SUSE SolidDriver Program has estab-lished standards and best practices for pack-aging and delivering kernel drivers to be

installed on SUSE Linux Enterprise prod-ucts. These standards ensure ease of in-stallation, compatibility, supportability, aswell as trust and integrity that provide cus-tomers with confidence and assurance wheninstalling third party kernel drivers.

An overview of the standard and techniquesset forth by the SUSE SolidDriver Programhas been presented in this document.Customers can use this information to helpevaluate the completeness and suitabilityof drivers that are given to them. SUSEpartners can use the general advice thathas been provided to consider how they canbetter deliver their drivers in a standard,compatible and user satisfactory manner.

www.suse.com

©2018 SUSE LLC. All rights reserved. SUSE and the SUSE logo are registered tradmarks of SUSE LLC in the United

States and other countries.