HP ProLiant server power management on SUSE Linux ... · PDF fileCollaborative Power Control...

Technical white paper

HP ProLiant server power management on SUSE Linux Enterprise Server 11 and 12 Fourth edition

Table of contents Introduction ............................................................................................................................................................................ 2

HP Power Regulator ............................................................................................................................................................. 3

HP Dynamic Power Capping ................................................................................................................................................. 4

Power monitoring with HP iLO 4 ......................................................................................................................................... 5

HP Dynamic Power Capping demonstration with HP iLO 4 ............................................................................................. 5

HP Insight Control power management ............................................................................................................................. 7

HP ProLiant power management with SLES11 and SLES12 ........................................................................................... 7

Collaborative Power Control with SLES11 SP2 and SLES12 ........................................................................................ 10

Idle power states with SLES11 and SLES12 ................................................................................................................... 11

Additional SLES11 and SLES12 power management features .................................................................................... 11

Summary ............................................................................................................................................................................. 12

Click here to verify the latest version of this document

http://h20195.www2.hp.com/V2/GetDocument.aspx?docname=4AA5-4761ENW&cc=us&lc=en

Technical white paper | HP ProLiant server power management on SUSE Linux Enterprise Server 11 and 12

2

Introduction

For data centers with fixed capacity for power and cooling distribution, rising compute and storage requirements create significant challenges. Within these data centers, power management is an increasingly important factor in boosting the efficiency of power provisioning.

This document provides an overview of the processor-based power-saving features of HP ProLiant servers. This white paper also explores the power management features—HP Power Regulator, HP Dynamic Power Capping, and Collaborative Power Control (CPC)—embedded in the HP ProLiant server platforms. We also examine the importance of using HP Insight Control power management software with automated energy optimization to provide real-time information about power and cooling consumption in the data center.

This paper discusses these power management features and their relationship to the SUSE Linux Enterprise Server 11 (SLES11) and SUSE Linux Enterprise Server 12 (SLES12) operating systems. The discussions include new features available with ProLiant Gen8 and ProLiant Gen9 Intel®-based servers and SLES11 and SLES12.

The SLES11 and SLES12 operating systems and HP ProLiant servers combine to use processor-based features to achieve better power efficiency. The processor-based features include the following:

• Performance states (P-states) define a set of fixed operating frequencies and voltages, where P0 represents the highest operating frequency and voltage. Power savings occur by entering P-states with lower frequency and voltage levels. Management of P-states is through the platform firmware or the operating system.

• CPU states (C-states), excluding the C0 state, represent idle states and determine the power consumed when a processor is idle. C0 is a non-idle state, with higher C-states representing idle conditions with increasing power savings. C-states are under the control of the operating system.

HP ProLiant servers are also capable of utilizing the various processor states to enable innovative, system-independent power management features. States implemented in the hardware and firmware include:

• HP Power Regulator—Providing a facility to control processor power usage and performance, either statically or dynamically, depending on the mode selected.

• HP Dynamic Power Capping—Enabling you to limit the power consumed by a server.

The Power Regulator and Power Capping technologies work in tandem. To make the operating system aware of Power Capping, HP provides the CPC technology with ProLiant Gen8 and later Intel-based servers, and SP1 and later SLES11 and SLES12 operating systems. This two-way communication mechanism between the operating system and platform firmware can be used by the operating system and hardware collaboratively to choose the appropriate performance level for the server.

This document is intended for readers familiar with HP ProLiant servers and the SLES11 and SLES12 operating systems.


3

HP Power Regulator

As a configurable processor usage feature, HP Power Regulator allows you to choose from several options for enabling the server to manage P-states, or to delegate control of regulating P-states to the operating system. You can configure the Power Regulator from the Power Management component of HP Insight Control.

HP Power Regulator is included within the firmware on both Intel-based and AMD-based ProLiant servers. ProLiant servers provide the following HP Power Regulator modes, selectable using the ROM Based Setup Utility (RBSU) or through HP Integrated Lights Out (iLO) 4.

• HP Dynamic Power Savings control of P-states resides within the firmware. This mode allows the firmware to change the P-state of the processor to match the server load. However, when the CPC setting is enabled in RBSU, the operating system and the firmware collaborate to attain the desired frequency for a processor. When CPC is disabled, this mode allows the firmware to exclusively control the P-states of a processor to match the server load. Dynamic Power Savings is the default mode on HP ProLiant Gen8 and later servers.

• HP Static Low Power control of P-states resides within the firmware. The P-state of the processor is static, and it is set to the P-state that corresponds to the lowest operating frequency of the processor.

• HP Static High Performance control of P-states resides with the firmware. The P-state of the processor is static, and it is set to P0, which corresponds to the highest operating frequency of the processor.

• OS Control controls P-states through the SLES11 and SLES12 operating systems. SLES11 and SLES12 manage the P-states according to the policy set by an administrator, through the operating system.

For the first three HP Power Regulator modes, the firmware maintains exclusive control of P-states by restricting SLES11 and SLES12 from using them. These modes cause SLES11 and SLES12 to report in /var/log/messages that “CPU frequency scaling is not on your processor.” The operating system will not attempt to manage the P-states of the processors. SLES11 SP2 and SLES12 include an enhancement that allows the operating systems to collaborate with the platform in HP Dynamic mode on ProLiant Gen8 and later servers. The fourth mode, OS Control, allows the ProLiant server firmware to delegate the duty of managing P-states to the SLES11 and SLES12 operating systems. You can adjust the Power Regulator Settings through the RBSU or the HP iLO 4 interface, as shown in figure 1. Transitions to and from OS Control require a system reboot for the change to take effect, but you can change the system between the other three modes dynamically. Figure 1. Modifying the Power Regulator settings


4

To adjust the CPC setting, access the RBSU as shown in figure 2. If you modify this setting, you must reboot the system for the change to take effect.

Figure 2. Adjusting the CPC settings in the RBSU

For more information on HP Power Regulator technology, see:

h20000.www2.hp.com/bc/docs/support/SupportManual/c00300430/c00300430.pdf

HP Dynamic Power Capping

HP Dynamic Power Capping satisfies data center power provisioning requirements by allowing you to assign a power budget to a single server or a group of servers. The HP ProLiant server platform enforces that limit by changing the processor P-states and throttling the processor clock in an operating system-independent manner. Dynamic Power Capping operates independent of the HP Power Regulator setting and can occur in any setting. Dynamic Power Capping is also independent of operating system power management.

As illustrated in figure 1, you can use HP iLO 4 to configure a power cap. HP iLO displays important information about maximum power for the power supply, the maximum power observed for the server, and the power used by the server when idle. You can use this information to select an appropriate power cap, either by specifying the absolute maximum watts or a percentage of the maximum observed power of the server.

For more information on HP Dynamic Power Capping, visit: hp.com/go/dpc

For an in-depth presentation on the HP Dynamic Power Capping technology, refer to:

h20000.www2.hp.com/bc/docs/support/SupportManual/c01549455/c01549455.pdf

http://h20000.www2.hp.com/bc/docs/support/SupportManual/c00300430/c00300430.pdf

http://www.hp.com/go/dpc



5

Power monitoring with HP iLO 4

HP iLO 4 can monitor current power consumption along historical timelines with the scale-out or advanced license. As shown in figure 3, HP iLO 4 displays the current power consumption, as well as the peak and average power consumption for the past 24-hour and 20-minute time periods.

Figure 3. HP iLO4 monitoring power consumption

For more information on the HP Integrated Lights-Out management technology, refer to: hp.com/go/iLO

HP Dynamic Power Capping demonstration with HP iLO 4

The following demonstration shows HP Dynamic Power Capping in action by increasing the workload on an HP ProLiant DL380p Gen8 Server under the SLES11 SP2 64-bit operating system. Figure 4 shows the iLO configuration setting for a server where the capping threshold is set to 130 watts. This setting means the maximum power consumption is limited to approximately 130 watts (i.e., “capped”). You can base power capping values on historical observations. Doing so will ensure an appropriate limit without invoking processor states that might decrease performance, while also ensuring a determinate power consumption envelope.

Figure 4. HP iLO4 power settings

http://www.hp.com/go/iLO


6

Figure 5 illustrates what happens when power capping is not set. The server’s power consumption escalates when the workload increases. You can also see that the maximum power was 143 watts and the minimum power was 137 watts during the past five minutes.

Figure 5. Power capping disabled

Figure 6 demonstrates that when capping is enabled, the platform limits power consumption to 130 watts—even with an increase in workload—to satisfy the power budget set by the user.

Figure 6. Power capping enabled


7

HP Insight Control power management

HP Insight Control power management is an integrated power monitoring and management application that allows you to monitor power and temperature across multiple servers. It also provides central access to power control mechanisms such as power capping, power regulator, and server power off. In addition, you can also change the power cap settings on groups of servers all at once. HP ProLiant Gen8 and later servers and HP Insight Control Power Management (beginning with v7.0) include “Automatic Energy Optimization,” which includes major enhancements for power management:

• The Power Discovery Services (PDS) solution for Gen8 and later servers extends to HP BladeSystems attached to intelligent Power Distribution Units (iPDUs). You no longer need to manually create/edit the rack in Insight Control power management to add the powered-by connections for the BladeSystems that can be automatically discovered.

• Under-utilized server reporting helps identify unused or under-utilized machines wasting data center resources. This enhanced reporting helps in planning under-utilized system retirement or repurposing. The report summarizes power usage analyzed by CPU utilization and by age of the system.

• Enhanced reports based on HP Systems Insight Manager’s (SIM) Crystal Reporting feature enable you to schedule and run power/thermal reports and automatically receive scheduled email reports.

• Importing Configuration Updates enables you to update the physical and power topology configuration into Insight Control. You can script configuration updates for export from other data center infrastructure management tools such as HP Asset Center, and then import them into HP Insight Control. Currently only initial configurations can be imported.

Insight Control power management software is a plug-in for the HP SIM that monitors and controls HP ProLiant and HP Integrity servers throughout the data center.

For more information on HP Insight Control power management, visit: hp.com/go/dpc

For more information on HP SIM, visit: hp.com/go/hpsim

HP ProLiant power management with SLES11 and SLES12

SLES11 and SLES12 can manage ProLiant servers’ power usage by adjusting the processor P-states when the HP Power Regulator setting in RBSU is in OS Control mode. Typically, within the Linux operating system, a governor dictates policy, while a suitable P-state driver accomplishes the actual P-state transition. SLES11 and SLES12 offer a choice of governors, each implementing a different policy. These governors range from “powersave,” which selects the P-state corresponding to the lowest processor frequency, to “performance,” which selects the P-state corresponding to the highest processor frequency. The default governor for

SLES11 is the “ondemand” governor. For SLES12, the default governor is “performance.” Both governors dynamically adjust the processor P-states to match the load on the server.

On Intel-based ProLiant platforms, SLES11 and SLES12 natively utilize Intel's Demand Based Switching with Enhanced Intel SpeedStep® Technology. On AMD-based ProLiant platforms, SLES11 and SLES12 utilize AMD's PowerNow! technology. Table 1 lists the P-state driver on Intel-based and AMD-based ProLiant G6, G7 or later platforms.

Table 1. P-state drivers on HP ProLiant servers

Processor family P-state Driver (SLES11) P-state Driver (SLES12)

Intel® Xeon® acpi-cpufreq intel_pstate

AMD Opteron powernow-k8 powernow-k8


http://www.hp.com/go/hpsim


8

For SLES11 and SLES12 to manage a processor’s power consumption, the firmware must communicate to the OS information about the processor P-states and their associated frequencies. You can see this information in files and directories in “/sys/devices/system/cpu”.

SLES11 also provides the “cpufreq-util” command, which provides information about the P-states of the processors in the system in a user-friendly format. When used without arguments, “cpufreq-info” displays information about all processor cores, including the P-state driver, the frequency range defined by the processor, the available frequency steps (i.e., the P-states), the available and current governors, and the current frequency. This command also provides options to display specific information or information about specific CPUs as illustrated in the sample output shown below.

Example 1 shows output for CPU 0 in OS Control mode (DL380p Gen8). The governor used under the OS Control mode can be changed dynamically in “/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor” for each CPU. SLES11 also provides the “cpufreq-set” command for selecting the governor.

Example 1. Output for CPU 0 in OS Control mode

# cpufreq-info -c 0

cpufrequtils 004: cpufreq-info (C) Dominik Brodowski 2004-2006

Report errors and bugs to http://bugs.opensuse.org, please.

analyzing CPU 0:

driver: acpi-cpufreq

CPUs which need to switch frequency at the same time: 0

hardware limits: 1.20 GHz - 2.30 GHz

available frequency steps: 2.30 GHz, 2.30 GHz, 2.20 GHz, 2.10 GHz, 2.00 GHz, 1.90 GHz, 1.80 GHz, 1.70 GHz, 1.60 GHz, 1.50 GHz, 1.40 GHz, 1.30 GHz, 1.20 GHz

available cpufreq governors: conservative, userspace, powersave, ondemand, performance

current policy: frequency should be within 1.20 GHz and 2.30 GHz. The governor "ondemand" may decide which speed to use within this range.

current CPU frequency is 1.20 GHz (asserted by call to hardware.

For more information about the “cpufreq-util” and “cpufeq-set” commands, please refer to the SLES11 manual pages found at: suse.com/documentation/sles11/

SLES12 provides the “cpupower” command to provide information about the P-states of the processors in the system in a user-friendly format. When invoked with “cpupower –c all frequency-info,” it displays information about all processor cores, including the P-state driver, the frequency range defined by the processor, the available frequency steps (i.e., the P-states), the available and current governors, and the current frequency. This command also provides options to display specific information or information about specific CPUs, as illustrated in the sample output shown below.

Example 2 shows output for CPU 0 in OS Control mode (DL580p Gen8). The governor used under the OS Control mode can be changed dynamically in “/sys/devices/system/cpu/cpu*/cpufreq/scaling_governor” for each CPU. SLES12 also provides the “cpupower frequency-set” sub-command for selecting the governor.

http://www.suse.com/documentation/sles11/


9

Example 2. Output for CPU 0 in OS Control mode

# cpupower -c 0 frequency-info

analyzing CPU 0:

driver: intel_pstate

CPUs which run at the same hardware frequency: 0

CPUs which need to have their frequency coordinated by software: 0

maximum transition latency: 0.97 ms.

hardware limits: 1.20 GHz - 3.40 GHz

available cpufreq governors: performance, powersave

current policy: frequency should be within 1.20 GHz and 3.40 GHz. The governor "performance" may decide which speed to use within this range.

current CPU frequency is 2.73 GHz (asserted by call to hardware).

boost state support:

Supported: yes

Active: yes

3200 MHz max turbo 4 active cores




For more information about the “cpupower” command, refer to the SLES12 manual pages at: suse.com/documentation/sles12/

http://www.suse.com/documentation/sles12/


10

Collaborative Power Control with SLES11 SP2 and SLES12

When ProLiant servers are under OS Control mode for power management, the platform might still impose power capping without the knowledge of the operating system. First introduced on Intel-based ProLiant G6 servers, and included in all Gen8 and later ProLiant servers, is the ability for the server and the operating system to collaborate on power management. The HP CPC mechanism is capable of providing capping-related feedback to the operating system, as well as collaborating with the operating system in managing the server’s power consumption. This combination provides the quick response time of HP Dynamic Power Capping, while also providing correct processor power information to the operating system.

CPC utilizes the Processor Clocking Control (PCC) interface, which coordinates processor performance between the platform firmware and the operating system. Jointly developed by HP and Microsoft, the PCC interface is publicly available, allowing other platform vendors to implement it. Platform firmware releases since August 2009 for Intel-based ProLiant G6 or later servers include features for CPC.

For more information on PCC, refer to: acpica.org/sites/acpica/files/Processor-Clocking-Control-v1p0.pdf

When a CPC-enabled server is in HP Dynamic Power Capping mode, the firmware does not present P-state information to the operating system. Instead, the firmware presents the minimum and maximum frequencies of the processor, allowing the OS to pick any frequency within that range, rather than restricting the processor to a specific P-state. As with OS Control mode, the operating system governor will provide policy and request the required performance from the processor through the new PCC driver (pcc-cpufreq) in SLES11 SP2 and SLES12. The platform firmware will strive to honor the requested frequency. If the processor is capped at that time for any reason, then the platform firmware will inform the OS that the request could not be accomplished due to capping. When capping is not configured, the PCC driver will continue to function in lieu of the P-state driver, as shown in the sample output (example 3) for cpu0 for a DL380p Gen8 Server with SLES11 SP2 x64.

Example 3. PCC driver function in lieu of the P-state driver

# cpufreq-info -c 0

cpufrequtils 004: cpufreq-info (C) Dominik Brodowski 2004-2006 Report errors and bugs to http://bugs.opensuse.org, please. analyzing CPU 0:

driver: pcc-cpufreq

CPUs which need to switch frequency at the same time: 0 hardware limits: 1.20 GHz - 2.30 GHz

available cpufreq governors: conservative, userspace, powersave, ondemand, performance

current policy: frequency should be within 1.20 GHz and 2.30 GHz. The governor "ondemand" may decide which speed to use within this range.

current CPU frequency is 1.91 GHz (asserted by call to hardware.

Notice that unlike the server under OS Control, the driver is “pcc-cpufreq”, and there are no preset frequency steps.

http://www.acpica.org/sites/acpica/files/Processor-Clocking-Control-v1p0.pdf


11

Idle power states with SLES11 and SLES12

Processor power use at idle is a crucial factor in determining a server’s power consumption when there is no workload to execute. Typically, when a processor has no work to perform, the operating system places the processor in a halt state signified as C1. Newer-generation processors include deep C-states, allowing SLES11 and SLES12 to take advantage of these states. Although C-states can significantly reduce power consumption, the drawback of going to a deeper C-state is the latency associated with the time it takes the processor to wake up from its idle state and resume executing instructions. Information about the C-states for system processors is available in “/sys/devices/system/cpu/cpu*/cpuidle/state*”.

Note It is possible to configure a server without idle C-states. To do so, select the “No C-states” setting in RBSU.

Additional SLES11 and SLES12 power management features

SLES11 and SLES12 provide a comprehensive set of features for managing the power usage of ProLiant servers. Tickless idle is a kernel feature that eliminates periodic timer ticks when system CPUs are idle. The Linux kernel uses periodic timer events, called timer ticks, for each CPU. These ticks facilitate process accounting, scheduler load balancing, and maintaining per-CPU timer events. The one drawback, however, is that when systems are idle, these periodic ticks wake up the system and bring it out of its power-saving "sleep" state –roughly every millisecond for newer kernels operating at 1,000 Hz. This “wake-up call” causes unnecessary power consumption. With tickless idle in SLES11 and SLES12, the periodic timer tick is eliminated when the CPU is idle—which allows the CPU to remain in power-saving states for longer periods. The result is lower overall system power consumption and reduced costs.

SLES11 and SLES12 include more granular, file-based power profiles, making enterprise-wide power management easier.

Additional tools are available for monitoring system power consumption. For example, utilizing the powerTOP tool, it is possible to identify the processes most responsible for waking up a processor from its idle state, and thereby driving up power consumption. Refer to the powerTOP documentation for further reading on what the output of powerTOP represents, as well as for tips and tricks on how to best tune your servers for maximum power savings.

Figure 7 shows the powerTOP output on an idle 1P ProLiant DL380p Gen8 Server with an Intel Xeon E5-2630 processor and 8 GB system memory. In this example, the average residency in the deepest C-state is about 64.6 ms. ACPI C4 actually corresponds to the hardware C7 state, which is the deepest C-state provided by the processors on that platform. The server achieved this state because the processor transitioned about 16 times from its idle state. The IPMI service was stopped on the server in the example.

Note When you halt the IPMI driver, you lose the ability to monitor the server. If the IPMI service is stopped to save power, you can resume the IPMI service with the command: “service ipmi restart.”


12

Figure 7. PowerTOP output on an idle 1P HP ProLiant DL380p Gen8 Server

Note To see the P-states (frequencies) information in the powerTOP output, you must set the HP Power Regulator to OS Control mode.

Summary

HP ProLiant servers can save power when under load and when idle. When featured in the hardware, the processor-based power management capabilities are automatically enabled by the firmware. These features operate with close coordination between the firmware and the SLES11 and SLES12 operating systems. Typically, you do not need to activate these features; ProLiant servers ship with the power-saving features already enabled.

Several innovative features—such as the HP Power Regulator, HP Dynamic Power Capping, and CPC technology—provide advanced power saving and budgeting features on HP ProLiant servers. When you use HP Insight Control power management software with automated energy optimization, you also benefit from significant enhancements for power management, including Intelligent Power Discovery for Blades, under-utilized server reporting, Crystal Reports, and Imported Configuration Updates.


Sign up for updates hp.com/go/getupdated

Share with colleagues

Rate this document

© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. The only warranties for HP products and services are set forth in the express warranty statements accompanying such products and services. Nothing herein should be construed as constituting an additional warranty. HP shall not be liable for technical or editorial errors or omissions contained herein.

Intel, Intel SpeedStep, and Intel Xeon are trademarks of Intel Corporation in the U.S. and other countries. AMD is a trademark of Advanced Micro Devices, Inc. Microsoft is a registered trademark of the Microsoft group of companies.

4AA5-4761ENW, September 2014

Resources Power Regulator overview

Power Regulator for ProLiant servers, technology brief

HP Insight Control power management software overview

HP Power Capping and HP Dynamic Power Capping for ProLiant servers

HP iLO Management Engine

HP Systems Insight Manager overview, library, and downloads

Linux on HP ProLiant servers

SLES on HP servers

Processor Clocking Control Interface Specification

AMD PowerNow! technology

Enhanced Intel SpeedStep Technology and Demand Based Switching on Linux

Linux cpufreq kernel documentation

Linux cpuidle kernel documentation

Intelligent Platform Management Interface

ProLiant technology white papers

http://www.hp.com/go/getupdated

http://www.facebook.com/sharer.php?u=http://h20195.www2.hp.com/V2/GetDocument.aspx?docname=4AA5-4761ENW.pdf

http://twitter.com/home/?status=HP ProLiant server power management on SUSE Linux Enterprise Server 11 and 12+@+http://h20195.www2.hp.com/V2/GetDocument.aspx?docname=4AA5-4761ENW

http://www.linkedin.com/shareArticle?mini=true&ro=true&url=http://h20195.www2.hp.com/V2/GetDocument.aspx?docname=4AA5-4761ENW&title=HP+ProLiant+server+power+management+on+SUSE+Linux+Enterprise+Server+11+and+12+&armin=armin

https://hpresearch.az1.qualtrics.com/SE/?SID=SV_6EuQKOr2Ku1CUzH&CMG=CMG490&Pubnumber=4AA5-4761ENW&BU=Servers&Doctype=TWP






















http://twitter.com/home/?status=HP%20ProLiant%20server%20power%20management%20on%20SUSE%20Linux%20Enterprise%20Server%2011%20and%2012+%40+http%3A%2F%2Fh20195.www2.hp.com%2FV2%2FGetDocument.aspx?docname=4AA5-4761ENW



http://www.hp.com/servers/power-regulator




http://h18013.www1.hp.com/products/servers/management/remotemgmt.html?jumpid=hpr_R1002_USEN

http://www.hp.com/go/hpsim

http://www.hp.com/go/proliantlinux

http://h18004.www1.hp.com/products/servers/linux/suse/index.html

https://acpica.org/sites/acpica/files/Processor-Clocking-Control-v1p0.pdf

http://www.amd.com/us/products/technologies/amd-powernow-technology/pages/amd-powernow-technology.aspx

http://softwarecommunity.intel.com/articles/eng/1611.htm

http://lxr.linux.no/linux+v2.6.32/Documentation/cpu-freq/

http://lxr.linux.no/linux+v2.6.32/Documentation/cpuidle/

http://www.intel.com/design/servers/ipmi/

http://www.hp.com/servers/technology

HP ProLiant server power management on SUSE Linux ... · PDF fileCollaborative Power Control...

Documents

Transcript of HP ProLiant server power management on SUSE Linux ... · PDF fileCollaborative Power Control...