Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August...
Transcript of Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August...
![Page 1: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/1.jpg)
Processing of hardware interrupts in LinuxPetr Holášek, Red HatAugust 17, 2015
![Page 2: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/2.jpg)
HW and kernel
![Page 3: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/3.jpg)
Interrupt
• Hardware interrupt vs softIRQ• Interrupt ReQuest from hardware• In system represented as interrupt vector• Pin-based vs MSI(-X)
![Page 4: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/4.jpg)
Pin-based IRQ
• Triggered by electronic signal• Pin can be shared• Possible race condition
![Page 5: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/5.jpg)
MSI(-X)
• Message Signaled Interrupts• Introduced with PCI 2.2• Triggered after write to an address• Improved version called MSI-X
![Page 6: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/6.jpg)
Interrupt controller
• APIC• LAPIC (local APIC) - at CPU• IOAPIC (I/O APIC) – at device
• Using system bus• APIC bus deprecated
![Page 7: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/7.jpg)
Interrupt handler
• Handles received interrupt• Need for speed• Most of the work deferred• Using tasklets or workqueues
![Page 8: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/8.jpg)
Interrupt handler• Multiple CPUs cannot parallelize interrupt handler
• Only one interrupt handler running on CPU at time
• CPUs can alternate in handling the handler
![Page 9: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/9.jpg)
userspace
![Page 10: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/10.jpg)
Kernel interfaces
• The only visible info for user• /proc/interrupts
• /proc/irq/<irqnum>/...
• /sys/devices/…/irq
• /proc/stats
![Page 11: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/11.jpg)
Interrupt affinity
• Mask of possibly receiving processors • /proc/irq/X/smp_affinity
• Hexadecimal mask or list• Its value doesn't mean much
![Page 12: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/12.jpg)
Interrupt distribution
• Should be done on multiprocessor systems• Storage devices, NICs• Risk of CPU overload or cache misses
![Page 13: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/13.jpg)
Hardware topology• NUMA node• Package• Cache domain – L2 or L3• CPU• numactl tool
![Page 14: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/14.jpg)
Optimal affinity layout• Identify and group all high-volume interrupts• Move them to unique single CPUs• Spread out lower-volume interrupts among other CPUs
• Do it within the device NUMA node
![Page 15: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/15.jpg)
irqbalance
![Page 16: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/16.jpg)
Irqbalance
• Interrupts load balancing daemon • Can improve performance and save power• https://github.com/Irqbalance/irqbalance• Support for NUMA
![Page 17: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/17.jpg)
Irqbalance basics• Balancing of interrupts is complex task• Periodic review of system• Affinity management among heterogeneous systems
![Page 18: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/18.jpg)
Irqbalance basics 2
• Don't migrate interrupt out of home NUMA node
• CPU load - time spent in interrupt and softIRQ context
![Page 19: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/19.jpg)
Irqbalance algorithm 1
• Parse all available interfaces• Evaluate overloaded processors• Evaluate the busiest IRQs• Rebalance IRQ on processors
![Page 20: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/20.jpg)
Irqbalance algorithm 2
• Set new smp_affinity values• Wait some time and repeat
![Page 21: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/21.jpg)
Irqbalance options
• Can respect affinity_hint set by driver• Can ignore selected IRQs• Can ignore isolated CPUs
![Page 22: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/22.jpg)
Alternatives to irqbalance
![Page 23: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/23.jpg)
“Premature optimization is the root of all evil.”Donald E. Knuth
![Page 24: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/24.jpg)
Manual pinning• Recent irqbalance 1.x addresses most of the discovered bugs
• But sometime manual pinning is still better• Real-time, HPC
![Page 25: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/25.jpg)
Rules of manual pinning• Don't set affinity mask to all CPUs• Move affinity to device rather than to process• Let the scheduler do its work• Consider faulty hardware
![Page 26: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/26.jpg)
Kernel IRQ balancing
• Dropped by 8b8e8c in 2008• Return is not planned so far• Interrupt locality ideas
![Page 27: Processing of hardware interrupts in Linux · interrupts in Linux Petr Holášek, Red Hat August 17, 2015. HW and kernel. Interrupt](https://reader034.fdocuments.us/reader034/viewer/2022042804/5f552694e386f6304579b831/html5/thumbnails/27.jpg)
Give irqbalance a second chance
• Explore recent version• Some new features are coming soon• Try to compare manual pinning and irqbalance