Memory Management in Tizen · DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL...

Memory Management in Tizen

SW Platform Team,

SW R&D Center

tizen.org 2

Contents

• Tizen Kernel Overview

• Memory Management in Tizen Kernel

• Memory Size Optimization

tizen.org 3

Tizen Kernel Overview

tizen.org 4


Kernel

Kernel

OAL (OEM Adaptation Layer)

Telephony

Plug-ins

GStreamer

Plug-ins

Sensor

Plug-ins

System

Plug-ins

OpenGL ES/EGL

Graphics Driver

Graphics Memory

Management

IOMMU DMA

BUF

Multimedia Power Management

CPUfre

q

Devf

req

Ther

mal …

DRM

Link1 Link2 V4L2

Storage

Block Layer

MMC/Flash

Input, Sensor,

…

Peripheral

CMA

OpenGL ES 1.0/2.0

X11 GStreamer EFL . . .

Multimedia FW

Telephony FW

. . . Sensor FW System FW

. . .

More at http://source.tizen.org/documentation/porting-guide

Core

DRM Link1: http:// elinux.org/images/7/71/Elce11_dae.pdf

DRM Link2: http://download.tizen.org/misc/media/conference2012/wednesday/ballroom-c/2012-05-09-1330-1410-

the_drm_(direct_rendering_manager)_of_tizen_kernel.pdf

http://elinux.org/images/7/71/Elce11_dae.pdf

http://download.tizen.org/misc/media/conference2012/wednesday/ballroom-c/2012-05-09-1330-1410-the_drm_(direct_rendering_manager)_of_tizen_kernel.pdf

tizen.org 5


Kernel

Kernel

OAL (OEM Adaptation Layer)

Telephony

Plug-ins

GStreamer

Plug-ins

Sensor

Plug-ins

System

Plug-ins

OpenGL ES/EGL

Graphics Driver

Graphics Memory

Management

IOMM

U

DMA

BUF

Multimedia Power Management

CPUfre

q

Devf

req

Ther

mal … DRM

Link1 Link2 V4L2

Storage

Block Layer

MMC/Flash

Input, Sensor,

…

Peripheral

CM

A

OpenGL ES 1.0/2.0

X11 GStreamer EFL . . .

Multimedia FW

Telephony FW

. . . Sensor FW System FW

. . .

More at http://source.tizen.org/documentation/porting-guide

Core Service Layer

DRM Link1: http:// elinux.org/images/7/71/Elce11_dae.pdf

DRM Link2: http://download.tizen.org/misc/media/conference2012/wednesday/ballroom-c/2012-05-09-1330-1410-

the_drm_(direct_rendering_manager)_of_tizen_kernel.pdf

• Memory Management in Tizen

– Coupled with Graphics & Multimedia devices.

• Graphics & Multimedia devices = DMA devices with HUGE buffers

http://elinux.org/images/7/71/Elce11_dae.pdf

http://download.tizen.org/misc/media/conference2012/wednesday/ballroom-c/2012-05-09-1330-1410-the_drm_(direct_rendering_manager)_of_tizen_kernel.pdf

tizen.org 6


Kernel / Device Drivers (BSP)

Kernel / Core & Subsystem

X Server

DRM Framework GEM

Display HDMI Virtual Display

Post Processor

G2D

EFL

Evas

2D GPU backend

X Video Drivers

Userptr

NEON 2D GPU backend

PIXMAN

EXA

V4L2 VB2

Codec Camera

GStreamer

OpenMAX

Libdrm

XV Extension

UMM

Open GL

Multimedia FW

XvImageSink

Core External Library

Core Framework

OEM Adaptation

Kernel

Kernel

GPU DDK

(G3D)

Radio

* Images from MS Office clipart and Samsung

Gstreamer-CameraSrc

DMABUF

tizen.org 7

Contents



– To Run Tizen OS.

• Basic Memory Management Components

• Buffer Sharing (UMM/DMABUF)

• Buffer Allocation (UMM/DMAAPI)

• Buffer Synchronization


tizen.org 8

Tizen Kernel MM, Graphics (DRM)



X Server

DRM Framework GEM


Post Processor

G2D

EFL

Evas

2D GPU backend

X Video Drivers

Userptr

NEON 2D GPU backend

PIXMAN

EXA

V4L2 VB2

Codec Camera

GStreamer

OpenMAX

Libdrm

XV Extension

UMM

Open GL

Multimedia FW

XvImageSink


Core Framework

OEM Adaptation

Kernel

Kernel

GPU DDK

(G3D)

Radio


Gstreamer-CameraSrc

DMABUF

tizen.org 9

Graphics: DRM/GEM (Graphics Execution Manager)

drm_fb_helper drm_mode_config

Linux

DRM Common Framework

GEM

drm_framebuffer fb_info Crtc Connector Encoder

GEM

Allocator

Plane

Common

Specific

drm_framebuffer Crtc Connector Encoder

Plane

tizen.org 10

Graphics: DRM/GEM (Graphics Execution Manager)

drm_fb_helper drm_mode_config

Linux

DRM Common Framework

GEM

drm_framebuffer fb_info Crtc Connector Encoder

GEM

Allocator

Plane

Common

Specific

drm_framebuffer Crtc Connector Encoder

Plane

Graphics Execution Manager • Framework developed by Intel

• To manage graphics memory

• Framework for buffer management. • Allocation and sharing.

tizen.org 11

Graphics: DRM/GEM Allocation

• GEM Allocation steps @ Tizen (Generic)

1. DRM_IOCTL_MODE_CREATE_DUMB

• Create GEM object(global) & user GEM handle(per process)

• dumb_create() of struct drm_driver

• No physical memory allocated.

2. DRM_IOCTL_MODE_MAP_DUMB

• Create fake mmap offset of a gem object and relay the object to user

• A hash key of the gem object.

• dumb_map_offset() of struct drm_driver

3. MMAP

• Request mmap based on the hash key as the offset

• Create user address space

• Setup cache attribute.

• Not mapped to physical memory, yet

tizen.org 12


• GEM Allocation steps @ Tizen (Generic)

4. On-demand Paging

– Implement & Register a fault handler that

• With a page fault, allocate a page and map the page.

– vma->vm_ops->fault = xxx_drm_gem_fault

5. Use!

6. DRM_IOCTL_MODE_DESTROY_DUMB

– Remove GEM handle & object

– Free memory

– Implement dumb_destroy() of struct drm_driver

tizen.org 13


• GEM Allocation steps @ Tizen (Exynos Only)

1. DRM_IOCTL_EXYNOS_GEM_CREATE

• Only use user-desired size and buffer types.

• Create gem object(global) & user gem handle(per process)

• physical memory allocated.

2. DRM_IOCTL_EXYNOS_GEM_MMAP

• Create user address space

• Map the user address space to physical memory

• LIBDRM of Exynos uses these APIs, not the generic.

tizen.org 14

Graphics: DRM/GEM Sharing

• GEM Sharing @ Tizen

• DRM_IOCTL_GEM_FLINK

– “I will share this GEM to others.”

– Create GEM object name for the given GEM handle

• Global key vaue for sharing

• DRM_IOCTL_GEM_OPEN

– “I want to use the shared GEM.”

– Create GEM handle based on the given GEM object name

• DRM_IOCTL_GEM_CLOSE

• You don’t need to implement. It’s already there with DRM.

GEM Create

GEM FLINK

GEM OPEN

gem handle gem object name gem handle

Process 1 Process 2

tizen.org 15

Tizen Kernel MM, Multimedia (V4L2/VB2)



X Server

DRM Framework GEM


Post Processor

G2D

EFL

Evas

2D GPU backend

X Video Drivers

Userptr

NEON 2D GPU backend

PIXMAN

EXA

V4L2 VB2

Codec Camera

GStreamer

OpenMAX

Libdrm

XV Extension

UMM

Open GL

Multimedia FW

XvImageSink


Core Framework

OEM Adaptation

Kernel

Kernel

GPU DDK

(G3D)

Radio


Gstreamer-CameraSrc

DMABUF

tizen.org 16

Multimedia: V4L2/VB2

• Tizen recommends to use V4L2 at Tizen kernel for Multimedia devices • Video input (codec & camera) & Radio

• However, as long as the kernel has:

– Gstreamer/OpenMAX plugins

– A method to share with other F/W via DMABUF of UMM,

• Tizen multimedia works.

• If V4L2/VB2 is used, things get easier.

tizen.org 17

Tizen Kernel MM, OpenGL/G3D-GPU



X Server

DRM Framework GEM


Post Processor

G2D

EFL

Evas

2D GPU backend

X Video Drivers

Userptr

NEON 2D GPU backend

PIXMAN

EXA

V4L2 VB2

Codec Camera

GStreamer

OpenMAX

Libdrm

XV Extension

UMM

Open GL

Multimedia FW

XvImageSink


Core Framework

OEM Adaptation

Kernel

Kernel

GPU DDK

(G3D)

Radio


Gstreamer-CameraSrc

DMABUF

tizen.org 18

OpenGL / G3D-GPU

• Most ARM SoC GPUs (MALI, SGX, …) use their own memory manager • E.g., Exynos4412/4210 Tizen Reference Kernel has Mali w/ UMP.

– Mali DDK modified to be compatible with UMM-DMABUF.

• If GPU drivers use DRM, it would be great. • (and make them GPL)

tizen.org 19

Contents









tizen.org 20

NEED FOR SOMETHING, A Scenario



X Server

DRM Framework GEM


Post Processor

G2D

EFL

Evas

2D GPU backend

X Video Drivers

Userptr

NEON 2D GPU backend

PIXMAN

EXA

V4L2 VB2

Codec Camera

GStreamer

OpenMAX

Libdrm

XV Extension

Open GL

Multimedia FW

XvImageSink


Core Framework

OEM Adaptation

Kernel

Kernel

GPU DDK

(G3D)

Radio


Gstreamer-CameraSrc

Camera fetches a video stream

This scenario is simplified for presentation.

Not an actual example of Tizen

tizen.org 21




X Server

DRM Framework GEM


Post Processor

G2D

EFL

Evas

2D GPU backend

X Video Drivers

Userptr

NEON 2D GPU backend

PIXMAN

EXA

V4L2 VB2

Codec Camera

GStreamer

OpenMAX

Libdrm

XV Extension

Open GL

Multimedia FW

XvImageSink


Core Framework

OEM Adaptation

Kernel

Kernel

GPU DDK

(G3D)

Radio


Gstreamer-CameraSrc


Draws its own images



tizen.org 22




X Server

DRM Framework GEM


Post Processor

G2D

EFL

Evas

2D GPU backend

X Video Drivers

Userptr

NEON 2D GPU backend

PIXMAN

EXA

V4L2 VB2

Codec Camera

GStreamer

OpenMAX

Libdrm

XV Extension

Open GL

Multimedia FW

XvImageSink


Core Framework

OEM Adaptation

Kernel

Kernel

GPU DDK

(G3D)

Radio


Gstreamer-CameraSrc



GPU Merges the two

/ Puts the merged image



tizen.org 23




X Server

DRM Framework GEM


Post Processor

G2D

EFL

Evas

2D GPU backend

X Video Drivers

Userptr

NEON 2D GPU backend

PIXMAN

EXA

V4L2 VB2

Codec Camera

GStreamer

OpenMAX

Libdrm

XV Extension

Open GL

Multimedia FW

XvImageSink


Core Framework

OEM Adaptation

Kernel

Kernel

GPU DDK

(G3D)

Radio


Gstreamer-CameraSrc



DRM-FIMD(display) Output



GPU Merges the two


tizen.org 24




X Server

DRM Framework GEM


Post Processor

G2D

EFL

Evas

2D GPU backend

X Video Drivers

Userptr

NEON 2D GPU backend

PIXMAN

EXA

V4L2 VB2

Codec Camera

GStreamer

OpenMAX

Libdrm

XV Extension

Open GL

Multimedia FW

XvImageSink


Core Framework

OEM Adaptation

Kernel

Kernel

GPU DDK

(G3D)

Radio


Gstreamer-CameraSrc



DRM-FIMD(display) Output

At a VB2 buffer

At a GEM buffer

GEM buffer



GPU Merges the two


From GEM + VB2 into its own (e.g., UMP)

tizen.org 25

Tizen Kernel MM, UMM



X Server

DRM Framework GEM


Post Processor

G2D

EFL

Evas

2D GPU backend

X Video Drivers

Userptr

NEON 2D GPU backend

PIXMAN

EXA

V4L2 VB2

Codec Camera

GStreamer

OpenMAX

Libdrm

XV Extension

UMM

Open GL

Multimedia FW

XvImageSink


Core Framework

OEM Adaptation

Kernel

Kernel

GPU DDK

(G3D)

Radio


Gstreamer-CameraSrc

DMABUF

A mechanism to share between

DRM, V4L2, and others

w/o memcpy

tizen.org 26

Tizen Kernel MM, UMM



X Server

DRM Framework GEM


Post Processor

G2D

EFL

Evas

2D GPU backend

X Video Drivers

Userptr

NEON 2D GPU backend

PIXMAN

EXA

V4L2 VB2

Codec Camera

GStreamer

OpenMAX

Libdrm

XV Extension

UMM

Open GL

Multimedia FW

XvImageSink


Core Framework

OEM Adaptation

Kernel

Kernel

GPU DDK

(G3D)

Radio


Gstreamer-CameraSrc

DMABUF

Requirement from Tizen platform and hardware

• Different Memory Managers: GEM, VB2, GPU-adhoc, …

• Share buffers

• w/o memcpy

• From and to users

• Never expose directly to users (e.g., physical address)

UMM DMABUF!

tizen.org 27

Tizen Kernel Memory Management Unified Memory Management (UMM)

• Introduced by Jesse Barker, 2011

• Includes

– DMABUF (sharing buffer)

– DMA Mapping API for Allocation.

– CMA (Contiguous Memory Allocator) IOMMU (MMU for I/O devs)

tizen.org 28

Tizen Kernel MM, UMM DMA Buffer Sharing: DMABUF

• Export

– GEM/VB2/… object DMABUF

• Import

– DMABUF GEM/VB2/… object

• Userspace sees DMABUF as a File Descriptor

tizen.org 29

DMA Buffer Sharing: DMABUF, Example Camera Display

X server

Video encoder

Xvsin

k

Camera src

User space

Kernel space DRM KMS GEM Codec Camera V4L2 V4L2

struct dma_buf

Display Codec Camera

Memory

Camera App

FD

HW

tizen.org 30

DMA Buffer Sharing: DMABUF, Example Conceptual Data Flow

X server

Video encoder

Xvsin

k

Camera src

User space

Kernel space DRM KMS GEM Codec Camera V4L2 V4L2

struct dma_buf


Memory

Camera App

FD

HW

tizen.org 31

DMABUF Usage Example 1/4

Camera App

1) Request V4L2 camera buffer (U)

2) Allocate CMA buffer (K)

3) Request a camera frame at the V4L2 buffer (U)

4) Store the camera frame & Notify user (K)

5) Request DMABUF export for the V4L2 camera buffer (U)

6) dma_buf_exporter() (K)

- Create DMABUF from V4L2 buffer

7) dma_buf_fd() (K)

- Provide FD of the DMABUF to user

X server

Video encoder

Xvsin

k

Camera src

User space

Kernel space DRM KMS

GEM Codec Camera V4L2 V4L2

struct dma_buf


Memory

Camera App

FD

HW

tizen.org 32


Camera App








7) dma_buf_fd() (K)


X server

Video encoder

Xvsin

k

Camera src

User space



struct dma_buf


Memory

Camera App

FD

HW

tizen.org 33


Camera App








7) dma_buf_fd() (K)


X server

Video encoder

Xvsin

k

Camera src

User space



struct dma_buf


Memory

Camera App

FD

HW

tizen.org 34


Camera App

8) Request FD->GEM conversion (U)

9) dma_buf_get(fd) (K)

- Get DMABUF from FD

10)dma_buf_attach(dma-buf) / dma_buf_map_attachment() (K)

- Get Buffer from DMABUF

11) Import as GEM, send user (K)

12) Request GEM object name (U)

13) Return GEM object name (K)

14) Send GEM object name to X (U)

X server

Video encoder

Xvsin

k

Camera src

User space



struct dma_buf


Memory

Camera App

FD

HW

tizen.org 35


Camera App

8) Request FD->GEM conversion (U)

9) dma_buf_get(fd) (K)

- Get DMABUF from FD

10) dma_buf_attach(dma-buf) / dma_buf_map_attachment() (K)

- Get Buffer from DMABUF

11) Import as GEM, send user (K)

12) Request GEM object name (U)

13) Return GEM object name (K)

14) Send GEM object name to X (U)

X server

Video encoder

Xvsin

k

Camera src

User space



struct dma_buf


Memory

Camera App

FD

HW

tizen.org 36


X server

15) Convert given GEM object name to GEM. Display its content. (U & K)

X server

Video encoder

Xvsin

k

Camera src

User space



struct dma_buf


Memory

Camera App

FD

HW

tizen.org 37


X server

15) Convert given GEM object name to GEM. Display its content. (U & K)

X server

Video encoder

Xvsin

k

Camera src

User space



struct dma_buf


Memory

Camera App

FD

HW

tizen.org 38


Close after usage

1) “Free” the GEM object (U)

2) Remove reference from the DMABUF (K)

3) Close(DMABUF-FD) at cam app (U)

4) (No more reference to DMABUF) Release callback executed (K)

X server

Video encoder

Xvsin

k

Camera src

User space



struct dma_buf


Memory

Camera App

FD

HW

tizen.org 39


Close after usage


2) Remove reference from the DMABUF



X server

Video encoder

Xvsin

k

Camera src

User space



struct dma_buf


Memory

Camera App

FD

HW

tizen.org 40


Close after usage


2) Remove reference from the DMABUF



X server

Video encoder

Xvsin

k

Camera src

User space



struct dma_buf


Memory

Camera App

FD

HW

tizen.org 41

Sorry, Back to DRM/GEM…

• DMABUF Import / Export needs to be added

• DRM_IOCTL_PRIME_HANDLE_TO_FD • Export gem handle into dmabuf fd

• DRM_IOCTL_PRIME_FD_TO_HANDLE • Import dmabuf fd into gem handle

tizen.org 42

Contents









tizen.org 43

Tizen Kernel Memory Management Unified Memory Management (UMM)

• Introduced by Jesse Barker, 2011

• Includes

– DMABUF (sharing buffer)

– DMA Mapping API for Allocation.

– CMA (Contiguous Memory Allocator) IOMMU (MMU for I/O devs)

tizen.org 44

Tizen Kernel MM, UMM DMA Mapping API @ Tizen Reference

DRM / GEM

DMA Mapping Framework

IOMMU Buddy CMA

IOMMU

Common

Driver

System Memory

V4L2 / VB2

1 2

1: “Exynos-DRM” provides unified device address space for all DRM devices

2: Each V4L2 device has its own device address space

2 2

tizen.org 45

Contents









tizen.org 46

Tizen Kernel Memory Management

We Still Have A HUGE Problem!

tizen.org 47

Tizen Kernel MM, Buffer Sync.



X Server

DRM Framework GEM


Post Processor

G2D

EFL

Evas

2D GPU backend

X Video Drivers

Userptr

NEON 2D GPU backend

PIXMAN

EXA

V4L2 VB2

Codec Camera

GStreamer

OpenMAX

Libdrm

XV Extension

UMM

Open GL

Multimedia FW

XvImageSink


Core Framework

OEM Adaptation

Kernel

Kernel

GPU DDK

(G3D)

Radio


Gstreamer-CameraSrc

DMABUF

Synchronization?

tizen.org 48

Tizen Kernel MM, Buffer Sync. What’s the problem?

BUS

DRAM DRAM DRAM

CPU GPU

Simple Usage Scenario

1. CPU writes to buffer1.

2. CPU tells GPU to do something on buffer1.

3. GPU does something on buffer1.

4. GPU finishes.

5. CPU reads the buffer1

tizen.org 49


BUS

DRAM DRAM DRAM

CPU GPU





4. GPU finishes.


tizen.org 50


BUS

DRAM DRAM DRAM

CPU GPU





4. GPU finishes.


tizen.org 51


BUS

DRAM DRAM DRAM

CPU GPU





4. GPU finishes.


tizen.org 52


BUS

DRAM DRAM DRAM

CPU GPU





4. GPU finishes.


tizen.org 53






4. GPU finishes.


How to ensure CPU won’t use buffer 1 until step 4?

(esp., a user process)

tizen.org 54


BUS

DRAM DRAM DRAM

CPU,

Thread1 GPU

• What if there are two threads using GPU?

– And if the two look at the same buffer?

• Not even aware of it?

CPU,

Thread2

tizen.org 55


BUS

DRAM DRAM DRAM

CPU,

Thread1 GPU

• What if the two DMA devices (GPU, FIMC) share buffer?

– But hard to know it at drivers or user threads.

– FIMC never knows when GPU finishes

– FIMC never knows when Threads1 stops using “buffer1”

– Threads2 never know when GPU stops using “buffer1”

CPU,

Thread2 FIMC-IPP

(Image Post Processing)

tizen.org 56

Tizen Kernel Memory Management Buffer Synchronization

• TGL (Tizen Global Lock) @ Tizen 2.0

– … Let userspace handle the issue …

– Kernel patch required.

• Sync Framework (Google)

– Jun, 2012. Resources w/o DMABUF (similar with TGL)

• KDS (Kernel Dependency System, ARM)

– May 2012 / DMABUF-compatible

• DMA Fence Framework (Canonical/TI)

– Aug 2012 / DMABUF-compatible

– Work-In-Progress

tizen.org 57

Contents




– To Run w/ More Devices

tizen.org 58

Memory Size Optimization: Challenge

• A device with 512MB RAM

• Graphics/Multimedia devices want 400MB reserved

– No IOMMU.

• The userspace wants to run a web browser and HTML5 app!

•REALLY POSSIBLE?

tizen.org 59

Memory Size Optimization: Challenge

• A device with 512MB RAM

• Graphics/Multimedia devices want 400MB reserved

– No IOMMU

• The userspace wants to run a web browser and HTML5 app!

•REALLY POSSIBLE?

•WHY NOT?

tizen.org 60

Minimize H/W Reserved Memory CMA (Contiguous Memory Allocator)

• Camera wants to reserve 400MB.

– No IOMMU: too primitive to use paging.

• You don’t use cameras while web browsing.

– Free the 400MB reserved to camera, let userspace use it.

• CMA can do.

tizen.org 61

Minimize H/W Reserved Memory

• (Obsolete/Low-cost) ARM SoC’s DMA devices

– No IOMMU

– Use physically continuous memory chunk.

• No paging

• Potentially, a lot of memory is wasted.

BUS

CPU

MMU

L2 Cache

DRAM Controller

DRAM DRAM DRAM

CPU CPU

GPU

tizen.org 62

Minimize H/W Reserved Memory IOMMU

• (Modern/Mid-High-cost) ARM SoC’s DMA devices

– IOMMU for DMA devices!

– Use paging!

• Dynamically allocate & free

BUS

CPU

MMU

L2 Cache

DRAM Controller

DRAM DRAM DRAM

CPU CPU GPU

IOMMU

tizen.org 63

Minimize H/W Reserved Memory IOMMU

• Exynos4210/4412-based Tizen mobile device

– Achieved near-zero reserved memory.

– At the “menuscreen”, only around ~40MB allocated for display and GPU.

– Unfortunately, not included in the Tizen reference kernel.

• But easily supported by Vanilla kernel for Tizen reference boards

tizen.org 64

Minimize H/W Reserved Memory Results with 1GB RAM Device

-sh-4.1# free

total used free shared buffers cached

Mem: 1029428 302020 727408 0 16164 133084

-/+ buffers/cache: 152772 876656

Swap: 0 0 0

-sh-4.1# uname -a

Linux localhost 3.8.3-00841-gc1981b1 #34 SMP PREEMPT Wed Apr 10 10:24:04 KST 2013 armv7l GNU/Linux

Mostly available to kernel (1005.3 of 1024MB)

Even with Tizen loaded, mostly free to users (856MB of 1GB)

tizen.org 65

Memory Size Optimization: Still Hungry?

• Further optimization required for low-budget devices

– Running full features with 512MB?

– Or even < 512MB?

• What about multi-tasking or “background apps”?

tizen.org 66

Memory Size Optimization: SWAP

• Swap

– zRAM

• Used in some Samsung mobile devices successfully

– Or even swap-to-flash?

tizen.org 67

Memory Size Optimization: Multitasking / Background Apps

• Limit multitasking & background-app support

• Pseudo multitasking by freezing background apps

Thank you!

Memory Management in Tizen · DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL...

Documents

Transcript of Memory Management in Tizen · DRM Framework GEM Display HDMI Virtual Display Post Processor G2D EFL...