CPU scheduling and enforcement

(screenshots of live demo)

Andrew FergusonJuly 10, 2012

Cores in the web GUI

Cores in the web GUI

Cores in the web GUI

Unfair sharing

Start four threads spinning

Unfair sharing

Start four threads spinning

Then, a second 16threads spinning

Unfair sharing

Start four threads spinning

Then, a second 16threads spinning

Second process gets more CPU

Let’s use Kitten!

Let’s use Kitten!

Same programs as before

Let’s use Kitten!

Same programs as before

Added: CPU cores request

Playing with YARN

Launch oneapplication in YARN

Playing with YARN

Launch oneapplication in YARN Gets all cores, even though

requested only one

Playing with YARN

Launch second application, again with more threads

Playing with YARN

Now CPU is shared equally!

Launch second application, again with more threads

Playing with YARN

Playing with YARN

Request all the cores

two cores per containerx two containers = 4 cores

Request all the cores

First applicationis running

Request all the cores

First applicationis running

Second applicationIs waiting

Request all the cores

First applicationis running

Second applicationIs waiting

First application has all the cores

Request all the cores

How does this work?

Pieces of YARN

ResourceManager

NodeManager

NodeManager

NodeManager

ApplicationMaster

Pieces of YARN

ResourceManager

NodeManager

NodeManager

NodeManager

ApplicationMaster

Pieces of YARN

ResourceManager

NodeManager

NodeManager

NodeManager

ApplicationMaster

Node Manager Internals

Launched byContainerExecutor

Node Manager

Node Manager Internals

Launched byContainerExecutor

Node ManagerContainersMonitor

1. Monitors memory usage, and kills tasks over limit

2. Reports killed or failed tasks back to ResourceManager

3. Deletes temporary files created by executor

Node Manager Internals

Launched byContainerExecutor

Node ManagerContainersMonitor

1. Monitors memory usage, and kills tasks over limit

2. Reports killed or failed tasks back to ResourceManager

3. Deletes temporary files created by executor

Made pluggable,like executor, in

MAPREDUCE-4351

Node Manager Internals

Launched byContainerExecutor

Node ManagerContainersMonitor

1. Creates Cgroup for container, places processes in it

2. Reports killed or failed tasks back to ResourceManager

3. Deletes temporary files created by executor

4. Deletes Cgroup when finished

CgroupsCMin MAPREDUCE-4334

Cgroups hierarchycpu memory

Controller(find by reading

mount table)

$ sudo mount -t cgroup -o memory none /cgroups/mem

Cgroups hierarchycpu memory

hadoop-yarn . . . hadoop-yarn . . .

Controller(find by reading

mount table)

Top ofHierarchy

yarn.nodemanager.cgroups.path

$ sudo cgcreate -a hadoop_user_name -g memory:hadoop-yarn

Cgroups hierarchycpu memory

hadoop-yarn . . .

container_..._001

container_..._002

container_..._003

. . .

hadoop-yarn . . . container_..._001

container_..._002

container_..._003

. . .

Controller(find by reading

mount table)

Top ofHierarchy

yarn.nodemanager.cgroups.path

Container Cgroup

CPU processfrom before

IsolatedCPU cgroup