Yarn调度器学习总结 2015-09-29 21:05

Capacity Scheduler(容量调度器)

模型

假设有如下资源分配模型

无标签:

--root(100%)
  |--default(20%)
  |--department(30%)
    |--bi(60%)
    |--customer_service(30%)
    |--finance(10%)
  |--city(50%)
    |--guangzhou(40%)
    |--shenzhen(40%)
    |--zhuhai(20%)

normail标签(值同无标签场景):

--root(100%)
  |--default(20%)
  |--department(30%)
    |--bi(60%)
    |--customer_service(30%)
    |--finance(10%)
  |--city(50%)
    |--guangzhou(40%)
    |--shenzhen(40%)
    |--zhuhai(20%)

fastcpu标签:

--root(100%)
  |--default(0%)
  |--department(80%)
    |--bi(60%)
    |--customer_service(30%)
    |--finance(10%)
  |--city(20%)
    |--guangzhou(30%)
    |--shenzhen(60%)
    |--zhuhai(10%)

highmem标签:

--root(100%)
  |--default(0%)
  |--department(30%)
    |--bi(60%)
    |--customer_service(30%)
    |--finance(10%)
  |--city(70%)
    |--guangzhou(30%)
    |--shenzhen(60%)
    |--zhuhai(10%)

则配置如下:

yarn-site.xml:

1
2
3
4
<property>
  <name>yarn.resourcemanager.scheduler.class</name>
  <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler</value>
</property>

capacity-scheduler.xml:

注意:很多应用都会提交到default队列(不需要root.default,default支持),因此需要有一个default队列。

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
<property>
  <name>yarn.scheduler.capacity.root.queues</name>
  <value>default,city,department</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.default.capacity</name>
  <value>20</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
  <value>100</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.department.queues</name>
  <value>bi,customer_service,finance</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.department.capacity</name>
  <value>30</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.department.maximum-capacity</name>
  <value>50</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.department.minimum-user-limit-percent</name>
  <value>100</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.department.user-limit-factor</name>
  <value>1</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.department.bi.capacity</name>
  <value>60</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.department.bi.maximum-capacity</name>
  <value>70</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.department.bi.minimum-user-limit-percent</name>
  <value>100</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.department.bi.user-limit-factor</name>
  <value>1</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.department.customer_service.capacity</name>
  <value>30</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.department.customer_service.maximum-capacity</name>
  <value>40</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.department.customer_service.minimum-user-limit-percent</name>
  <value>100</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.department.customer_service.user-limit-factor</name>
  <value>1</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.department.finance.capacity</name>
  <value>10</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.department.finance.maximum-capacity</name>
  <value>20</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.department.finance.minimum-user-limit-percent</name>
  <value>100</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.department.finance.user-limit-factor</name>
  <value>1</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.city.queues</name>
  <value>guangzhou,shenzhen,zhuhai</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.city.capacity</name>
  <value>50</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.city.maximum-capacity</name>
  <value>90</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.city.minimum-user-limit-percent</name>
  <value>100</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.city.user-limit-factor</name>
  <value>1</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.city.guangzhou.capacity</name>
  <value>40</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.city.guangzhou.maximum-capacity</name>
  <value>60</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.city.guangzhou.minimum-user-limit-percent</name>
  <value>100</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.city.guangzhou.user-limit-factor</name>
  <value>1</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.city.shenzhen.capacity</name>
  <value>40</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.city.shenzhen.maximum-capacity</name>
  <value>60</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.city.shenzhen.minimum-user-limit-percent</name>
  <value>100</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.city.shenzhen.user-limit-factor</name>
  <value>1</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.city.zhuhai.capacity</name>
  <value>20</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.city.zhuhai.maximum-capacity</name>
  <value>40</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.city.zhuhai.minimum-user-limit-percent</name>
  <value>100</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.city.zhuhai.user-limit-factor</name>
  <value>1</value>
</property>
<property>
  <name>yarn.scheduler.capacity.root.city.zhuhai.acl_submit_applications</name>
  <value>hadoop,root</value>
</property>

参考文档

  1. Hadoop Best Practices: Scheduling in YARN
Tags: #Yarn    Post on Yarn