Spark函数之foreach和foreachPartition 2015-08-21 21:10

foreach

foreach是对RDD中的每一个元素执行指定的函数。

1
2
3
4
5
6
val c = sc.parallelize(List("cat", "dog", "tiger", "lion"), 2)
c.foreach(x => println(x + "s are yummy"))
cats are yummy
dogs are yummy
tigers are yummy
lions are yummy

foreachPartition

是对RDD中的每一个Partition执行指定的函数。

1
2
3
4
5
val b = sc.parallelize(List(1, 2, 3, 4, 5, 6, 7, 8, 9), 3)
b.foreachPartition(x => println(x.reduce(_ + _)))
6
15
24
Tags: #Spark    Post on Spark-API