Hadoop partitioner及自定义partitioner

xiaoxiao2021-02-28  127

一、hadoop partitioner

所有partitioner都继承自抽象类Partitioner ,实现getPartition(KEY var1, VALUE var2, intvar3),hadoop自带的partitioner有:

(1)TotalOrderPartitioner

一般做全局排序的时候使用

(2)KeyFieldBasedPartitioner

(3)BinaryPartitioner public int getPartition(BinaryComparable key, V value, int numPartitions) { int length = key.getLength(); int leftIndex = (this.leftOffset + length) % length; int rightIndex = (this.rightOffset + length) % length; int hash = WritableComparator.hashBytes(key.getBytes(), leftIndex, rightIndex - leftIndex + 1); return (hash & 2147483647) % numPartitions; }(4)HashPartitioner 默认分区,通过ke的hash值确定分区public int getPartition(K key, V value, int numReduceTasks) { return (key.hashCode() & 2147483647) % numReduceTasks; } 二、自定义paritionerkey为id,实现按key的末三位为分区public class RTFPartitioner extends Partitioner<Text, Text> { @Override public int getPartition(Text key, Text value, int numPartitions) { String k = key.toString(); int par = Integer.parseInt(k.substring(k.length() - 2)); return par; } }

转载请注明原文地址: https://www.6miu.com/read-40664.html

最新回复(0)