Yarn核心——Protobuf

Yarn将Protobuf用在RPC通信中，基于Protobuf的方式进行通信。Yarn RPC中的所有参数都采用Protocol Buffer进行序列化和反序列化，体积小，速度快。

(1)Proto文件 student.proto

package tutorial; option java_package="com.jackniu.hadoop_yarn.com.proto"; option java_outer_classname="StudentProtos"; message Student{ required int32 ID=1; required string name=2; required string sex=3; message StudentPhone{ required string number=1; optional int32 type=2; } repeated StudentPhone phones=4; }

(2) 编译

bin/protoc --java_out=javacode code/student.proto

(3) 编译完成部分代码

package com.jackniu.hadoop_yarn.com.proto; public final class StudentProtos { private StudentProtos() {} public static void registerAllExtensions( com.google.protobuf.ExtensionRegistry registry) { } public interface StudentOrBuilder extends com.google.protobuf.MessageOrBuilder { // required int32 ID = 1; /** * <code>required int32 ID = 1;</code> */ boolean hasID(); /** * <code>required int32 ID = 1;</code> */ int getID();

(4) 使用

package com.jackniu.hadoop_yarn.com.proto; /** * Created by JackNiu on 2017/9/1. */ public class ProtocolBufferExample { public static void main(String[] args) { StudentProtos.Student student = StudentProtos.Student.newBuilder().setName("jack").setID(1).setSex("2") .addPhones(StudentProtos.Student.StudentPhone.newBuilder().setNumber("12345").setType(1)) .addPhones(StudentProtos.Student.StudentPhone.newBuilder().setNumber("67890").setType(2)) .build(); System.out.println(student); // 在发送端发送 } } ID: 1 name: "jack" sex: "2" phones { number: "12345" type: 1 } phones { number: "67890" type: 2 }

(5) 发送和读取

package com.jackniu.hadoop_yarn.com.proto; import org.apache.hadoop.mapred.FileOutputFormat; import java.io.FileInputStream; import java.io.FileOutputStream; /** * Created by JackNiu on 2017/9/1. */ public class ProtocolBufferExample { public static void main(String[] args) throws Exception{ StudentProtos.Student student = StudentProtos.Student.newBuilder().setName("jack").setID(1).setSex("2") .addPhones(StudentProtos.Student.StudentPhone.newBuilder().setNumber("12345").setType(1)) .addPhones(StudentProtos.Student.StudentPhone.newBuilder().setNumber("67890").setType(2)) .build(); System.out.println(student); //在发送端发送 FileOutputStream output = new FileOutputStream("example.txt"); student.writeTo(output); output.close(); FileInputStream inputStream = new FileInputStream("example.txt"); StudentProtos.Student receive_student = StudentProtos.Student.parseFrom(inputStream); System.out.println(receive_student); } } 输出: example.txt jack 2" 12345 " 67890 控制台输出 ID: 1 name: "jack" sex: "2" phones { number: "12345" type: 1 } phones { number: "67890" type: 2 } ID: 1 name: "jack" sex: "2" phones { number: "12345" type: 1 } phones { number: "67890" type: 2 }

Yarn中通信都是通过protobuf定义的，默认实现protobuf。