Google Protocol Buffer

Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler.

How does protocol buffer work?

By defining the protocol buffer message types in .proto files, we can specify how our information to be serialized and how they are structured. Here is a basic example of a .proto file that defines a message containing information about a person:

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    required string number = 1;
    optional PhoneType type = 2 [default = HOME];
  }

  repeated PhoneNumber phone = 4;
}

Each message type has one or more uniquely numbered fields, and each field has a name and a value type, where values types can be numbers (integer or floating point), booleans, strings, raw bytes, or even other protocol buffer message types, allowing us to structure our data hierarchically. The field can be optional or required.

Once we have the message defined, we can run the protocal buffer compiler for our chosen language on the .proto file to generate data access classes Person. We can then use this class in our application to populate, serialize, and retrieve Person protocol buffer messages.

In Java we can write codes to utilize the Person class like this.

Person john = Person.newBuilder()
    .setId(1234)
    .setName("John Doe")
    .setEmail("jdoe@example.com")
    .build();
output = new FileOutputStream(args[0]);
john.writeTo(output);

And similar codes in C++.

Person john;
fstream input(argv[1],
    ios::in | ios::binary);
john.ParseFromIstream(&input);
id = john.id();
name = john.name();
email = john.email();

With Protocol Buffer, we can add new fields to our message formats without breaking backwards-compatibility; old binaries simply ignore the new field when parsing. Therefore, we can extend our protocol without worrying about breaking existing code.

Basic Steps

  1. Define message formats in a .proto file.
  2. Use the protocol buffer compiler.
  3. Use the Java/C++/Python protocol buffer API to write and read messages.

Define message formats in a .proto file

The definitions in a .proto file are simple: we add a message for each data structure we want to serialize, then specify a name and a type for each field in the message. Let's use the an address book application as an example. To define our messages, we start with the addressbook.proto.

package tutorial;

option java_package = "com.example.tutorial";
option java_outer_classname = "AddressBookProtos";

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;

  enum PhoneType {
    MOBILE = 0;
    HOME = 1;
    WORK = 2;
  }

  message PhoneNumber {
    required string number = 1;
    optional PhoneType type = 2 [default = HOME];
  }

  repeated PhoneNumber phone = 4;
}

message AddressBook {
  repeated Person person = 1;
}

package & java_package

The .proto file starts with a package declaration package tutorial, which helps to prevent name conflicting. It will by default used as the Java package unless we explicitly specify the java_package as shown above. It is recommended to always specify a package to avoid name collisions in Protocol Buffer name spaces even in non-Java languages.

java_outer_classname

The java_outer_classname option defines the class name which should contain all of the classes in this file. If we don't give a java_outer_classname explicitly, it will be generated by converting the file name to camel case. For example, "my_proto.proto" would, by default, use "MyProto" as the outer class name.

Message

Next, we have the message definitions. A message is just an aggregate containing a set of typed fields. Several basic types are available: bool, int32, float, double and string.

Complex message type

A more complicated structure is supported. As shown above, Person message contains PhoneNumber messages, while the AddressBook message contains Person messages. We can even define message types nested inside other messages – as we can see, the PhoneNumber type is defined inside Person. Enum is also supported– here we have a phone number that can be one of MOBILE, HOME, or WORK.

Number tag

The " = 1", " = 2" markers on each element identify the unique "tag" that field uses in the binary encoding. Tag numbers 1-15 require one less byte to encode than higher numbers, so as an optimization we can decide to use those tags for the commonly used or repeated elements, leaving tags 16 and higher for less-commonly used optional elements. Each element in a repeated field requires re-encoding the tag number, so repeated fields are particularly good candidates for this optimization.

Required Optional Repreated

Each field must be annotated with one of the following modifiers:

  • required: a value for the field must be provided, otherwise the message will be considered "uninitialized". Trying to build an uninitialized message will throw a RuntimeException. Parsing an uninitialized message will throw an IOException.
  • optional: the field may or may not be set. If an optional field value isn't set, a default value is used. For simple types, we can specify our own default value, as we've done for the phone number type in the example. Otherwise, a system default is used: zero for numeric types, the empty string for strings, false for bools.
  • repeated: the field may be repeated any number of times (including zero). The order of the repeated values will be preserved in the protocol buffer.

More

We'll find a complete guide to writing .proto files – including all the possible field types – in the Protocol Buffer Language Guide.

Compiling Protocol Buffers

Now that we have a .proto, the next thing we need to do is generate the classes we'll need to read and write AddressBook messages. To do this, we need to run the protocol buffer compiler protoc on our .proto:

  • If we haven't installed the compiler, download the package.
  • Now run the compiler, specifying the source directory (where our application's source code lives with the current directory as default), the destination directory (where we want the generated code to go; often the same as the source directory), and the path to our .proto.
protoc -I=$SRC_DIR --java_out=$DST_DIR $SRC_DIR/addressbook.proto

Assuming we want Java classes, we use the --java_out option here. Similar options are provided for other supported languages.

This generates com/example/tutorial/AddressBookProtos.java in our specified destination directory.

The Protocol Buffer API - Java

If we look in AddressBookProtos.java, we can see that it defines a class called AddressBookProtos, nested within which is a class for each message we specified in addressbook.proto. Each class has its own Builder class that we use to create instances of that class.

Here are some of the accessors for the Person class.

// required string name = 1;
public boolean hasName();
public String getName();

// required int32 id = 2;
public boolean hasId();
public int getId();

// optional string email = 3;
public boolean hasEmail();
public String getEmail();

// repeated .tutorial.Person.PhoneNumber phone = 4;
public List<PhoneNumber> getPhoneList();
public int getPhoneCount();
public PhoneNumber getPhone(int index);

Meanwhile, Person.Builder has the same getters plus setters:

// required string name = 1;
public boolean hasName();
public java.lang.String getName();
public Builder setName(String value);
public Builder clearName();

// required int32 id = 2;
public boolean hasId();
public int getId();
public Builder setId(int value);
public Builder clearId();

// optional string email = 3;
public boolean hasEmail();
public String getEmail();
public Builder setEmail(String value);
public Builder clearEmail();

// repeated .tutorial.Person.PhoneNumber phone = 4;
public List<PhoneNumber> getPhoneList();
public int getPhoneCount();
public PhoneNumber getPhone(int index);
public Builder setPhone(int index, PhoneNumber value);
public Builder addPhone(PhoneNumber value);
public Builder addAllPhone(Iterable<PhoneNumber> value);
public Builder clearPhone();

As we can see, there are simple JavaBeans-style getters and setters for each field.

Create instances

Here is an example about how to create an instance of Person.

Person john = Person.newBuilder()
    .setId(1234)
    .setName("John Doe")
    .setEmail("jdoe@example.com")
    .addPhone(
      Person.PhoneNumber.newBuilder()
        .setNumber("555-4321")
        .setType(Person.PhoneType.HOME))
    .build();

Serialization and Deserialization

To persist the data, we can simply run the following codes.

Person john = Person.newBuilder()
    .setId(1234)
    .setName("John Doe")
    .setEmail("jdoe@example.com")
    .addPhone(
      Person.PhoneNumber.newBuilder()
        .setNumber("555-4321")
        .setType(Person.PhoneType.HOME))
    .build();

// Write to file
FileOutputStream output = new FileOutputStream("target/person.ser");  
john.writeTo(output);          
output.close();

Once persisted, we can read the data as such.

// Read from file
Person person = Person.parseFrom(new FileInputStream("target/person.ser");

The Protocol Buffer API - C++

For example in C++, we can write codes like:

Person person;
person.set_name("John Doe");
person.set_id(1234);
person.set_email("jdoe@example.com");
fstream output("myfile", ios::out | ios::binary);
person.SerializeToOstream(&output);

Then later on we can read the message back like:

fstream input("myfile", ios::in | ios::binary);
Person person;
person.ParseFromIstream(&input);
cout << "Name: " << person.name() << endl;
cout << "E-mail: " << person.email() << endl;

Check out this post for more information:)

Reference

Google Developer Protocol Buffer

最后編輯于
?著作權(quán)歸作者所有,轉(zhuǎn)載或內(nèi)容合作請聯(lián)系作者
  • 序言:七十年代末阅酪,一起剝皮案震驚了整個濱河市喜最,隨后出現(xiàn)的幾起案子嘁灯,更是在濱河造成了極大的恐慌,老刑警劉巖瓷式,帶你破解...
    沈念sama閱讀 222,729評論 6 517
  • 序言:濱河連續(xù)發(fā)生了三起死亡事件坪稽,死亡現(xiàn)場離奇詭異赌躺,居然都是意外死亡,警方通過查閱死者的電腦和手機峰档,發(fā)現(xiàn)死者居然都...
    沈念sama閱讀 95,226評論 3 399
  • 文/潘曉璐 我一進店門,熙熙樓的掌柜王于貴愁眉苦臉地迎上來寨昙,“玉大人讥巡,你說我怎么就攤上這事√蚰模” “怎么了欢顷?”我有些...
    開封第一講書人閱讀 169,461評論 0 362
  • 文/不壞的土叔 我叫張陵,是天一觀的道長尸红。 經(jīng)常有香客問我吱涉,道長,這世上最難降的妖魔是什么外里? 我笑而不...
    開封第一講書人閱讀 60,135評論 1 300
  • 正文 為了忘掉前任怎爵,我火速辦了婚禮,結(jié)果婚禮上盅蝗,老公的妹妹穿的比我還像新娘鳖链。我一直安慰自己,他們只是感情好墩莫,可當我...
    茶點故事閱讀 69,130評論 6 398
  • 文/花漫 我一把揭開白布芙委。 她就那樣靜靜地躺著,像睡著了一般狂秦。 火紅的嫁衣襯著肌膚如雪灌侣。 梳的紋絲不亂的頭發(fā)上,一...
    開封第一講書人閱讀 52,736評論 1 312
  • 那天裂问,我揣著相機與錄音侧啼,去河邊找鬼牛柒。 笑死,一個胖子當著我的面吹牛痊乾,可吹牛的內(nèi)容都是我干的皮壁。 我是一名探鬼主播,決...
    沈念sama閱讀 41,179評論 3 422
  • 文/蒼蘭香墨 我猛地睜開眼哪审,長吁一口氣:“原來是場噩夢啊……” “哼蛾魄!你這毒婦竟也來了?” 一聲冷哼從身側(cè)響起湿滓,我...
    開封第一講書人閱讀 40,124評論 0 277
  • 序言:老撾萬榮一對情侶失蹤滴须,失蹤者是張志新(化名)和其女友劉穎,沒想到半個月后茉稠,有當?shù)厝嗽跇淞掷锇l(fā)現(xiàn)了一具尸體描馅,經(jīng)...
    沈念sama閱讀 46,657評論 1 320
  • 正文 獨居荒郊野嶺守林人離奇死亡,尸身上長有42處帶血的膿包…… 初始之章·張勛 以下內(nèi)容為張勛視角 年9月15日...
    茶點故事閱讀 38,723評論 3 342
  • 正文 我和宋清朗相戀三年而线,在試婚紗的時候發(fā)現(xiàn)自己被綠了铭污。 大學(xué)時的朋友給我發(fā)了我未婚夫和他白月光在一起吃飯的照片。...
    茶點故事閱讀 40,872評論 1 353
  • 序言:一個原本活蹦亂跳的男人離奇死亡膀篮,死狀恐怖嘹狞,靈堂內(nèi)的尸體忽然破棺而出,到底是詐尸還是另有隱情誓竿,我是刑警寧澤磅网,帶...
    沈念sama閱讀 36,533評論 5 351
  • 正文 年R本政府宣布,位于F島的核電站筷屡,受9級特大地震影響涧偷,放射性物質(zhì)發(fā)生泄漏。R本人自食惡果不足惜毙死,卻給世界環(huán)境...
    茶點故事閱讀 42,213評論 3 336
  • 文/蒙蒙 一燎潮、第九天 我趴在偏房一處隱蔽的房頂上張望。 院中可真熱鬧扼倘,春花似錦确封、人聲如沸。這莊子的主人今日做“春日...
    開封第一講書人閱讀 32,700評論 0 25
  • 文/蒼蘭香墨 我抬頭看了看天上的太陽。三九已至纠拔,卻和暖如春秉剑,著一層夾襖步出監(jiān)牢的瞬間,已是汗流浹背稠诲。 一陣腳步聲響...
    開封第一講書人閱讀 33,819評論 1 274
  • 我被黑心中介騙來泰國打工秃症, 沒想到剛下飛機就差點兒被人妖公主榨干…… 1. 我叫王不留候址,地道東北人吕粹。 一個月前我還...
    沈念sama閱讀 49,304評論 3 379
  • 正文 我出身青樓种柑,卻偏偏與公主長得像,于是被迫代替她去往敵國和親匹耕。 傳聞我的和親對象是個殘疾皇子聚请,可洞房花燭夜當晚...
    茶點故事閱讀 45,876評論 2 361

推薦閱讀更多精彩內(nèi)容