最近研究了下google protobuf協(xié)議,順便對(duì)比了一下json,xml,java序列化相關(guān)的數(shù)據(jù)對(duì)比,從幾個(gè)緯度進(jìn)行對(duì)比。
別人的相關(guān)測(cè)試數(shù)據(jù):
http://code.google.com/p/thrift-protobuf-compare/wiki/Benchmarking測(cè)試緯度
序列化時(shí)間
反序列化時(shí)間
bytes大小
測(cè)試代碼
準(zhǔn)備protobuf文件
Message.proto文件代碼
import "InnerMessage.proto";
package demo;
option java_package = "com.agapple.protobuf.data";
option java_outer_classname = "MessageProtos";
option optimize_for = SPEED ; //CODE_SIZE,LITE_RUNTIME
option java_generic_services = false;
message Message {
required string strObj = 1 [default="hello"];
optional int32 int32Obj = 2;
optional int64 int64Obj = 3;
optional uint32 uint32Obj = 4;
optional uint64 uint64Obj = 5;
optional sint32 sint32Obj = 6;
optional sint64 sint64Obj = 7;
optional fixed32 fixed32Obj = 8;
optional fixed64 fixed64Obj = 9;
optional sfixed32 sfixed32Obj = 10;
optional sfixed64 sfixed64Obj = 11;
optional bool boolObj = 12;
optional bytes bytesObj = 13;
optional float folatObj = 14 [deprecated=true];
repeated double doubleObj = 15 [packed=true]; //
optional InnerMessage innerMessage = 16;
}
Innermessage.proto代碼
import "EnumType.proto";
package demo;
option java_package = "com.agapple.protobuf.data";
option java_outer_classname = "InnerMessageProtos";
message InnerMessage {
optional string name = 1 [default = "name"];
optional int32 id = 2;
optional EnumType type = 3 [default = UNIVERSAL];
}
Enumtype.proto代碼
package demo;
option java_package = "com.agapple.protobuf.data";
option java_outer_classname = "EnumTypeProtos";
enum EnumType {
UNIVERSAL = 0;
WEB = 1;
IMAGES = 2;
LOCAL = 3;
NEWS = 4;
PRODUCTS = 5;
VIDEO = 6;
}
基本上把protobuf支持的類型都囊括了,包括嵌套類型,枚舉類型,以及各種int,uint,bool,bytes。
依賴關(guān)系是Message.proto依賴了InnerMessage對(duì)象,而InnerMessage對(duì)象里包含了一個(gè)自定義枚舉類型EnumType。
關(guān)于類型的使用可參見:
http://code.google.com/intl/zh/apis/protocolbuffers/docs/reference/java-generated.htmlhttp://code.google.com/intl/zh/apis/protocolbuffers/docs/proto.html生成protobuf javabean
C代碼
cd /home/ljh/work/code/src/main/java
/home/ljh/work/protobuf/bin/protoc --proto_path=com/agapple/protobuf/ --java_out=. com/agapple/protobuf/EnumType.proto
/home/ljh/work/protobuf/bin/protoc --proto_path=com/agapple/protobuf/ --java_out=. com/agapple/protobuf/InnerMessage.proto
/home/ljh/work/protobuf/bin/protoc --proto_path=com/agapple/protobuf/ --java_out=. com/agapple/protobuf/Message.proto
通過protobuf自帶的protoc進(jìn)行編譯,指定了protobuf文件的路徑, 具體的文檔:
http://code.google.com/intl/zh/apis/protocolbuffers/docs/proto.html#generating運(yùn)行腳本后就會(huì)生成對(duì)應(yīng)的3個(gè)javabean文件: MessageProtos , InnerMessageProtos , EnumTypeProtos。
最后構(gòu)造測(cè)試的protobuf bean代碼
Java代碼
private static MessageProtos.Message getProtobufBean() {
com.agapple.protobuf.data.MessageProtos.Message.Builder messageBuilder = MessageProtos.Message.newBuilder();
messageBuilder.setStrObj("message");
messageBuilder.setFolatObj(1f);
messageBuilder.addDoubleObj(1d);
messageBuilder.addDoubleObj(2d);
messageBuilder.setBoolObj(true);
messageBuilder.setBytesObj(ByteString.copyFrom(new byte[] { 1, 2, 3 }));
messageBuilder.setInt32Obj(32);
messageBuilder.setInt64Obj(64l);
messageBuilder.setSint32Obj(232);
messageBuilder.setSint64Obj(264);
messageBuilder.setFixed32Obj(532);
messageBuilder.setFixed64Obj(564);
messageBuilder.setSfixed32Obj(2532);
messageBuilder.setSfixed64Obj(2564);
messageBuilder.setUint32Obj(632);
messageBuilder.setUint64Obj(664);
com.agapple.protobuf.data.InnerMessageProtos.InnerMessage.Builder innerMessageBuilder = InnerMessageProtos.InnerMessage.newBuilder();
innerMessageBuilder.setId(1);
innerMessageBuilder.setName("inner");
innerMessageBuilder.setType(EnumType.PRODUCTS);
messageBuilder.setInnerMessage(innerMessageBuilder);
return messageBuilder.build();
}
準(zhǔn)備純Pojo Bean
同樣的,為了和json , xml以及java序列化有個(gè)很好的對(duì)比,新建了3個(gè)純的pojo bean: MessagePojo , InnerMessagePojo , EnumTypePojo。
屬性和proto的bean保持一致。
構(gòu)建bean對(duì)象
Java代碼
private static MessagePojo getPojoBean() {
MessagePojo bean = new MessagePojo();
bean.setStrObj("message");
bean.setFolatObj(1f);
List<Double> doubleObj = new ArrayList<Double>();
doubleObj.add(1d);
doubleObj.add(2d);
bean.setDoubleObj(doubleObj);
bean.setBoolObj(true);
bean.setBytesObj(new byte[] { 1, 2, 3 });
bean.setInt32Obj(32);
bean.setInt64Obj(64l);
bean.setSint32Obj(232);
bean.setSint64Obj(264);
bean.setFixed32Obj(532);
bean.setFixed64Obj(564);
bean.setSfixed32Obj(2532);
bean.setSfixed64Obj(2564);
bean.setUint32Obj(632);
bean.setUint64Obj(664);
InnerMessagePojo innerMessagePojo = new InnerMessagePojo();
innerMessagePojo.setId(1);
innerMessagePojo.setName("inner");
innerMessagePojo.setType(EnumTypePojo.PRODUCTS);
bean.setInnerMessage(innerMessagePojo);
return bean;
}
具體的測(cè)試代碼
定義測(cè)試Template接口
Java代碼
interface TestCallback {
String getName();
byte[] writeObject(Object source);
Object readObject(byte[] bytes);
}
統(tǒng)一的測(cè)試模板
Java代碼
private static void testTemplate(TestCallback callback, Object source, int count) {
int warmup = 10;
// 先進(jìn)行預(yù)熱,加載一些類,避免影響測(cè)試
for (int i = 0; i < warmup; i++) {
byte[] bytes = callback.writeObject(source);
callback.readObject(bytes);
}
restoreJvm(); // 進(jìn)行GC回收
// 進(jìn)行測(cè)試
long start = System.nanoTime();
long size = 0l;
for (int i = 0; i < count; i++) {
byte[] bytes = callback.writeObject(source);
size = size + bytes.length;
callback.readObject(bytes);
// System.out.println(callback.readObject(bytes));
bytes = null;
}
long nscost = (System.nanoTime() - start);
System.out.println(callback.getName() + " total cost=" + integerFormat.format(nscost) + "ns , each cost="
+ integerFormat.format(nscost / count) + "ns , and byte sizes = " + size / count);
restoreJvm();// 進(jìn)行GC回收
}
在測(cè)試模板方法中,使用了warmup預(yù)熱的概念,就是預(yù)先執(zhí)行目標(biāo)方法一定的次數(shù),用于避免因?yàn)閖it的優(yōu)化影響系統(tǒng)測(cè)試。 同時(shí)包含了每次測(cè)試模板調(diào)用完成后system.gc保證下一輪的功能測(cè)試
相應(yīng)的restoreJvm方法:
Java代碼
private static void restoreJvm() {
int maxRestoreJvmLoops = 10;
long memUsedPrev = memoryUsed();
for (int i = 0; i < maxRestoreJvmLoops; i++) {
System.runFinalization();
System.gc();
long memUsedNow = memoryUsed();
// break early if have no more finalization and get constant mem used
if ((ManagementFactory.getMemoryMXBean().getObjectPendingFinalizationCount() == 0)
&& (memUsedNow >= memUsedPrev)) {
break;
} else {
memUsedPrev = memUsedNow;
}
}
}
private static long memoryUsed() {
Runtime rt = Runtime.getRuntime();
return rt.totalMemory() - rt.freeMemory();
}
最后相應(yīng)的測(cè)試?yán)樱?div style="height:15px;">
Java代碼
final int testCount = 1000 * 500;
final MessageProtos.Message protoObj = getProtobufBean();
final MessagePojo pojoOBj = getPojoBean();
// Serializable測(cè)試
testTemplate(new TestCallback() {
public String getName() {
return "Serializable Test";
}
@Override
public byte[] writeObject(Object source) {
try {
ByteArrayOutputStream bout = new ByteArrayOutputStream();
ObjectOutputStream output = new ObjectOutputStream(bout);
output.writeObject(source);
return bout.toByteArray();
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
@Override
public Object readObject(byte[] bytes) {
try {
ByteArrayInputStream bin = new ByteArrayInputStream(bytes);
ObjectInputStream input = new ObjectInputStream(bin);
return input.readObject();
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
}, pojoOBj, testCount);
// protobuf測(cè)試
testTemplate(new TestCallback() {
public String getName() {
return "protobuf test";
}
@Override
public byte[] writeObject(Object source) {
if (source instanceof MessageProtos.Message) {
MessageProtos.Message message = (MessageProtos.Message) source;
return message.toByteArray();
}
return null;
}
@Override
public Object readObject(byte[] bytes) {
try {
return MessageProtos.Message.parseFrom(bytes);
} catch (InvalidProtocolBufferException e) {
e.printStackTrace();
}
return null;
}
}, protoObj, testCount);
// json測(cè)試
final ObjectMapper objectMapper = new ObjectMapper();
final JavaType javaType = TypeFactory.type(pojoOBj.getClass());
// JSON configuration not to serialize null field
objectMapper.getSerializationConfig().setSerializationInclusion(JsonSerialize.Inclusion.NON_NULL);
// JSON configuration not to throw exception on empty bean class
objectMapper.getSerializationConfig().disable(SerializationConfig.Feature.FAIL_ON_EMPTY_BEANS);
// JSON configuration for compatibility
objectMapper.configure(Feature.ALLOW_UNQUOTED_FIELD_NAMES, true);
objectMapper.configure(Feature.ALLOW_UNQUOTED_CONTROL_CHARS, true);
testTemplate(new TestCallback() {
public String getName() {
return "Jackson Test";
}
@Override
public byte[] writeObject(Object source) {
try {
return objectMapper.writeValueAsBytes(source);
} catch (JsonGenerationException e) {
e.printStackTrace();
} catch (JsonMappingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
@Override
public Object readObject(byte[] bytes) {
try {
return objectMapper.readValue(bytes, 0, bytes.length, javaType);
} catch (JsonParseException e) {
e.printStackTrace();
} catch (JsonMappingException e) {
e.printStackTrace();
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
}, pojoOBj, testCount);
// Xstream測(cè)試
final XStream xstream = new XStream();
testTemplate(new TestCallback() {
public String getName() {
return "Xstream test";
}
@Override
public byte[] writeObject(Object source) {
return xstream.toXML(source).getBytes();
}
@Override
public Object readObject(byte[] bytes) {
return xstream.fromXML(new ByteArrayInputStream(bytes));
}
}, pojoOBj, testCount);
2011年3月10號(hào)補(bǔ)充 =========================================================
增加了hessian 3.1.5版本基于二進(jìn)制序列化的測(cè)試
Xml代碼
<dependency>
<groupId>com.caucho</groupId>
<artifactId>hessian</artifactId>
<version>3.1.5</version>
</dependency>
測(cè)試了3種情況:
hessian 2協(xié)議
hessian 2協(xié)議 + deflat壓縮
hessian 1協(xié)議
測(cè)試代碼:
Java代碼
// hessian 2 with no deflat
testTemplate(new TestCallback() {
public String getName() {
return "hessian 2 with no deflat";
}
@Override
public byte[] writeObject(Object source) {
try {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
Hessian2Output out = new Hessian2Output(bos);
// out.startMessage();
out.writeObject(source);
// out.completeMessage();
out.flush();
return bos.toByteArray();
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
@Override
public Object readObject(byte[] bytes) {
try {
ByteArrayInputStream bin = new ByteArrayInputStream(bytes);
Hessian2Input in = new Hessian2Input(bin);
// in.startMessage();
Object obj = in.readObject();
// in.completeMessage();
return obj;
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
}, pojoOBj, testCount);
// hessian 2 with deflat
final Deflation envelope = new Deflation();
testTemplate(new TestCallback() {
public String getName() {
return "hessian 2 with deflat";
}
@Override
public byte[] writeObject(Object source) {
try {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
Hessian2Output out = new Hessian2Output(bos);
out = envelope.wrap(out);
out.writeObject(source);
out.flush();
out.close(); // 記得關(guān)閉
return bos.toByteArray();
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
@Override
public Object readObject(byte[] bytes) {
try {
ByteArrayInputStream bin = new ByteArrayInputStream(bytes);
Hessian2Input in = new Hessian2Input(bin);
in = envelope.unwrap(in);
Object obj = in.readObject();
in.close();
return obj;
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
}, pojoOBj, testCount);
// hessian 1 with no deflat
testTemplate(new TestCallback() {
public String getName() {
return "hessian 1 with no deflat";
}
@Override
public byte[] writeObject(Object source) {
try {
ByteArrayOutputStream bos = new ByteArrayOutputStream();
HessianOutput out = new HessianOutput(bos);
out.writeObject(source);
out.flush();
return bos.toByteArray();
} catch (Exception e) {
e.printStackTrace();
}
return null;
}
@Override
public Object readObject(byte[] bytes) {
try {
ByteArrayInputStream bin = new ByteArrayInputStream(bytes);
HessianInput in = new HessianInput(bin);
Object obj = in.readObject();
in.close();
return obj;
} catch (IOException e) {
e.printStackTrace();
}
return null;
}
}, pojoOBj, testCount);
測(cè)試結(jié)果
序列化數(shù)據(jù)對(duì)比
bytes字節(jié)數(shù)對(duì)比
具體的數(shù)字:
protobufjacksonxstreamSerializablehessian2hessian2壓縮hessian1
序列化(單位ns)11545421 92406 101892679410076629027
反序列化(單位ns)13348743 117329 640273787118843237596
bytes97311 664 824374283495
protobuf 不管是處理時(shí)間上,還是空間占用上都優(yōu)于現(xiàn)有的其他序列化方式。內(nèi)存暫用是java 序列化的1/9,時(shí)間也是差了一個(gè)數(shù)量級(jí),一次操作在1us左右。缺點(diǎn):就是對(duì)象結(jié)構(gòu)體有限制,只適合于內(nèi)部系統(tǒng)使用。
json格式在空間占用還是有一些優(yōu)勢(shì),是java序列化的1/2.6。序列化和反序列化處理時(shí)間上差不多,也就在5us。當(dāng)然這次使用的jackson,如果使用普通的jsonlib可能沒有這樣好的性能,jsonlib估計(jì)跟java序列化差不多。
xml相比于java序列化來說,空間占用上有點(diǎn)優(yōu)勢(shì),但不明顯。處理時(shí)間上比java序列化多了一個(gè)數(shù)量級(jí),在100us左右。
以前一種的java序列化,表現(xiàn)得有些失望
hessian測(cè)試有點(diǎn)意外,具體序列化數(shù)據(jù)上還步入json。性能上也不如jackjson,輸?shù)帽容^徹底。
hessian使用壓縮,雖然在字節(jié)上有20%以上的空間提升,但性能上差了4,5倍,典型的以時(shí)間換空間??偟膩碚f還是google protobuf比較給力
總結(jié)
以后在內(nèi)部系統(tǒng),數(shù)據(jù)cache存儲(chǔ)上可以考慮使用protobuf。跟外部系統(tǒng)交互上可以考慮使用json。
有興趣的同學(xué),可以研究一下google protobuf的marshall的方式:
http://code.google.com/intl/zh/apis/protocolbuffers/docs/encoding.html(###)