Friday, June 27, 2008

hackystat data in json

so, i got to thinking that it would be cool to switch to json for our REST data-interchange format. it will definitely speed up the network transfer. for example, 10 of my xml offline files from my last post about hackystat performance is 1.64 MB. but the same data stored in inline (compact) json is 876 KB. half the size! i'm not sure but i think the parsers for json are really fast too.

here is an example of the xml and json formats (i found a cool little converter at
http://www.thomasfrank.se/xml_to_json.html:


<sensordata>
   <timestamp>2008-06-26T21:11:54.314-10:00</timestamp>
   <runtime>2008-06-26T21:11:54.314-10:00</runtime>
   <tool>Checkstyle</tool>
   <sensordatatype>Code&rdatatype>
   <resource>E:\java\svn\hidden\hidden\hidden\package.html</resource>
   <owner>kagawaa@hahah.hahaha</owner>
   <properties/>
</sensordata>


{
   sensordata:{
     timestamp:'2008-06-26T21:11:54.314-10:00',
     runtime:'2008-06-26T21:11:54.314-10:00',
     tool:'Checkstyle',
     sensordatatype:'CodeIssue',
     resource:'E:\java\svn\hidden\hidden\hidden\package.html',
     owner:'kagawaa@hahah.hahaha',
     properties:{}
   }
}

trust, me if you put json in inline form you'll get a big savings (i guess thats the same for a lot of formats).

one of the cool things about Ruby On Rails is that (and this is just an example) RoR handles all the different data-interchange formats is. it lets the client decide. here is a snippet from a rails controller:

respond_to do |format|
   format.html
   format.xml { render :xml => @sensordata.to_xml }
   format.yaml { render :inline => @sensordata.to_yaml }
   format.js { render :text => @sensordata.to_json }
   format.json { render :json => @sensordata.to_json }
end

anyway, just a thought.

2 comments:

Philip Johnson said...

Interesting idea, but if we're going to support a second format besides XML, I'd be interested in looking at protocol buffers, which are binary on the wire:

http://code.google.com/apis/protocolbuffers/

But I agree with your point that data transmission throughput might be improved in certain cases with a more compact representation.

p.s. It would be helpful to post ideas like this to hackystat-dev (perhaps as a link to your blog entry). I just stumbled across this posting accidentally and I don't know if any other hackystat developers have even seen it.

lalee said...

Yes, with JSON you do reap a bit of savings in Packet Size. Though, the problem with JSON (and REST) is that you really need control over both ends of the communication channel. With XML (and SOAP), you are able to get some schema information within the data packet, or from parsing a WSDL.

I have
blogged about this JSON issue some time ago when I was experimenting with Web 2.0 Toolkits.

As for bandwidth savings.. it's all text anyway (XML or JSON), so as long as those MIME types are configured in web server (and the client can handle it), the transmission is gzipped over the wire.