hackystat-sensorbase-postgres size

we've been using the hackystat-sensorbase-postgres module for a couple of weeks now. and everything looks pretty good. no failed sensors, no need to restart the sensorbase (knock on wood). everything looks great.

so, i wanted to check out the statistics of the postgres tables, so i ran the analyze and vaccum maintenance executable. and viewed the report. here is what i found.

hackystat-sensorbase-postgres stores about 750,000 sensor data entries in 100 MB.
hackystat-sensorbase-uh stores about 350,000 sensor data entries in 1 GB (according to philip in this email thread, i need to validate that number)

woah! well, haha. i guess that should be expected. the awesome thing is that i think our database will perform well no matter how much data is in our database. and we should see fast query times all day, every day. :)

by the way, i'd imagine that we'd hit 2 million sensor data entries a month. thats just a guess, i'll let you know when we reach a month of up time.

hackystat dev

i've been pretty busy working on hackystat during off time. its pretty fun getting back into hackystat hacking. we've been doing pretty good. actually, its not exactly hackystat hacking. we've been doing a mix of custom and open source hacking. for example, we've been accessing jira directly to get jira issue data. and we've been thinking of creating a mini-feed/stream of consciousness. that doesn't really map too well with hackystat.

anyway, i'm enjoying our latest hackystat development. we have a pretty good plan and we are making progress. one of the things that i enjoy is the collaboration. we do "night time internet". we are all online at night hacking a way. we all know the problems and goals so collaboration over chat is pretty easy. some times we also do lunch time dev, where we spend an hour hacking during "lunch". the other cool thing we do is we have a requirement to do one blog post related to hackystat per week (this is in an internal blog). it keeps us on track. oh, oh, oh.... the other thing i do is keep on my guys to update their jira issues. i don't care if they spent 30 minutes working on something, i want to know about it. this helps keep the project active and helps keep each other aware of whats going on.

we got one more interesting thing in the works... hopefully it will be successful. i'll blog about it later once we get things more developed.

anyway, i've been busy. haha. thats why i haven't posted too much. but, i will start it up again. i promise!

hackystat-sensorbase-postgres ftw!

so, i've been hacking on hackystat the last week or so. for the most part, i've been concentrating on sensorbase, more specifically the hackystat-sensorbase-postgres module. when i first started working on this, we saw numbers like this with shellperf (for a 100 entries):

Postgres trial 1: 78.6 Milliseconds/sensordata instance

Postgres trial 2: 111.25 Milliseconds/sensordata instance

Postgres trial 3: 78.75 Milliseconds/sensordata instance

Postgres trial 4: 62.34 Milliseconds/sensordata instance

after a few days of hacking and an OS change to linux we got numbers like this

Postgres trial 5: 16.36 Milliseconds/sensordata instance

and this number is with 400k sensordata and 1.3 million sensordata_properties entries in our database.

queries
one of the great things about using a database is the ability to run queries. here are a couple that i just wrote:

this one gets all the data sensor data from today.

select * from sensordata, hackyuser 
where hackyuser.email='emailaddress@hackystat.org' 
and sensordata.owner_id = hackyuser.id 
and resource like '%fooProject%'
and tstamp > current_date

this one gets the exact snapshots per day

select date_trunc('day', runtime), tool, max(runtime) from (
select distinct runtime, tool from sensordata, hackyuser 
where hackyuser.email='emailaddress@hackystat.org' 
and sensordata.owner_id = hackyuser.id 
and resource like '%fooProject%'
) runtime_tool
group by date_trunc('day', runtime), tool
order by date_trunc('day', runtime)

(formatted output)
"2008-06-27" "2008-06-27 00:39:33.458" "Checkstyle"
"2008-06-27" "2008-06-27 00:40:02.802" "JavaNCSS"  
"2008-06-27" "2008-06-27 00:39:41.145" "JUnit"   
"2008-06-27" "2008-06-27 00:39:58.63"  "PMD"   
"2008-06-27" "2008-06-27 00:40:09.834" "SCLC"   
"2008-06-29" "2008-06-29 14:15:26.607" "Checkstyle"
"2008-06-29" "2008-06-29 12:27:54.546" "JavaNCSS"  
"2008-06-29" "2008-06-29 12:27:44.89"  "JUnit"   
"2008-06-29" "2008-06-29 12:27:53.062" "PMD"   
"2008-06-29" "2008-06-29 12:27:57.156" "SCLC"   

the totally awesome thing is that even though there is hundreds of thousands of entries we can execute these queries in less than a second. its totally fast.

issues
there are always issues; here are a couple

deletes take forever - postgres isn't really optimized for deletes. so they take much longer than updates or even inserts. i've seen that even deleting 10 records can cause a http timout on another request. so i'm thinking the approach we should take is; disable deletes. deletes aren't really that important and seem like an administrator type function or at the very least asynchronous. anyway, the problem seems to be that delete is used in the test cases. so.. i left it in for now.

count (*) takes forever - postgres has some issues with counting a huge table. so, instead of count (*), i'm using

select relname, n_live_tup, last_analyze 
from pg_stat_user_tables
where relname like '%'

this is really fast, but is an estimate because the stats could be out of date.

thats it for now. things seem to be all good with the sensorbase. at least for now.

hackystat data in json

so, i got to thinking that it would be cool to switch to json for our REST data-interchange format. it will definitely speed up the network transfer. for example, 10 of my xml offline files from my last post about hackystat performance is 1.64 MB. but the same data stored in inline (compact) json is 876 KB. half the size! i'm not sure but i think the parsers for json are really fast too.

here is an example of the xml and json formats (i found a cool little converter at
http://www.thomasfrank.se/xml_to_json.html:


<sensordata>
   <timestamp>2008-06-26T21:11:54.314-10:00</timestamp>
   <runtime>2008-06-26T21:11:54.314-10:00</runtime>
   <tool>Checkstyle</tool>
   <sensordatatype>Code&rdatatype>
   <resource>E:\java\svn\hidden\hidden\hidden\package.html</resource>
   <owner>kagawaa@hahah.hahaha</owner>
   <properties/>
</sensordata>


{
   sensordata:{
     timestamp:'2008-06-26T21:11:54.314-10:00',
     runtime:'2008-06-26T21:11:54.314-10:00',
     tool:'Checkstyle',
     sensordatatype:'CodeIssue',
     resource:'E:\java\svn\hidden\hidden\hidden\package.html',
     owner:'kagawaa@hahah.hahaha',
     properties:{}
   }
}

trust, me if you put json in inline form you'll get a big savings (i guess thats the same for a lot of formats).

one of the cool things about Ruby On Rails is that (and this is just an example) RoR handles all the different data-interchange formats is. it lets the client decide. here is a snippet from a rails controller:

respond_to do |format|
   format.html
   format.xml { render :xml => @sensordata.to_xml }
   format.yaml { render :inline => @sensordata.to_yaml }
   format.js { render :text => @sensordata.to_json }
   format.json { render :json => @sensordata.to_json }
end

anyway, just a thought.

hackystat performance

i started recent discussion on hackystat-dev about some weirdness i was noticing when sending offline data to the hackystat sensorbase.

here is what i said,

I had a lot of offline data

I executed an ant sensor

the sensor started to send the offline data

i decided that i didn't want to wait for the offline data and did a Ctrl+C to kill the ant task.

the client seemed to recover fine.i didn't check the task manager or anything.

but, i had a remote desktop connection to the sensorbase and noticed that the server continued to received data for quite a while. i eventually stopped the server too and restarted it.

the discussion continued i got a somewhat strange response from philip that caught me a little off guard.

> Is it the asynchronous nature of a REST post?
No, it's the nature of TCP and socket-based communication:

<http://www.ncsa.uiuc.edu/~vwelch/net_perf/tcp_windows.html>

Google on "socket buffer size" for more related links.
Again, this is only my _hypothesis_ as to why you were receiving data after you
killed the sending process.

austen and i started to talk about this at work. austen initially agreed with philip. but, i still scratched my head... hm.. doing a search on the "socket buffer size" showed me that

Typical network latency from Sunnyvale to Reston is about 40ms, and Windows XP has a default TCP buffer size of 17,520 bytes. Therefore, Bob's maximum possible throughput is:

17520 Bytes / .04 seconds = .44 MBytes/sec = 3.5 Mbits/second

The default TCP buffer size for Mac OS X is 64K, so with Mac OS X he would have done a bit better, but still nowhere near the 100Mbps that should be possible.

65936 Bytes / .04 seconds = 1.6 MBytes/sec = 13 Mbits/second

from http://www.onlamp.com/pub/a/onlamp/2005/11/17/tcp_tuning.html

so... the delay i was seeing couldn't possibly be from 64K buffers could it? i'm not sure. so i designed a little experiment. here is what i did.

i created a builds worth of offline data by giving my sensorshell.properties file a bogus password. NOTE a builds worth of data is 11.3 MB.

i shutdown the sensorbase (we are using austens postgres version)

deleted all the logs from the sensorbase and my client

brought up a build for a project and executed ant checkstyle (which calls the sensor

i watch the consoles and logs

here is some interesting results.

here is my Checkstyle.log file (edited to save horizontal space):
22:44:57 Hackystat SensorShell Version: 8.1.530
22:44:57 SensorShell started at: Thu Jun 26 22:44:56 HST 2008
22:44:57 SensorProperties
sensorshell.autosend.maxbuffer : 250
sensorshell.autosend.timeinterval : 1.0
sensorshell.logging.level : INFO
sensorshell.multishell.autosend.timeinterval : 0.05
sensorshell.multishell.batchsize : 499
sensorshell.multishell.enabled : false
sensorshell.multishell.maxbuffer : 500
sensorshell.multishell.numshells : 10
sensorshell.offline.cache.enabled : true
sensorshell.offline.recovery.enabled : true
sensorshell.sensorbase.host : http://blah:9876/sensorbase
sensorshell.sensorbase.user : kagawaa@hahaha.hahaha
sensorshell.statechange.interval : 30
sensorshell.timeout : 10
sensorshell.timeout.ping : 2
sensorshell.properties file location: C:\..\sensorshell.properties
22:44:57 Type 'help' for a list of commands.
22:44:59 Host: http://naraku:9876/sensorbase/ is available.
22:44:59 User akagawa@referentia.com is authorized to login at this host.
22:44:59 Maximum Java heap size (bytes): 66650112
22:44:59 AutoSend time interval set to 60 seconds
22:45:00 Pinged http://blah:9876/sensorbase/ in 188 ms. Result is: true
22:45:00 Checking for offline data to recover.
22:45:00 Invoking offline recovery on 48 files.
22:45:59 Timer-based invocation of send().
22:46:59 Timer-based invocation of send().
22:47:59 Timer-based invocation of send().
22:48:59 Timer-based invocation of send().
22:49:55 #> quit
22:49:55 #> send
22:49:55 Pinged http://blah:9876/sensorbase/ in 0 ms. Result is: false
22:49:55 Server not available. Storing commands offline.
22:49:55 Stored 4 sensor data instances in:
C:\...\sensorshell\offline\2008.06.26.22.49.55.426.xml
22:49:55 Quitting SensorShell started at: Thu Jun 26 22:44:56 HST 2008
22:49:55 Total sensor data instances sent: 0


at the end of the sensor execution it says it can't send 4 sensor data instances! what happened to the server!? hm.. what is going on. for some reason that indicates to me that sending the offline data made the sensorbase unavailable. so, i looked in the Checkstyle-offline-recovery.log file

22:45:00 Maximum Java heap size (bytes): 66650112
22:45:00 AutoSend disabled.
22:45:00 Invoking offline recovery on 48 files.
22:45:00 Recovering offline data from: 2008.06.26.21.12.01.081.xml
22:45:00 Found 251 instances.
22:45:00 Invoking send(); buffer size > 250
22:45:00 #> send
22:45:01 Pinged http://blah:9876/sensorbase/ in 204 ms. Result is: true
22:45:01 Attempting to send 251 sensor data instances.
Available memory (bytes): 63071744
22:45:12 Error sending data: org.hackystat....SensorBaseClientException:
1001: Unable to complete the HTTP call due to a communication error
with the remote server. Read timed out
22:45:12 org.hackystat.sensorbase.client.SensorBaseClientException: 1001:
Unable to complete the HTTP call due to a communication error with the
remote server. Read timed out at org.hackystat.sensorbase.client.
SensorBaseClient.putSensorDataBatch(SensorBaseClient.java:827)
at org.hackystat.sensorshell.command.SensorDataCommand.
send(SensorDataCommand.java:82)
22:45:12 Exception during send(): org.hackystat.sensorshell.
SensorShellException: Could not send data: error in SensorBaseClient
22:45:12 About to send data
22:45:12 Successfully sent: 0 instances.
22:45:12 Did not send all instances.
C:\...\sensorshell\offline\2008.06.26.21.12.01.081.xml not deleted.


in the Checkstyle-offline-recovery.log file, i can't find one successful send. but, i know i sent data over. i have no idea but maybe the "Timer-based invocation of send()." was able to send data, but there is no log of that timber-base send. anyway, so there was a problem with the server. but, the logs on the server show nothing. i just get a whole bunch of these

22:55 Put: 2008-06-26T21:11:56 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:55 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:55 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:56 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:55 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:56 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:55 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:56 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:55 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:56 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:55 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:56 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:55 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:56 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:55 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:56 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:55 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:56 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:55 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:56 k@gm.com Checkstyle CodeIssue
22:55 Put: 2008-06-26T21:11:55 k@gm.com Checkstyle CodeIssue


so i have no idea why the server wasn't responding. in the end off all of that i only see 1,742 entries in the database (there might be a problem with the sensordatatype for coverage). so that's 10 minutes of work for a very small amount of data entries. i would guess there were tens of thousands of possible entries in the offline data.

here is another thing. note that in my Checkstyle.log it says that my sensor process ended at

last entry in from Checkstyle.log
06/26 22:49:55 Total sensor data instances sent: 0

last entry on sensorbase
06/26 22:55 Put: 2008-06-26T21:11:55 k@gm.com Checkstyle CodeIssue


thats 6 minutes after the sensor ended. so, getting back to those 64K buffer caches... that seems a little strange to me. if thats really what is happening there must be a lot of caches.

anyway, there are two things happening.

1) i can send data to the hackystat sensorbase server way way faster than hackystat can consume
2) because of that backlog on the server its not letting me send more data

that worries me. after all 11.3 MB of data is really tiny; after all its just one full build of our system with sensors turned on. here are some questions:

1) how fast can hackystat consume data?
2) what is happening such that data is being processed on the server 6 minutes after the client is finished sending?
3) whats the current bottleneck?
4) how many users can send data (and how much data) simultaneously without killing the server?

hm.. there are a lot of moving parts to this so this is just the start of looking into this. but, all i know is that 11.3 MBs is really small. in my other project we move that amount of data in an handful of seconds not even close to minutes.

ps. i had to delete my offline data. :(

to estimate features you have to measure features

i just read an article from 6th sense analytics. i've been talking a little bit about putting metrics at the task level in another email thread. i thought i share this posting with the entire group (so you can also start reading 6th sense's blogs). anyway, here is what the article said:

We all know that developers can't estimate time, don't we? But why not? Developers are very smart. We deal with abstract concepts every day. You'd think we'd be able to learn this skill over time. But somehow we never get better at work estimation. Some people attribute it to a developer's innate optimism, and that's part of it. But there's a more important factor here.

There's an old saying... "Those who ignore history are doomed to repeat it". And as Einstein put it "Insanity is doing the same thing over and over and expecting a different result".

Here are you steps to improving your estimation attempts.
1) Estimate smaller (more granular) features
2) Always look back at estimates vs. actuals
3) Get your feedback as fast as possible

and i would agree. the article is focusing on time estimates, but i think one could utilize this technique for other metrics. one of the useful skills that i have been learning as i gain more experience is being able to think about and analyze the impacts of a new task, feature, improvement, or bug. i am learning to be able to evaluate the situation first before just diving into code. i do this by learning the necessary APIs, grounding the problem, and tackling small pieces at time. one thing that i feel is lacking in my quest to improve in this area is software metrics. i can use historical (previous task) data to help guide my planning; even if the planning is a few minutes.

to be honest, i'm not sure it would be super useful. but it sure sounds useful. it seems that it just increases the fidelity that you have to make decisions and learn. i really think that putting metrics next to tasking will be useful to my development process. tasking is very important in the commercial world; it drives the development process. we don't have the luxury of saying, "oh, coverage is low lets do some testing". testing needs to be in-process during our task work.

there are all sorts of things that we can do with tasking. for example, "hm.... i wonder if someone in the company (on another project) has seen this sort of bug before." anyway, thats a different issue; and something we haven't talked about in a while.

anyway, we have to start measuring features before we can estimate features.

UPDATE: this posting is what i kinda meant in driving your project with hackystat.

driving a project with hackystat

in CSDL (the collaborative software development laboratory) we explored a permanent display of metrics in our telemetry wall. basically it was a control center for trends on a 9 monitor display. we played around with that for a little while but lost interest in it - it sat there for months without using it. i came up the idea of the telemetry wall 2.0, which made the telemetry wall a little more collaborative; not just metrics.

the UH ICS software engineering students are working on this idea in the informative workspace project. they are making good progress.



but, i had a realization the other day about what else i want to have on the informative workspace. i don't want the informative workspace to be just a useful tool in passing. i want it to be able to also drive the project. i want it to be the place where we go to give our scrum meetings. i don't think we are quite there yet. there are a lot of things to figure out. but, basically, i think the telemetry wall (or informational workspace as it is called now) can help focus on day to day management of the project. help project managers and the team manage their project by providing a more robust and higher fidelity set of information. i don't think we focused a lot on this in the past; telemetry and even trajectory focuses at a different set of problems.

you can't even ask them to push a button

fogcreek put out a blog advertising their integration with timepost:

Timepost is a desktop time tracking solution that integrates with web-based project collaboration software. Timepost offers reliable offline time tracking for Basecamp, Cashboard, FogBugz, FreshBooks, Harvest, and Tick.

wow.. so that sounds like LEAP's time recorder to me. well, i'm not surprised by this little recorder thing. fogbugs has pushed their evidence-based scheduling (EBS) for a while now. within EBS is a timesheet function:

To get good results from EBS, always let FogBugz know what you’re working on, so it can fill out timesheets for you. All you have to do is click on the Working On menu when you start on a different task, and FogBugz takes care of the rest. It even stops the clock automatically at the end of the workday, and restarts it the next morning. You can preprogram your lunch breaks and vacations, too. That way you only need to tell FogBugz when you switch from one case to another.

haha. preprogram lunch breaks. omg. thats pretty funny. i wonder if the guys at fogcreek seen this paper:

You can't even ask them to push a button: Toward ubiquitous, developer-centric, empirical software engineering, Philip M. Johnson, The NSF Workshop for New Visions for Software Design and Productivity: Research and Applications, Nashville, TN, December, 2001.

Abstract: Collection and analysis of empirical software project data is central to modern techniques for improving software quality, programmer productivity, and the economics of software project development. Unfortunately, barriers surrounding the cost, quality, and utility of empirical project data hamper effective collection and application in many software development organizations.

This paper describes Hackystat, an approach to enabling ubiquitous collection and analysis of empirical software project data. The approach rests on three design criteria: data collection and analysis must be developer-centric rather than management-centric; it must be in-process rather than between-process, and it must be non-disruptive---it must not require developers to interrupt their activities to collect and/or analyze data. Hackystat is being implemented via an open source, sensor and web service based architecture. After a developer instruments their commercial development environment tools (such as their compiler, editor, version control system, and so forth) with Hackystat sensors, data is silently and unobtrusively collected and sent to a centralized web service. The web service runs analysis mechanisms over the data and sends email notifications back to a developer when ``interesting'' changes in their process or product occur.

Our research so far has yielded an initial operational release in daily use with a small set of sensors and analysis mechanisms, and a research agenda for expansion in the tools, the sensor data types, and the analyses. Our research has also identified several critical technical and social barriers, including: the fidelity of the sensors; the coverage of the sensors; the APIs exposed by commercial tools for instrumentation; and the security and privacy considerations required to avoid adoption problems due to the spectre of ``Big Brother''.

haha. i'm poking fun at fogcreek, but today i realized that it would actually be great to know what i worked on and how long i worked on it. i just don't think that a timer and/or a timesheet is the way to do it. the context switching would be too much of a hassle. so i guess the bottom line is that i think it would be cool to read a log of my tasks. but, i'm not willing to click on a timer.

hackystat takes a different approach than fogbugs. similarly 6th sense analytics wrote about status reports in their blog. they write:

I like having status reports, but I hate writing them. So I generate them. One of the original aspects of the vision for 6th Sense was that we could generate status reports.

first of all, the automatically generated status reports are almost as silly as a pushing a button and writing everything down. fogbugs gives you a lot of good context. and hackystat gives you a lot of good objective data. but, i think they are even better when it is joined together.


context + objective info + automation + reporting = awareness, collaboration, and improvement.

google's summer of code

Here is an email from Philip

Greetings,

I am happy to inform you that Google has selected Project Hackystat as a participant in this year's Summer of Code

In brief, this means that students from around the world can apply to work on a Hackystat-related project this summer and Google will pay them US$4500 for their efforts. This is an excellent opportunity and I hope many students will take advantage of it. The timeline is:

* March 31: Student application deadline
* April 14: Accepted student proposals announced
* May 26: Students begin working; Google issues initial payments.
* Aug 18: "Pencils down" date. Projects should be submitted.

Please do not hesitate to contact me if you (or students you supervise) would be interested in participating and would like further details. Please also forward this email on to any students you know who might be interested. Note that as long as the student is enrolled in a college or university program as of April 14, 2008, they are eligible to participate.

We have published an "Ideas" page with potential summer projects

However, if you have another interesting idea involving the Hackystat framework, don't hesitate to discuss it with me. We are very open to new ideas.

Cheers, Philip

this is great news! congratulations philip and all hackystat hackers out there. i really can't wait to see how this will all work out. it should be a lot of fun. i'm really looking forward to working with the students. even though i don't hack as much as should be on hackystat i hopefully can help with domain expertise. well, i guess this is a lot of motivation to get back to hacking. but, it is really all about the students. it is an excellent opportunity for students. so all you students out there take this opportunity seriously. do that and we'll have a lot of fun!

i'm ready to help. austen is too.

google's summer of code is making students awesome!

hackystat problem v2.0

my last hackystat problem blog was solved pretty swiftly see the mailing list; SensorBase REST API now includes "snapshot", performance improved and DPD zero issue bug .

Tonight I tried to use the Telemetry Viewer to check some data points out. I used the Telemetry Viewer deployed on the public server (v.8.1.214) and i have Firefox 2.0. Check out the video for a description of the bug:




the problem seems to be the following:
1) all the charts for the project Hackystat doesn't seem to do anything in the viewer. i click on "Chart" and nothing happens. (in the video you see the app flicker, but nothing really happens.)
2) i can't seem to see my data in the Default project. A script error appears. and i get an empty blue chart.

what's unsure:
1) i'm not sure if the public version of the Telemetry Viewer is the latest and greatest version.
2) does it supposed to work with the version of Firefox i have? or is anything else on the client side preventing the right functionality?
3) has no one else seen these problems?

UPDATE
it works locally at our work place.. not sure whats wrong with the public server.

hackystat problems

i was inspired to work on hackystat this weekend. being that i've been out of the game for a while, so i decided to check out the user interfaces first. unfortunately, things didn't work out so well. here are some highlights of my journey into trouble.

i started out by executing a sensor:

C:\java\svn-hackv8\hackystat-sensor-ant>ant -f checkstyle.build.xml 
checkstyle.sensor

Buildfile: checkstyle.build.xml

checkstyle.sensor:
[hacky-checkstyle] 35 Checkstyle issues sent to 
http://dasha.ics.hawaii.edu:9876/sensorbase/ (1 secs.)

BUILD SUCCESSFUL
Total time: 5 seconds
C:\java\svn-hackv8\hackystat-sensor-ant>

then i went to view it in the sensor data viewer. but, oops where did that data go? i remember something about it not being historical and it just shows the new data. okay, that is strange requirement, but maybe it isn't "broken". but then again, thats a pretty strange requirement. who is going to go to the sensor data viewer before sending data so you can watch new data coming in. anyway.... moving on....


i fired up sensor data browser and found the data. it looks pretty good there.


okay, so what now? i have data, so what can i do with it? i fired up the telmetry viewer and found that none of the charts worked. well, i guess i'm not sure if the telemetry viewer was busted, or if it was the DPD service, or if it was the sensor base client, or if i didn't check the sensor data browser first i might have thought i didn't send the data. i had to double check at this point; i checked my sensor log file, sensor data browser, and then tried to fire up the DPD rest api...


following the REST api instructions for the DPD service, in constructed a URL that i thought would show me some data. i executed the url and found an empty set of data. hm....


i kind of don't understand what is wrong at this point. it would be great if i knew at the very least if my url was right. i tried typing in some crazy date and got a blank page. i guess that validates that i got the previous URL right. i think...


i'm not sure what is wrong at this point. to summarize here is what happened.
1) sent some checkstyle data (validated it in my sensor logs)
2) saw the data in sensor data browser
3) didn't see the data in the sensor data viewer, because it only shows new data
4) none of the telemetry charts worked
5) i couldn't figure out if my URL for the REST DPD service was right or not and i couldn't see the DPD data as i hoped
6) my last chance at doing something was ended when i couldn't create a new project in the project viewer.


needs more transparency
so, i guess i wasn't too successful in working on hackystat tonight. i'm even more confused when i started out. i think the problem is the transparency into what is going on. the distributed nature of the applications hides the "knowledge" of what is actually going on. an outside user would have no clue that the telemetry viewer uses the DPD service, which uses the sensorbase (actually i'm not even sure thats correct). the transparency in data flow is set to pitch black.

hm....... i guess it's just me.

atlassian has a stream of consciousness, hackystat should have on too

atlassian just put out an interesting blog; JIRA Studio: Stream of Development Consciousness.

basically, they want to expose a developers activity across different functions in one single place. its an outboard brain type of thing. see help me help you and hackystat and my outboard brain.

atlassian's attempt to do this is really cool because they are actually providing the functionality in an integrated fashion. the only problem with their approach is that it is limited to atlassian products. duh... hackystat can expose so many more hooks and so many more different types of activities.

we will get there one day. i just wanted to point out that where i once thought we were so a head of the game, in terms of metrics and reporting, people like atlassian is a head of the game in other areas.

wow.. i just had a cool idea. what if hackystat used open social to expose certain metrics. thats way cool.

hackystat ui idea

i know it is so lame, but i'll admit it. i haven't really looked at javafx. but, after looking at it really quickly, it seems to me that javafx could be a potential implementation tool for a hackystat user interface. it seems similar to adobe air, but better.

here is an interesting demo. note josh's comment:

One thing I've noticed when building my demos is my development style. It's totally different than when writing Java programs. When I code Java, I think first about the objects and interfaces, create empty .java files, and then start writing the implementation. When I write JavaFX script using the preview mode in NetBeans my style is completely different. I immediately start throwing up shapes and UI components, constantly moving code around and renaming things. Once I have something solid only then do I split it into separate classes. It's definitely a less rigorous but more fluid development experience. And, dare I say it, more fun!

javafx seems like a cool little thing. i'm not sure how useful it really is. it maybe worth checking out for a quick and dirty hackystat ui. maybe its something we can use for a client side telemetry wall viewer.

anyway... just an idea.

JavaNCSS Hackystat Sensor

I just started to look at how to write a Hackystat sensor for the JavaNCSS results. A couple of interesting thoughts and issues came up.

is Resource enough?
what is a Resource in Hackystat? well, according to the Rest API Specification, a Resource is a "A URI indicating the resource (file, website, etc.) on which this sensor data was collected." Hmm.. Okay that seems fair enough. But, the problem I'm faced with is that JavaNCSS produces package, object, and method level metrics. Thus, it would be impossible to capture and utilize all of JavaNCSS's metrics.

for example, here is some sample data:
<package>
<name>org.hackystat.sensor.ant.checkstyle</name>
<classes>2</classes>
<functions>7</functions>
<ncss>129</ncss>
<javadocs>6</javadocs>
<javadoc_lines>32</javadoc_lines>
<single_comment_lines>19</single_comment_lines>
<multi_comment_lines>17</multi_comment_lines>
</package>

<object>
<name>org.hackystat.sensor.ant.checkstyle.CheckstyleSensor</name>
<ncss>67</ncss>
<functions>4</functions>
<classes>0</classes>
<javadocs>4</javadocs>
</object>

<function>
<name>org.hackystat.sensor.ant.checkstyle.CheckstyleSensor.CheckstyleSensor()</name>
<ncss>2</ncss>
<ccn>1</ccn>
<javadocs>1</javadocs>
</function>

so you see that package has some data that object does not (single_comment_lines and multi_comment_lines). and function has ccn (Cyclomatic Complexity Number). To satisfy Hackystat's requirement I'd just have to use the object metrics, which means missing out on the single_comment_lines, multi_comment_lines, and ccn. That seems bad to me.

Mapping inner classes to files
Currently, when the tool results do not provide the file path (ie provides a class name org.hackystat.sensors.ant.checkstyle.CheckstyleSensor), we do a matching to match the class name with a file. I'm not sure how that works exactly, but I do wonder if it works correctly for inner classes. Basically, would org.hackystat.sensor.ant.checkstyle.CheckstyleSensor.Temp (where Temp is an inner Class) map correctly to c:\java\svn\hackystat-ant-sensors\src\org\hackystat\sensor\ant\checkstyle\CheckstyleSensor.java?
I'm not sure it would.

JavaNCSS bug
Investigating that inner class issue, I found that JavaNCSS does not capture inner class metrics correctly. it seems that JavaNCSS doesn't report separate metrics for inner classes. i created two temporary inner classes within the CheckstyleSensor to test out whether JavaNCSS would handle it correctly. unfortunately, it seems the only metrics i get out of JavaNCSS about those inner classes are rolled up in existing elements, meaning i don't get a separate object element for the inner classes. instead, i get this:

<object>
<name>org.hackystat.sensor.ant.checkstyle.CheckstyleSensor</name>
<ncss>75</ncss>
<functions>4</functions>
<classes>2</classes>
<javadocs>4</javadocs>
</object>

the classes=2 is supposed to represent the two inner classes that i added. i suppose, one could argue that the inner classes belong to CheckstyleSensor and that should be okay to contain the metrics within the CheckstyleSensor object. But, wouldn't it be better if we could distinguish the metrics between CheckstyleSensor and its inner classes. anyway... i didn't particularly like that about JavaNCSS.


Anyway, a Hackystat JavaNCSS should be relatively easy to implement. i probably could have finished it faster than writing this blog. but, i wanted to hash out some issues that i was thinking about. in fact, i think i should bring up the Resources and mapping to file questions to the hackystat-dev mailing list.

hackystat and my outboard brain

i just had an interesting idea that branches off of what wired has been talking about a lot; an outboard brain. see Welcome to a Science Journalist's Outboard Brain and help me help you.

basically, what i want is a hackystat and outboard brain combined. this is interesting because, i can learn a lot from people's outboard brain. i can learn a lot more from "watching" them work.

imagine for example, that you knew everything that philip thought was cool and you knew what kind of things philip was working on and the processes he used to work on them. if you had that then, you'd be pretty aware of philip's intentions, his direction, and you'd learn a whole lot from that.

this is a cool concept. but, i haven't had enough time to think deeper about this.

more clover work

i've made some progress on getting clover working and the clover hackystat sensor. i feel compelled to talk a little more about clover. here are a few of my other clover posts; implementing the clover sensor and metrics clouds.

so, the clover tool looks pretty cool. however, i'm not sure how much better it is than something like emma. one thing is certain, clover has a lot nice reports. here is an example:



that is pretty nice. i suppose that is the benefits of a commercial tool. however, i found bad sides of a commercial tool. for example, the licensing costs is ridiculous:



for some reason the forums were pretty empty. i guess it could be because of the recent buy out of Cenqua. i want a bigger community behind it.



or maybe its the cost of the tool itself. anyway, i'm not sure why no one is posting. all i know is that i would be crazy to want to buy this tool at full price.

i do think clover is a good tool and i would use it.

implementing the clover sensor

i just finished an initial implementation of the hackystat ant-based clover sensor, which process the xml clover report to send to hackystat. as usual creating an ant sensor is pretty easy assuming that the report that the tool creates is in xml and makes sense.

here are some issues that i came across.

different attribute names
after writing the sensor, i noticed that the attributes names are different than emma's. for example, clover generated Coverage data has statement_Covered and statement_Uncovered. however, emma generated Coverage data has line_Covered and line_Uncovered. i addition, because there are no required attributes (other than timestamps, type, etc) the Coverage DPD needs to be "oblivious" to the different granularities. this all brings up issues with DPD implementations, will the DPDs contain the last snapshot per tool? if a project uses multiple coverage tools, you might have a random sampling of last batches in the DPDs coming from different tools. that would make it really hard to use the DPDs over time. it seems that the generality of everything that is going on is going to make it harder for the analysis writer or even user to know what is going on, because they would have to know about the details. i thought the whole idea of the abstraction hierarchy was to make "abstractions". let me just throw this out as well, what if we wanted to combine emma's coverage values and clovers coverage values (they are a different set). there is no way to do that.

issues with the clover ant task
it actually took longer to configure the clover tool with the ant tasks than it took to write the sensor. in fact, i wasn't able to configure the ant clover tasks the way i wanted. so, technically i'm not done. for some reason i wasn't able to clean before doing the clover-setup task. i don't get way that is. but, i won't bother with it for now.

our test cases are junk
i need to work on those test cases. basically, i think following this will help.

clover can send filemetric data
clover's data looks like this:

<metrics classes="1" methods="18" coveredmethods="7" 
  conditionals="36" coveredconditionals="11" 
  statements="141" coveredstatements="77" 
  elements="195" coveredelements="95" 
  ncloc="270" loc="456" />

to me that looks like coverage data and filemetric data. so... would it be wrong to send both?

chat about smart snapshots

the hackystat gang just finished up a long thread about large datasets in DPDs. i fought long and hard in that thread to no avail. but, i understand the competing idea and don't necessarily think its bad. both solutions have its advantages and disadvantages.

through chatting i kinda uncovered something else that i kinda don't like. a disadvantage in my mind - but not necessarily wrong. here it goes:

aaron: where is the smart snapshot?
aaron: the smart thing is going in reverse order?
austen: right. since snapshots are the latest data set
aaron: so you get thirty minutes chunks.
austen: ya
aaron: but what if the timestamps just so happen to span across the thirty minutes?
austen: then u would be done
austen: im not sure wut u are getting at
aaron: hm...
aaron: timestamp 1:59
aaron: timestamp 2:00
aaron: timestamp 2:01
aaron: all part of the same runtimestamp of 2:04
aaron: if you get timestamps from 2:00 - 2:30
aaron: then you are missing part of the data.
aaron: you have no idea whether the data actually spans more than 30 minutes.
aaron: the batch could be hours and hours long.
aaron: but the runtimestamp could be at 2:04
aaron: you'll never know when to stop.
aaron: so, i'm saying that algorithm is an approximation.
aaron: unless there was a way to say get all data with runtimestamp = 2:04
austen: wouldn't u just loop backwards till u find the end of the snapshot?
austen: within the specified time span
aaron: when do you know its done?
aaron: what is all the data?
aaron: theoretically you'll never know if you are done.
aaron: in practice i guess you could say if i don't get any data in the next bucket i look at then i guess i can call it done (aprox).
aaron: still don't get it?
austen: u know its done when u are getting data whose run time is different.
aaron: not.. necessarily.
aaron: because batched data can be mixed right?
aaron: you could be sending data simultaneously
aaron: from the same user
aaron: intermixing data with different runtimestamps but very similar timestamps
austen: if its from the same tool invocation , the sensor will send runtimes that will be the same
austen: so no
aaron: what...
aaron: what if i had two windows open. run full-build at the same time.
aaron: the windows one is a better example.
austen: are u saying that it is broken cause the data will not be ordered?
aaron: the data will be ordered but based on timestamp
aaron: not runtimestamp
austen: yah. ok I see
aaron: basically we are doing approximation.
aaron: we are hoping that the 30 minute chunk has all the data.
aaron: and that it didn't actually span over 2 hours
aaron: becasue there is no way to actually know if it spanned over 2 hours
aaron: which is fine... i suppose.
aaron: that is what happens when you have smart on one side and not flexible on the otherside.
aaron: we need smart and flexible to be optimized and exact
aaron: right now we are sort of optimized and approximated. which is fine.. but people need to understand that is the case.
aaron: and it will be harder to debug if the approximations aren't that great.

to sum up, the current "smart" snapshot idea will definitely speed things up a lot. however, i think there are scary disadvantages. it is an approximation - theoretically you have no idea how big snapshots are. so, when can you actually stop? in this method the be exact you would have to search the entire day to be sure you have every single data point. but, we probably won't do that. in fact, i think people will implement it differently. so there will be different approximations. that might be bad. approximations might make it harder to debug. but on the other hand, it isn't wrong to do approximations.

anyway... that is all i have to say about that. i'm hoping that the performance issues will be solved pretty quickly with the methodologies that we are adopting.

metrics and clouds and other cool ideas

so, i've talked about atlassian's clover metrics before:
two roads and in metrics, metrics, metrics

so, you would think i should move on to another subject. but i won't. so too bad. haha. anyway, i found this on the clover website:



here is what they say about that graphic:

The Project Risks Cloud highlights those classes that are the most complex, yet are the least covered by your tests. The larger and redder the class, the greater the risk that class poses for your project.
...
This Cloud highlights the "low hanging coverage fruit" of your project. You will achieve the greatest increase in overall project coverage by covering the largest, reddest classes first.

wow.. okay that is a cool idea; metrics clouds. but, again i think they (atlassian) is trapped by limitations of their framework and infrastructure. just doing coverage to identify low hanging fruit is probably really really basic, not to mention potentially meaningless and might actually do more harm than good. its like doing Priority Ranked Inspection with just one metric. nevertheless, the cloud idea is a good one. and yet again, hackystat could do much better.

why i work on hackystat

hackystat is an awesome project. and i work on hackystat for three reasons. but, the priorities of those reasons is probably not what you would expect. here they are:

i want to help lead my peers - part of what i do and what i really love doing is helping the "younger" or newer hackystat developers understand what hackystat is all about. actually, it is the soft skill mentoring that is really motivating. i have the chance to help other hackystat developers grown their critical thinking about software engineering research. thats fun and thats what it is all about.

i learn a lot from my peers - typically everyone that works on hackystat are better than me in programming. yes, i even think that the "younger" developers are better at programming than i am. i learn a lot from them. i also learn a lot on how to be a leader, by following leaders like philip. i also love to hear, "aaron, that is stupid", every time i hear that makes me better.

(the last reason) i like the actual application of hackystat. i like that hackystat is successful and can be even more successful. i like the domain that we are in, because i like software engineering research. but, i put this last because the other two are much much more important to me. i'd probably be as involved if hackystat was a chat client.

so, you see that it is the people and the environment that makes the biggest difference for me. if you are working in a crazy cool project with dumb people, in a dumb environment then i bet you are hating it. i would. people make all the difference. moral of the story: people is what makes software engineering awesome. but people can also make software engineering a pain in the ass.