INTERVAL-ALPHA

interval-alpha.perl.pl is an  Alpha calculation program. Right now it
is limited to calculating alpha with Krippendorff's interval distance
metric, but additional metrics should be easy to add. Two nice
features of this program are: 

1. Reliability is calculated not only for all annotators together, but
   for all subsets of annotators. Subsets are ordered by reliability,
   which makes it easy to visually identify outlier coders. 

2. Confidence intervals are calculated, not only for all annotators
   together but also for all subsets. The results on two real data
   sets demonstrate -- that the variability in reliability due to
   substituting coders is greater than the variability due to
   substituting items.  

An obvious issue is that the number of subsets of annotators grows
exponentially with the number of annotators. This is more of a
conceptual problem than a computational one: already with 5 or 6
annotators, the number of all subsets (26 and 57, respectively)
becomes too big to be comprehensible, while running 10000 samples per
group on a 500 item dataset takes less than an hour. If you have more
than 6 annotators, it's probably better to compare just pairs of
annotators, or groups of size n-1 and n-2. This can be accomplished by
an easy modification to the program.