Log in Help
Print
Home 〉 sale 〉 rnti-09 〉 cross-validation 〉 stars 〉 compare_test_train.txt
 
***ALG1***
annotation sets:
*Output*
*Key*

For the document: review-11_517.xml_00155
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (11.0, 0.0, 7.0, 7.0);  (precision, recall, F1)= (0.6111111, 0.6111111, 0.6111111);  Lenient: (0.6111111, 0.6111111, 0.6111111)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (11.0, 0.0, 7.0, 7.0);  (precision, recall, F1)= (0.6111111, 0.6111111, 0.6111111);  Lenient: (0.6111111, 0.6111111, 0.6111111)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 0.0, 3.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 3_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 2.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 4.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (11.0, 0.0, 1.0, 4.0);  (precision, recall, F1)= (0.9166667, 0.73333335, 0.8148148);  Lenient: (0.9166667, 0.73333335, 0.8148148)


For the document: review-10_218.xml_00156
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)


For the document: review-10_90.xml_00157
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (3.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (3.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

For each pair of annotators, and for each label:
(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (3.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)


For the document: review-11_135.xml_00158
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 1.0, 1.0);  (precision, recall, F1)= (0.5, 0.5, 0.5);  Lenient: (0.5, 0.5, 0.5)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 1.0, 1.0);  (precision, recall, F1)= (0.5, 0.5, 0.5);  Lenient: (0.5, 0.5, 0.5)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 0.0, 1.0);  (precision, recall, F1)= (1.0, 0.5, 0.6666667);  Lenient: (1.0, 0.5, 0.6666667)


For the document: review-11_269.xml_00159
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators, and for each label:

For the document: review-10_233.xml_0015A
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (17.0, 0.0, 4.0, 4.0);  (precision, recall, F1)= (0.8095238, 0.8095238, 0.8095238);  Lenient: (0.8095238, 0.8095238, 0.8095238)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (17.0, 0.0, 4.0, 4.0);  (precision, recall, F1)= (0.8095238, 0.8095238, 0.8095238);  Lenient: (0.8095238, 0.8095238, 0.8095238)

For each pair of annotators, and for each label:
(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 4.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (17.0, 0.0, 0.0, 4.0);  (precision, recall, F1)= (1.0, 0.8095238, 0.8947368);  Lenient: (1.0, 0.8095238, 0.8947368)


For the document: review-10_172.xml_0015B
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 1.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 1.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 0.0, 1.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)


For the document: review-10_484.xml_0015C
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (10.0, 0.0, 5.0, 5.0);  (precision, recall, F1)= (0.6666667, 0.6666667, 0.6666667);  Lenient: (0.6666667, 0.6666667, 0.6666667)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (10.0, 0.0, 5.0, 5.0);  (precision, recall, F1)= (0.6666667, 0.6666667, 0.6666667);  Lenient: (0.6666667, 0.6666667, 0.6666667)

For each pair of annotators, and for each label:
(0,1), label= 2_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 3_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 3.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (10.0, 0.0, 0.0, 5.0);  (precision, recall, F1)= (1.0, 0.6666667, 0.8);  Lenient: (1.0, 0.6666667, 0.8)


For the document: review-10_556.xml_0015D
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (15.0, 0.0, 2.0, 2.0);  (precision, recall, F1)= (0.88235295, 0.88235295, 0.88235295);  Lenient: (0.88235295, 0.88235295, 0.88235295)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (15.0, 0.0, 2.0, 2.0);  (precision, recall, F1)= (0.88235295, 0.88235295, 0.88235295);  Lenient: (0.88235295, 0.88235295, 0.88235295)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.5, 1.0, 0.6666667);  Lenient: (0.5, 1.0, 0.6666667)

(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (14.0, 0.0, 0.0, 2.0);  (precision, recall, F1)= (1.0, 0.875, 0.93333334);  Lenient: (1.0, 0.875, 0.93333334)


For the document: review-10_54.xml_0015E
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (7.0, 0.0, 4.0, 3.0);  (precision, recall, F1)= (0.6363636, 0.7, 0.6666667);  Lenient: (0.6363636, 0.7, 0.6666667)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (7.0, 0.0, 4.0, 3.0);  (precision, recall, F1)= (0.6363636, 0.7, 0.6666667);  Lenient: (0.6363636, 0.7, 0.6666667)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 3.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (6.0, 0.0, 1.0, 3.0);  (precision, recall, F1)= (0.85714287, 0.6666667, 0.75);  Lenient: (0.85714287, 0.6666667, 0.75)


For the document: review-10_294.xml_0015F
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (15.0, 0.0, 3.0, 3.0);  (precision, recall, F1)= (0.8333333, 0.8333333, 0.8333333);  Lenient: (0.8333333, 0.8333333, 0.8333333)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (15.0, 0.0, 3.0, 3.0);  (precision, recall, F1)= (0.8333333, 0.8333333, 0.8333333);  Lenient: (0.8333333, 0.8333333, 0.8333333)

For each pair of annotators, and for each label:
(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 3.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (15.0, 0.0, 0.0, 3.0);  (precision, recall, F1)= (1.0, 0.8333333, 0.90909094);  Lenient: (1.0, 0.8333333, 0.90909094)


For the document: review-10_7.xml_00160
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (23.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.9583333, 1.0, 0.9787234);  Lenient: (0.9583333, 1.0, 0.9787234)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (23.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.9583333, 1.0, 0.9787234);  Lenient: (0.9583333, 1.0, 0.9787234)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (22.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.95652175, 1.0, 0.9777778);  Lenient: (0.95652175, 1.0, 0.9777778)


For the document: review-11_64.xml_00161
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (4.0, 0.0, 2.0, 2.0);  (precision, recall, F1)= (0.6666667, 0.6666667, 0.6666667);  Lenient: (0.6666667, 0.6666667, 0.6666667)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (4.0, 0.0, 2.0, 2.0);  (precision, recall, F1)= (0.6666667, 0.6666667, 0.6666667);  Lenient: (0.6666667, 0.6666667, 0.6666667)

For each pair of annotators, and for each label:
(0,1), label= 3_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (4.0, 0.0, 0.0, 2.0);  (precision, recall, F1)= (1.0, 0.6666667, 0.8);  Lenient: (1.0, 0.6666667, 0.8)


For the document: review-10_260.xml_00162
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators, and for each label:

For the document: review-11_443.xml_00163
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (10.0, 0.0, 13.0, 11.0);  (precision, recall, F1)= (0.4347826, 0.47619048, 0.45454544);  Lenient: (0.4347826, 0.47619048, 0.45454544)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (10.0, 0.0, 13.0, 11.0);  (precision, recall, F1)= (0.4347826, 0.47619048, 0.45454544);  Lenient: (0.4347826, 0.47619048, 0.45454544)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (2.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

(0,1), label= 3_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 4.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 8.0, 1.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (8.0, 0.0, 1.0, 10.0);  (precision, recall, F1)= (0.8888889, 0.44444445, 0.5925926);  Lenient: (0.8888889, 0.44444445, 0.5925926)


For the document: review-11_327.xml_00164
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (4.0, 0.0, 1.0, 1.0);  (precision, recall, F1)= (0.8, 0.8, 0.8000001);  Lenient: (0.8, 0.8, 0.8000001)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (4.0, 0.0, 1.0, 1.0);  (precision, recall, F1)= (0.8, 0.8, 0.8000001);  Lenient: (0.8, 0.8, 0.8000001)

For each pair of annotators, and for each label:
(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (4.0, 0.0, 0.0, 1.0);  (precision, recall, F1)= (1.0, 0.8, 0.88888896);  Lenient: (1.0, 0.8, 0.88888896)


For the document: review-11_186.xml_00165
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 1.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 1.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators, and for each label:
(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 0.0, 1.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)


For the document: review-10_261.xml_00166
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators, and for each label:

For the document: review-11_374.xml_00167
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (10.0, 0.0, 5.0, 4.0);  (precision, recall, F1)= (0.6666667, 0.71428573, 0.68965524);  Lenient: (0.6666667, 0.71428573, 0.68965524)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (10.0, 0.0, 5.0, 4.0);  (precision, recall, F1)= (0.6666667, 0.71428573, 0.68965524);  Lenient: (0.6666667, 0.71428573, 0.68965524)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (3.0, 0.0, 1.0, 1.0);  (precision, recall, F1)= (0.75, 0.75, 0.75);  Lenient: (0.75, 0.75, 0.75)

(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 2.0, 1.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (7.0, 0.0, 2.0, 2.0);  (precision, recall, F1)= (0.7777778, 0.7777778, 0.77777773);  Lenient: (0.7777778, 0.7777778, 0.77777773)


For the document: review-11_28.xml_00168
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)


For the document: review-10_413.xml_00169
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (16.0, 0.0, 9.0, 9.0);  (precision, recall, F1)= (0.64, 0.64, 0.64);  Lenient: (0.64, 0.64, 0.64)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (16.0, 0.0, 9.0, 9.0);  (precision, recall, F1)= (0.64, 0.64, 0.64);  Lenient: (0.64, 0.64, 0.64)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 4.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 2_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 4.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (16.0, 0.0, 0.0, 9.0);  (precision, recall, F1)= (1.0, 0.64, 0.7804878);  Lenient: (1.0, 0.64, 0.7804878)


For the document: review-10_118.xml_0016A
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (2.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (2.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

For each pair of annotators, and for each label:
(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (2.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)


Macro averaged over 22 documents:

For each pair of annotators, each type and each label:
Annotation type *comment*
For each pair of annotators
For pair (Output,Key):   (correct, paritalCorrect, spurious, missing)= (6.818182, 0.0, 2.6818182, 2.4545455);  (precision, recall, F1)= (0.5957183, 0.60455143, 0.5999657);  Lenient: (0.5957183, 0.60455143, 0.5999657)

pairAnns=(Output,Key), type=comment, label=1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.45454547, 0.0, 0.36363637, 0.18181819);  (precision, recall, F1)= (0.2840909, 0.3068182, 0.2916667);  Lenient: (0.2840909, 0.3068182, 0.2916667)

pairAnns=(Output,Key), type=comment, label=2_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 0.09090909, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

pairAnns=(Output,Key), type=comment, label=3_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 0.36363637, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

pairAnns=(Output,Key), type=comment, label=4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.5909091, 0.09090909);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

pairAnns=(Output,Key), type=comment, label=5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (6.3636365, 0.0, 0.27272728, 2.1818182);  (precision, recall, F1)= (0.654409, 0.5187915, 0.57209855);  Lenient: (0.654409, 0.5187915, 0.57209855)

Overall pairs:   (correct, paritalCorrect, spurious, missing)= (6.818182, 0.0, 2.6818182, 2.4545455);  (precision, recall, F1)= (0.5957183, 0.60455143, 0.5999657);  Lenient: (0.5957183, 0.60455143, 0.5999657)

Overall pairs and types:   (correct, paritalCorrect, spurious, missing)= (6.818182, 0.0, 2.6818182, 2.4545455);  (precision, recall, F1)= (0.5957183, 0.60455143, 0.5999657);  Lenient: (0.5957183, 0.60455143, 0.5999657)

***ALG2***
annotation sets:
*Output*
*Key*

For the document: review-11_517.xml_0045B
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (12.0, 0.0, 6.0, 3.0);  (precision, recall, F1)= (0.6666667, 0.8, 0.72727275);  Lenient: (0.6666667, 0.8, 0.72727275)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (12.0, 0.0, 6.0, 3.0);  (precision, recall, F1)= (0.6666667, 0.8, 0.72727275);  Lenient: (0.6666667, 0.8, 0.72727275)

For each pair of annotators, and for each label:
(0,1), label= 3_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 2.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 3.0, 0.0);  (precision, recall, F1)= (0.25, 1.0, 0.4);  Lenient: (0.25, 1.0, 0.4)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (11.0, 0.0, 1.0, 3.0);  (precision, recall, F1)= (0.9166667, 0.78571427, 0.84615386);  Lenient: (0.9166667, 0.78571427, 0.84615386)


For the document: review-10_218.xml_0045C
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)


For the document: review-10_90.xml_0045D
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (3.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (3.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

For each pair of annotators, and for each label:
(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (3.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)


For the document: review-11_135.xml_0045E
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 1.0, 1.0);  (precision, recall, F1)= (0.5, 0.5, 0.5);  Lenient: (0.5, 0.5, 0.5)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 1.0, 1.0);  (precision, recall, F1)= (0.5, 0.5, 0.5);  Lenient: (0.5, 0.5, 0.5)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 0.0, 1.0);  (precision, recall, F1)= (1.0, 0.5, 0.6666667);  Lenient: (1.0, 0.5, 0.6666667)


For the document: review-11_269.xml_0045F
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators, and for each label:

For the document: review-10_233.xml_00460
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (15.0, 0.0, 6.0, 4.0);  (precision, recall, F1)= (0.71428573, 0.7894737, 0.75000006);  Lenient: (0.71428573, 0.7894737, 0.75000006)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (15.0, 0.0, 6.0, 4.0);  (precision, recall, F1)= (0.71428573, 0.7894737, 0.75000006);  Lenient: (0.71428573, 0.7894737, 0.75000006)

For each pair of annotators, and for each label:
(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 4.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (15.0, 0.0, 2.0, 4.0);  (precision, recall, F1)= (0.88235295, 0.7894737, 0.83333343);  Lenient: (0.88235295, 0.7894737, 0.83333343)


For the document: review-10_172.xml_00461
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)


For the document: review-10_484.xml_00462
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (13.0, 0.0, 2.0, 0.0);  (precision, recall, F1)= (0.8666667, 1.0, 0.92857146);  Lenient: (0.8666667, 1.0, 0.92857146)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (13.0, 0.0, 2.0, 0.0);  (precision, recall, F1)= (0.8666667, 1.0, 0.92857146);  Lenient: (0.8666667, 1.0, 0.92857146)

For each pair of annotators, and for each label:
(0,1), label= 2_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 3_Star_Review:   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (2.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.6666667, 1.0, 0.8);  Lenient: (0.6666667, 1.0, 0.8)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (10.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)


For the document: review-10_556.xml_00463
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (14.0, 0.0, 3.0, 1.0);  (precision, recall, F1)= (0.8235294, 0.93333334, 0.875);  Lenient: (0.8235294, 0.93333334, 0.875)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (14.0, 0.0, 3.0, 1.0);  (precision, recall, F1)= (0.8235294, 0.93333334, 0.875);  Lenient: (0.8235294, 0.93333334, 0.875)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 2.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (14.0, 0.0, 0.0, 1.0);  (precision, recall, F1)= (1.0, 0.93333334, 0.9655172);  Lenient: (1.0, 0.93333334, 0.9655172)


For the document: review-10_54.xml_00464
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (7.0, 0.0, 4.0, 3.0);  (precision, recall, F1)= (0.6363636, 0.7, 0.6666667);  Lenient: (0.6363636, 0.7, 0.6666667)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (7.0, 0.0, 4.0, 3.0);  (precision, recall, F1)= (0.6363636, 0.7, 0.6666667);  Lenient: (0.6363636, 0.7, 0.6666667)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 3.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (6.0, 0.0, 1.0, 3.0);  (precision, recall, F1)= (0.85714287, 0.6666667, 0.75);  Lenient: (0.85714287, 0.6666667, 0.75)


For the document: review-10_294.xml_00465
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (17.0, 0.0, 1.0, 1.0);  (precision, recall, F1)= (0.9444444, 0.9444444, 0.9444444);  Lenient: (0.9444444, 0.9444444, 0.9444444)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (17.0, 0.0, 1.0, 1.0);  (precision, recall, F1)= (0.9444444, 0.9444444, 0.9444444);  Lenient: (0.9444444, 0.9444444, 0.9444444)

For each pair of annotators, and for each label:
(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (2.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.6666667, 1.0, 0.8);  Lenient: (0.6666667, 1.0, 0.8)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (15.0, 0.0, 0.0, 1.0);  (precision, recall, F1)= (1.0, 0.9375, 0.9677419);  Lenient: (1.0, 0.9375, 0.9677419)


For the document: review-10_7.xml_00466
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (22.0, 0.0, 2.0, 1.0);  (precision, recall, F1)= (0.9166667, 0.95652175, 0.9361702);  Lenient: (0.9166667, 0.95652175, 0.9361702)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (22.0, 0.0, 2.0, 1.0);  (precision, recall, F1)= (0.9166667, 0.95652175, 0.9361702);  Lenient: (0.9166667, 0.95652175, 0.9361702)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (22.0, 0.0, 1.0, 1.0);  (precision, recall, F1)= (0.95652175, 0.95652175, 0.95652175);  Lenient: (0.95652175, 0.95652175, 0.95652175)


For the document: review-11_64.xml_00467
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (3.0, 0.0, 3.0, 2.0);  (precision, recall, F1)= (0.5, 0.6, 0.54545456);  Lenient: (0.5, 0.6, 0.54545456)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (3.0, 0.0, 3.0, 2.0);  (precision, recall, F1)= (0.5, 0.6, 0.54545456);  Lenient: (0.5, 0.6, 0.54545456)

For each pair of annotators, and for each label:
(0,1), label= 3_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (3.0, 0.0, 1.0, 2.0);  (precision, recall, F1)= (0.75, 0.6, 0.6666667);  Lenient: (0.75, 0.6, 0.6666667)


For the document: review-10_260.xml_00468
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators, and for each label:

For the document: review-11_443.xml_00469
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (10.0, 0.0, 13.0, 9.0);  (precision, recall, F1)= (0.4347826, 0.5263158, 0.47619045);  Lenient: (0.4347826, 0.5263158, 0.47619045)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (10.0, 0.0, 13.0, 9.0);  (precision, recall, F1)= (0.4347826, 0.5263158, 0.47619045);  Lenient: (0.4347826, 0.5263158, 0.47619045)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 2.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 3_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 4.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 7.0, 0.0);  (precision, recall, F1)= (0.125, 1.0, 0.22222222);  Lenient: (0.125, 1.0, 0.22222222)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (9.0, 0.0, 0.0, 9.0);  (precision, recall, F1)= (1.0, 0.5, 0.6666667);  Lenient: (1.0, 0.5, 0.6666667)


For the document: review-11_327.xml_0046A
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (4.0, 0.0, 1.0, 1.0);  (precision, recall, F1)= (0.8, 0.8, 0.8000001);  Lenient: (0.8, 0.8, 0.8000001)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (4.0, 0.0, 1.0, 1.0);  (precision, recall, F1)= (0.8, 0.8, 0.8000001);  Lenient: (0.8, 0.8, 0.8000001)

For each pair of annotators, and for each label:
(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (4.0, 0.0, 0.0, 1.0);  (precision, recall, F1)= (1.0, 0.8, 0.88888896);  Lenient: (1.0, 0.8, 0.88888896)


For the document: review-11_186.xml_0046B
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

For each pair of annotators, and for each label:
(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)


For the document: review-10_261.xml_0046C
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators, and for each label:

For the document: review-11_374.xml_0046D
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (12.0, 0.0, 3.0, 1.0);  (precision, recall, F1)= (0.8, 0.9230769, 0.85714287);  Lenient: (0.8, 0.9230769, 0.85714287)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (12.0, 0.0, 3.0, 1.0);  (precision, recall, F1)= (0.8, 0.9230769, 0.85714287);  Lenient: (0.8, 0.9230769, 0.85714287)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (3.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.75, 1.0, 0.85714287);  Lenient: (0.75, 1.0, 0.85714287)

(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 2.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (9.0, 0.0, 0.0, 1.0);  (precision, recall, F1)= (1.0, 0.9, 0.9473684);  Lenient: (1.0, 0.9, 0.9473684)


For the document: review-11_28.xml_0046E
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 1.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 1.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 1.0, 0.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.0, 0.0, 0.0, 1.0);  (precision, recall, F1)= (0.0, 0.0, 0.0);  Lenient: (0.0, 0.0, 0.0)


For the document: review-10_413.xml_0046F
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (23.0, 0.0, 2.0, 2.0);  (precision, recall, F1)= (0.92, 0.92, 0.92);  Lenient: (0.92, 0.92, 0.92)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (23.0, 0.0, 2.0, 2.0);  (precision, recall, F1)= (0.92, 0.92, 0.92);  Lenient: (0.92, 0.92, 0.92)

For each pair of annotators, and for each label:
(0,1), label= 1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (4.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

(0,1), label= 2_Star_Review:   (correct, paritalCorrect, spurious, missing)= (1.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

(0,1), label= 4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (2.0, 0.0, 2.0, 0.0);  (precision, recall, F1)= (0.5, 1.0, 0.6666667);  Lenient: (0.5, 1.0, 0.6666667)

(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (16.0, 0.0, 0.0, 2.0);  (precision, recall, F1)= (1.0, 0.8888889, 0.94117653);  Lenient: (1.0, 0.8888889, 0.94117653)


For the document: review-10_118.xml_00470
For the annotation type *comment*
Compute the IAA for 2 Annotator(s) and on 1 document(s).
Annotation feature=*rating*
Fmeasures averaged over 1 pairs of annotators.
  (correct, paritalCorrect, spurious, missing)= (2.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

For each pair of annotators:
(0,1):   (correct, paritalCorrect, spurious, missing)= (2.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)

For each pair of annotators, and for each label:
(0,1), label= 5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (2.0, 0.0, 0.0, 0.0);  (precision, recall, F1)= (1.0, 1.0, 1.0);  Lenient: (1.0, 1.0, 1.0)


Macro averaged over 22 documents:

For each pair of annotators, each type and each label:
Annotation type *comment*
For each pair of annotators
For pair (Output,Key):   (correct, paritalCorrect, spurious, missing)= (7.2727275, 0.0, 2.2272727, 1.3636364);  (precision, recall, F1)= (0.6147003, 0.6542348, 0.63304156);  Lenient: (0.6147003, 0.6542348, 0.63304156)

pairAnns=(Output,Key), type=comment, label=1_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.4090909, 0.0, 0.4090909, 0.0);  (precision, recall, F1)= (0.17045455, 0.18181819, 0.17532468);  Lenient: (0.17045455, 0.18181819, 0.17532468)

pairAnns=(Output,Key), type=comment, label=2_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.045454547, 0.0, 0.045454547, 0.0);  (precision, recall, F1)= (0.045454547, 0.045454547, 0.045454547);  Lenient: (0.045454547, 0.045454547, 0.045454547)

pairAnns=(Output,Key), type=comment, label=3_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.045454547, 0.0, 0.3181818, 0.0);  (precision, recall, F1)= (0.045454547, 0.045454547, 0.045454547);  Lenient: (0.045454547, 0.045454547, 0.045454547)

pairAnns=(Output,Key), type=comment, label=4_Star_Review:   (correct, paritalCorrect, spurious, missing)= (0.4090909, 0.0, 1.1818181, 0.0);  (precision, recall, F1)= (0.14583334, 0.27272728, 0.17676769);  Lenient: (0.14583334, 0.27272728, 0.17676769)

pairAnns=(Output,Key), type=comment, label=5_Star_Review:   (correct, paritalCorrect, spurious, missing)= (6.3636365, 0.0, 0.27272728, 1.3636364);  (precision, recall, F1)= (0.65284926, 0.5571863, 0.5953047);  Lenient: (0.65284926, 0.5571863, 0.5953047)

Overall pairs:   (correct, paritalCorrect, spurious, missing)= (7.2727275, 0.0, 2.2272727, 1.3636364);  (precision, recall, F1)= (0.6147003, 0.6542348, 0.63304156);  Lenient: (0.6147003, 0.6542348, 0.63304156)

Overall pairs and types:   (correct, paritalCorrect, spurious, missing)= (7.2727275, 0.0, 2.2272727, 1.3636364);  (precision, recall, F1)= (0.6147003, 0.6542348, 0.63304156);  Lenient: (0.6147003, 0.6542348, 0.63304156)