SSSS Annotation Notes

July 15, 2013

For the training set:

Annotator 1 was speaker 6.  Atypically he did the annotations before
the Annotators Guide was available, he's not a student, and he's not
naive about information retrieval technology.

Annotator 2 was speaker 1.  Atypically he knows a lot about one
technique of possible value for this task. 

Annotator 3 was speaker 28.  Atypically she's not quite part of the CS
community, being a math major, although she was taking a CS course and
does IT work in her job.

Annotator 4 was speaker 2. Atypically, she has experience doing search
in audio archives, having spent about 10 hours doing so as part of the
experiment reported in our Interspeech 2012 paper.

Annotator 5 was speaker 30.

Annotator 6 was speaker 5.  Atypically, he is a member of the research
lab, having joined a few months ago, and roughly knows our aims and
favorite methods.



For the testset: 

Annotator 1 was the same as annoator 3 for the testset.

Annotators 2 and 3 were CS undergraduates.  They worked at the the
same table, and may have shared some thoughts on the tags.

Annotator 4 was a high-school student.  

The most salient difference between the testset and trainingset tags
was that there were many fewer long regions tagged in the testset.  In
the training set there was one region tagged over 4 minutes long
(tv-shows), four more over 3 minutes (playing-video-games,
programming-projects, movies-tv-shows, course-experiences), and 58
over two minutes.  In the testset there were only 5 longer than 2
minutes, and these were all for just one category of one annotator.

One reason for the difference is that we noticed the over-long tags in
the training set, and suggested to the testset annotators that it was
okay to leave large sections of the data without any tags, and that
could be appropriate to break up a long region on a single vague topic
(for example entertainment), into more specific contributions.
Another possible factor is that, with only 6 conversations to work on,
the testset annotators didn't suffer the same degree of desire to get
it over quickly.



in /home/users/nigel/papers/mediaeval/annotation-notes.txt