Results and Evaluation

This chapter presents the graphical results obtained from the experiments described in chapter 4, and commentary on the successful and unsuccessful topic detections by the system (when compared to a human annotator). The graphs show human-annotated and automatically detected topic breaks, superimposed over a representation of the cosine measure function from which the automatic topic breaks derive. This allows `near misses' to be examined, as well as a more complete view of topic cohesion trends beyond a thresholded binary view of topic change.

In each graph presented in this section, the black line represents the smoothed similarity value according to the cosine measure; the red lines represent the human-evaluated topic boundaries; and the dashed green lines represent the system's topic change location theories. The grey line represents the unsmoothed cosine similarity metric: its purpose in the graph is to allow visual detection of spikes which may have been removed by the smoothing algorithm in error. While the trough detection algorithm works using the smoothed data only, it is useful to see the cosine measure's original output for evaluation purposes.