2.1. Preliminary studies

Exploratory studies for identifying means for segregating text were first conducted. One way to classify documents or their parts is to list the words and their frequency in the text being considered. In a preliminary exercise, this approach was tested on some documents; the results of the classification were not always indicative of the content at the sentence level. As mentioned before, TextTiling is a well-known method for segmenting sections from a given text. It is freely available as an implementation in NTLK, a Python based Natural Language Tool Kit (Loper & Bird, Reference Loper and Bird2002). This method was tested on a portion of a test document with 4303 words (http://www.oup.com/us/static/companion.websites/9780195157826/chapter19.pdf). The outcome was compared with outcomes from the document being given to human subjects (see Fig. 2). The horizontal axis represents the position of each sentence in the document. The red markings at the bottom indicate areas in the document where 50% or more of subjects saw a coherent segment, that is, without considerable change of topic. The bottom row shows segmentation provided by the TextTiling implementation. While using the algorithm, two parameters (block length and the block size) had to be adjusted to obtain a reasonable number of segments. The combination with maximum number of segments (39) was considered.

Fig. 2. Segment breaks indicated by TextTiling and human subjects.

Some observations on the results are as follows:

• • The two segmentations (using Tiling and using manual subjects) match in most of the locations. However, TextTiling looks at paragraph breaks as a shift in focus, for example, for an itemization in the document, which was not perceived as a shift by all but one of the subjects. This, in contrast, was treated as four segments by TextTiling.

• • When there were multiple segments in a paragraph, except at one point (that arose due to formatting issues in the input), tiling performed as expected. In addition, tiling's segments matched with three subjects on four instances, with two subjects on three instances, with one subject on four instances, and with no subjects on one instance. Hence, the performance for tiling was taken to be satisfactory in this case.

The following conclusions were derived from this study:

• • TextTiling performed segmentation at the most prominent segment boundaries. However, it also performed segmentation at other boundaries not identified by the test subjects.

• • It was difficult to adjust the block-length and block-size parameters for every document. They directly influence the number of segments that are produced as output.

• • Segmentation is only the first step in filtering relevant portions from text. It was important to understand the content of a document and extract diagnostic knowledge from it. Even the next step, classification, cannot be handled using the output from tiling alone. Hence, extracting the semantic content was necessary for subsequent steps in the research.

An additional case for using understanding-based techniques was made by the fact that methods that analyze words and their meanings do not address the task of resolving pronouns and other anaphora. Anaphora are words that refer to other words that have been used previously in a text. For example, pronouns are used to avoid repeating words. Resolving anaphora is important because they implicitly contain references to other words (in the same sentence or in other sentences) and may not be captured and counted by such methods.

At this point, it is useful to consider a document as a discourse from the author to the reader. The discourse context is useful for identifying the things being talked about in a sentence. In a given discourse, the current context is defined by the entities that are being talked about, the activities that concern them, and the relations among these entities. The list of entities is called a discourse entity list (Allen, Reference Allen2011). Based on this idea, we now explain our proposed method for segregation (i.e., segmentation and classification).

3.1. Discourse analysis

3.1.2. Discourse representation structure (DRS)

The theory used to understand discourses in this work is called discourse representation theory (Kamp & Reyle, Reference Kamp and Reyle1993). This theory models discourses as a combination of entities and conditions. This information is represented in a structure called DRS (Blackburn & Bos, Reference Blackburn and Bos2006). An example (drawn using Python natural language toolkit) is shown in Figure 3. A DRS has two components: entities and conditions. Discourse referents are those referring to entities in a DRS. They may contain pronouns, which are in turn resolved using an identity assignment. Discourse conditions are first-order logic conditions showing the relations between discourse entities. The conditions are predicates that convey the meaning of these sentences. They can also contain statements that convey the resolution of pronouns. These conditions may also contain other DRSs. Figure 3 shows a DRS for an example sentence. In the figure, x1, x2, and x4 are top-level entities, where x1 is a discourse referent to the entity riveting, and n_riveting(x1) is a condition that says x1 refers to riveting. x4 is a process, which is explained using the condition prop(x2,[DRS]), where [DRS] is the nesting of a DRS within another that explains the plates and pin using other conditions.

Fig. 3. An example of a discourse representation structure for the sentence “Riveting is the process of joining two plates of metal using a pin called a rivet.”

3.2. Assumptions

The following assumptions were made in our research:

1. 1. A document is treated as a one-way discourse between the author and the reader.

2. 2. The knowledge represented in documents are correct and valid.

3. 3. Available semantic resources such as dictionaries and lexica are sufficient to cover the language used in technical documents.

3.3. Proposed method for segregation

Having established the need to use discourse analysis for our segregation exercise, we present below the steps of our proposed method:

1. 1. Segmentation (shown in Fig. 1a):

   1. a. Tokenize the input text into sentences.

   2. b. In every sentence, resolve anaphora. They may be within a sentence or across sentences.

   3. c. Obtain a list of discourse entities, including those referred to by anaphora, to obtain a discourse entity list for each sentence.

   4. d. Segment the sentences that are both physically close to one another and share parts of their discourse entity lists within a threshold.

2. 2. Classification (shown in Fig. 1b):

   1. a. Evaluate the entities in the discourse entity list of each segment to determine how many of them relate to the relevant domain. The basis for comparison are one or more ontologies.

   2. b. If the discourse entity list for a segment matches with the domain ontology by more than a certain threshold, then classify that segment as being related to aircraft assembly.

The next section describes the implementation and validation of the two steps in the segregation procedure.

4.1. Implementation and validation of segmentation

4.1.1. Implementation

This section describes the details of implementation and validation of the segmentation part. The implementation of the segmentation part of the proposed method required the use and integration of various natural language understanding and processing tools. An overview of the specific procedure developed for implementation is described in Figure 1a.

Assigning sentence indices

The first step is to tokenize the text into individual sentences. One possibility is to use punctuation markers, such as a period. However, in the next step, we use an anaphora resolution tool called JavaRAP (Qiu, Reference Qiu, Kan and Chua2004, and JavaRAP website). JavaRAP uses sentence indices to refer to anaphora and their antecedents. Hence, it is meaningful to use sentence and word indices used by JavaRAP. The JavaRAP Sentence Splitter utility is used, and it assigns indices starting with zero. For example, in Figure 4, (1, 6) refers to the seventh word in the second sentence.

Fig. 4. Anaphora resolution output.

Anaphora resolution

After assigning indices to sentences, the next step is to resolve the anaphora in the input text, for which the JavaRAP tool is used. Although Boxer and C&C Tools is capable of anaphora resolution, we chose to perform anaphora resolution separately due to the weak ability of these tools in this respect (Bos, Reference Bos2008). The text is supplied to JavaRAP, which outputs resolved anaphora as shown in Figure 4. The indices identified earlier are used to identify and connect anaphora and their referents.

Replacing anaphora

After resolving anaphora, we needed to replace anaphora with their discourse referents, so that the actual discourse entity is used while interpreting sentences. Otherwise, the correct entity would have to be substituted in the interpreted form. An example is shown below:

Original Sentence: “Riveting is a complicated process. It involves many parts and tools.”

After replacing “it,” the second sentence becomes “Riveting involves many parts and tools.”

As one would expect, the challenge in replacing anaphora with their referents is that the correct form of the referent has to be replaced. Consider the sentence: “Manufacturing is the process of realizing products from their design.” After replacing anaphora, this becomes “Manufacturing is the process of realizing products from products design.”

The modified sentence is almost fine, except for the missing apostrophe in the possessive noun. The correct form has to be captured using grammar rules. The bigger challenge is when the referents of an anaphora are phrases. Such complex forms are not yet handled in our implementation. This requires expansion of the procedure to interpret anaphora results.

Interpretation of sentences

The next step was to obtain a semantic interpretation of sentences to get their meaning. The input are individual sentences and the output are DRSs. The tool used for the interpretation task was the C&C and Boxer tool set (Curran et al., Reference Curran, Clark and Bos2007). These tools have interfaces built into the Natural Language Tool-Kit (NLTK); the implementation was written in Python. An example interpretation is shown in Figure 5. It is in a different form than the “boxed one shown in Figure 6. It is in a more computer-understandable form, a recursive list. Similar to Figure 3, there are a list of discourse referents here, for example, x1, x4, and discourse conditions, such as n_riveting(x1) and n_process(x4). The other conditions represents specific predicates in first-order logic; for example, v_join represents the action of joining, the predicate of connects two referents, and prop is a proposition composing another DRS.

Fig. 5. An example of discourse representation structure interpretation of a sentence.”

Fig. 6. An example of discourse entities in the sentence.

Obtaining discourse entity list

It was mentioned that semantic interpretation gives us the list of discourse entities. From the DRS interpretation of sentences, it is possible to directly obtain the list of discourse referents for that sentence. In Figure 5, the discourse referents are the variables x1, x2, and x4. One can get the predicate for discourse entities by reading the discourse conditions for an entity, for example, n_xxxxx(), ne_nam_xxxx(), and so on. This is different from using only the part-of-speech (POS) tags, because the conditions convey more information, for example, relation of an object to other objects and/or events. An example of the list of discourse entities for a sentence is shown in Figure 6.

Measure semantic similarity between sentences

The next step of implementation was to measure the similarity between every two consecutive sentences. This measure was used to identify segments based on large jumps in meaning. We measured how similar or different two consecutive sentences are to or from each other. For this, we propose a measure based on the semantic similarity between words. Examples of similarity measures between individual words in WordNet (Miller, Reference Miller1995) are Jiang-Conrath similarity, Lin Similarity, and path similarity (Mihalcea et al., Reference Mihalcea, Corley and Strapparava2006). Starting with such word similarity measures, we arrived at a similarity measure between two sets of words, the sets being discourse entity lists of adjacent sentences.

For example, consider the two lists

$$\eqalign{& \left[{\rm \char"AA }quantity\comma{\rm \char"BA } \, {\rm \char"AA }part\comma{\rm \char"BA } \, {\rm \char"AA }riveting\comma{\rm \char"BA } \, {\rm \char"AA }tools{\rm \char"BA } \right] \, {\rm and} \,\cr & \quad\left[{\rm \char"AA }surfaces\comma{\rm \char"BA } \, {\rm \char"AA }rivet\comma{\rm \char"BA } \, {\rm \char"AA }problem\comma{\rm \char"BA } \, {\rm \char"AA }access{\rm \char"BA } \right]}$$

The similarity for all the words from the first list to the words in the second list can be averaged to get a single measure. Even then, at least three choices were available for indicating similarity of one word from the first list to the words of the second list. These were

1. 1. maximum among similarities of a word from the first list to all words of the second list,

2. 2. minimum among similarities of a word from the first list to all words of the second list, and

3. 3. average of similarities of a word from the first list to all words of the second list.

Depending on which option is chosen from the above, the final averaging for the overall score between the two lists differs. In the first two cases, they would have to be averaged over the number of elements of the first set. In the third case, the average must be taken over the elements of both sets.

Table 1 shows how the above calculations for each word pair were carried out in the implementation. For each word pair such as “quantity”–“surfaces,” we used the Lin similarity measure of these words.

Table 1. Different similarity measures between words from two lists

For the above lists of words, the overall similarities between the lists are

• • average max: 0.4401

• • average min: 0.0

• • average of averages: 0.0381

Identification of segments from values of similarity

Once the similarity between sentences was determined, segment boundaries were identified by tracking large changes in the similarity values. Literature does not provide any definitive strategy to identify such changes. The closest that could be used is the one by Hearst (Reference Hearst1994), who used a method of segmentation based on change of slope and a cutoff value as threshold.

In this research, we propose a strategy for finding such segments based on a comparative study of a manual reading exercise versus our calculated values. This is discussed in the following section.

4.1.2. Validation of segmentation part

A comparative study was carried out to validate the segmentation implementation. The purpose was to check if the implementation would identify changes in topic as seen by human subjects.

A sample document was constructed for this purpose. It reflected the needs of the validation; it had sections with varying topics where some variations in topic were strong, while others were not so strong. The variation in topic was largely linear in nature (meaning the structure of discourse was not hierarchical). The document was mostly about assembly processes and riveting. Two completely unrelated sections (about “running” and “employee salaries”) were deliberately inserted in the document. There were 31 sentences in total. We did not compare this result with TextTiling because it was not clear what values of the parameters used in TextTiling would serve as a standard for comparison. Hence, the comparative study with human subjects was carried out.

A total of 31 subjects were asked to give a score of how similar adjacent sentences were on a scale of 1 to 5 (1 = most dissimilar and 5 = most similar). Their feedback is shown in Figure 7. Using subjects' scores, we marked segments that were considered large changes in the topic of discussion. Here, large changes were due to (a) a drastic decrease in the similarity scores and (b) low or very low score of similarity.

Fig. 7. Feedback obtained from 31 subjects for a total of 30 adjacent sentence-pair similarities.

Two gradations of the above scores were considered: a major change of a decrease of 3 or 4 for (a) and a value of 1 for (b) and a minor change of a decrease of 2 for (a) and a value of 2 for (b).

Table 2 shows a summary of how many subjects matched the above criteria. For a major cutoff, we considered 15 subjects for a value to be considered significant. The prominent segments are at Sentences 5, 7, 22, and 25. A minor segment can also be seen at 20, if we relax the cutoff to 10 subjects.

Table 2. Numbers of subjects for various values and differences of intersentence similarity

Note: The shaded areas indicate where a majority of subjects indicated a drastic decrease or a low value of similarity.

The implementation was tested on this text to calculate similarity scores between adjacent sentences. It was then used to calculate a single measure to indicate if the meaning (and the context that decides the topic) changes as the program reads through the text. To develop a score of similarity that would reflect a change in context, we tried various options including average-, min-, and max-based values of intersentence similarity.

Comparing the plots with the subjects' feedback, none of these individual measures corresponded well for most locations. We realized that this was because each of these measures behaved differently. To explain, consider the analogy of the two word lists as two clusters of points. The average of two words can be considered to represent the center of the clusters, and the min and max, respectively, would represent the two closest and farthest points.

It was decided to combine all three measures into a single measure, in the hope of using their individual characteristics. The proposed measure was calculated as follows:

• • Calculate the change in average-, minimum-, and maximum-based scores of every sentence pair. Hence, we have (n – 2) such difference values for n sentences and (n – 1) sentence pairs.

• • Normalize each of the difference values by dividing by the highest respective (absolute) value. The normalization is done so as to give equal weights to each of the scores.

• • Sum up the normalized difference values.

The measure for a given sentence pair i was thus calculated as

$$\hbox{SM}_i = \displaystyle{{\hbox{Avg}_i - \hbox{Avg}_{i - 1}} \over {\mathop {\max} \limits_i (\hbox{Avg}_i - \hbox{Avg}_{i - 1} )}} + \displaystyle{{\hbox{Max}_i - \hbox{Max}_{i - 1}} \over {\mathop {\max} \limits_i (\hbox{Max}_i - \hbox{Max}_{i - 1} )}} + \displaystyle{{\hbox{Min}_i - \hbox{Min}_{i - 1}} \over {\mathop {\max} \limits_i (\hbox{Min}_i - \hbox{Min}_{i - 1} )}}.$$

The absolute values of intersentence similarity and the values of SM _i are plotted as shown in Figures 8 and 9, respectively. In Figure 9, if we consider a cutoff of –0.5 for the values of SM _i , we get five points as shown in the plot. Two are the points indicated by a majority of subjects (Points 3, 5 corresponding to Sentences 5, 7). Further, at a cutoff of –0.25, we can observe another major point marked by subjects (Point 20–Sentence 22). A minor point indicated by subjects (Point 18–Sentence 20) also has a negative value. However, we still needed to explain the other minimum points in the plot, which have not been correspondingly rated by subjects.

• • Point 10 has a minimum value because the first of the two sentences and is a section heading having one word only. This single word (“Riveting”) is responsible for small values of average and min values, while the max value is high.

• • The other low values are Points 12, 14, 16, 17, 18, and 29.

  • • Point 12 was also marked by eight subjects, and also the word “sealed” has not been considered by the program, it being a verb.

  • • For Point 14, “riveted” was a verb and not considered for a comparison.

  • • Point 16 also has some issues, such as not considering “rivet,” but considering words like “such” and “thing” (with no WordNet Entry being present for “such”). Hence, correct similarity values have not been calculated at this place between the sentences “It is possible to rivet plates of large thickness, such as in bridges” and “It is also possible to rivet sheet-metal, as is the case in aircraft construction.”

  • • Point 17 is between Sentences 18 and 19. The word “rivet” is in a verb form in Sentence 18, thus reducing the value of the Similarity measure with Sentence 19.

  • • Point 18 has been indicated as a minor point (by 12 subjects for individual scores and 12 subjects for difference).

  • • Similarly Point 24 has low scores. There were two reasons. First, anaphora resolution for “them” failed. Second, no suitable entries for the word “salary” were found. Together it has resulted in the entity “salary” being counted out twice for similarity measurement. This is also possibly the reason that Point 23 has a high value, because we assign a default of 1.0 to Min to start with.

  • • Point 29 corresponds to sentence pairs 29–30 and 30–31. The dip in value is interesting because, both 30 and 31 are related to 29 only (it is similar to an itemization). Hence, the value between Sentence 30 and Sentence 31 is not high, although subjects have marked it so.

Fig. 8. Values of intersentence similarity between adjacent sentences.

Fig. 9. Values of SM _i for between adjacent sentences.

From the results, it was observed that a large decrease in slope indicates the presence of a change of topic. Titles of sections, when included, created an anomaly. In addition, a combination of the difference of average, min, and max values was necessary to distinguish segments.

Some issues and possible improvements in the implementation were the following:

• • Some words did not have an equivalent synset (synonym entry) in the WordNet lexicon. Riveting can be either a noun or a verb. We have currently chosen the closest noun of another form of the word using the morphy utility in NLTK WordNet.

• • Verbs can also be counted for semantic similarity; for example, manufacturing is a verb only. Hence, we used its closest WordNet entry. However, an attempt to do so did not raise the similarity score very well, because there were some verbs that do not contribute to the score. For example, phrasal verbs like “have” were distractive, lowering the score where one would normally ignore it.

• • A linear change in context was assumed, and this was not always the case.

4.2. Implementation and validation of classification

This section describes the implementation and validation of the classification part.

4.2.1. Implementation

The classification part of our proposed method was initially implemented on a sentence-by-sentence basis. The overall procedure for implementation (see Section 3.3 for the method) is shown in Figure 1b.

Tools used

For implementing the proposed method for classification, a combination of tools was used. These are

• • Boxer and C&C Tools, to provide a representation in the form of DRSs.

• • Python Natural Language ToolKit, to perform routine natural language processing tasks (e.g., tokenizing text)

• • Ontology related tool (Protégé), to manually read an existing ontology file (in OWL format).

• • LaTeX, to print classified sentences with different colors (see Fig. 10).

Fig. 10. Example of sentences classified using the implementation.

The Python script tokenized a given text into sentences. These sentences were then interpreted as DRSs, and for each the entities (indicated by Boxer) were extracted and listed as discourse entities. A list of aircraft-related terms was already obtained from an AIRCRAFT ontology (Ast et al., Reference Ast, Glas, Roehm and Luftfahrt2014) and was used as a reference. A screen grab of this ontology's hierarchy (as seen in Protégé) is shown in Figure 11. At this stage, this list is derived only from the class hierarchy of the ontology. It is always possible to expand this list from the object properties and the class descriptions.

Fig. 11. A screen grab of the AIRCRAFT ontology as seen in Protégé.

4.2.2. Validation

After the implementation of the classification method, the next step was to validate the method on test data. It was not clear whether a gold standard exists for a task comprising discourse segmentation and classification. Thus, a document that was available in the public domain was used for testing; it was part of the Wikipedia article for Riveting (http://www.en.wikipedia.org/wiki/Rivet). The document was manually classified by the researchers and by test subjects who were master's and doctoral students and members of project staff at the university. The document consisted of a mix of sentences that did and did not relate to the aircraft domain. It had 177 sentences, from various domains including aircraft domain. However, it did not contain any terms from the ontology other than the terms “aircraft” and “wing.”

Figure 12 shows the responses of 15 subjects as to which sentences they thought were related to the aircraft domain, versus the classification made by the implemented method. The figure shows the relevant sentences along the abscissa, and each subject's identifier along the ordinate axis. Shown at the top is the classification performed using the implementation. There were clear clusters of sentences that were common among these classifications.

Fig. 12. Comparison of the performance of the implementation against those of subjects.

Observations

A few interesting observations were made during the preliminary validation. The background domain knowledge of the subject seemed to influence how each subject understood domain-related terms. Domain knowledge might have helped subjects with their inferences for terms such as “aerodynamic drag” for classification (and this was not present in the AIRCRAFT ontology). Although different subjects treated the text differently (e.g., some of them read some sentences together with other sentences), there were a set of sentences that were classified by almost all of the subjects.

Another observation was that proximity of sentences seemed to be a factor for some subjects to decide the relevance of sentence to the aircraft domain. For example, if two sentences that are separated by a sentence in between had been classified as related to aircraft domain by most subjects, some subjects marked the in-between sentence as also related to that domain. This indicates that they implicitly inferred the sentence to be related to these two sentences even though the sentence did not contain the terms they were looking for.

5.1. Test document

In order to test the integrated method, a test document was needed that represented the objectives of the work, as well as the capabilities of our method. Certain considerations were made while preparing the document. For example, the variation in topic had to be linear. Too many anaphora in the text might affect the accuracy of the method, because anaphora resolution capabilities were currently limited, as seen in Section 4.1.2. The document also needed to have domain-related and unrelated sections. In addition, as the independent validation of the segmentation revealed in Section 4.1.2, the usage of related words within a segment is a factor in reading a segment together. The document must also explicitly convey itself, meaning that its interpretation should not depend on background knowledge of the reader. A test document was prepared using parts of real-life documents (e.g., about aircraft construction). It had 42 sentences. A sample of the document used, along with the subject's response, is shown in Figure 13.

Fig. 13. A subject's response sheet for the reading exercise.

5.2. Validation

For validation, we conducted a reading exercise with subjects in two steps. The subjects were master's students, doctoral students, and members of project staff at the university. They held at least a bachelor's degree in engineering. The requirement for each subject was that he/she must have had at least a minimum exposure to engineering terminology, and a reasonable grasp on the English language. Subjects' responses for segmentation and classification are presented in Table 3.

Table 3. Number of subjects for both segment boundaries and relevance of sentences

Note: The dark and light shaded areas show where a major and minor number of subjects indicated relevance and segment boundary, respectively.

In the first step, we asked 30 subjects to read the document (individually). They were clearly instructed to mark segments that conveyed a shift in topic of discussion, and to assign a score of 1–3 for each shift (1 = not so strong, 3 = very strong). They were also asked to explain the reason for that marking.

In the second step, the subjects were asked to mark the segments related to the aircraft domain. Domain relevance was limited to aircraft domain, as other domains like assembly needed more knowledge and further resources for their implementation (such as an assembly ontology). Subjects were asked to mark out the whole segment even if only a part of it was related. They were asked to give a score of how strong the segment relevance was, on a scale of 1–3 (1 = not so strong, 3 = very strong; the scale was kept from 1 to 3 only because, unlike the individual validation where it was 1 to 5, we do not have opposing qualities in marking, and only those with a strong value are marked anyways). Scores given by 30 subjects were consolidated and tabulated. We present the results of both steps in Table 3.

The Python program was then run to find out the segments and classify them. Similar to Section 5.2, the values of intersentence similarity and the difference between their adjacent values were plotted. Figures 14 and 15, respectively, show the absolute and similarity measure value plots of the intersentence similarity measures for the average, min, max, and sum variations.

Fig. 14. Values of intersentence similarity.

Fig. 15. Values of SM _i and max, avg, and min for adjacent sentences.

Here we also use a strategy similar to that in Section 4.1.2 to get a combination of low absolute values. We refined the strategy used earlier, by adding a heuristic. As usual, we took a combination of

• • Low values of average and max values: Here, cutoffs of 0.04 (major) and 0.03 (minor) for average and 0.3 (major) and 0.4 (minor) for max values were considered. Min values were not considered because there were many zero values.

• • Low value of Normalized Sum: Here, a major cutoff of –1 and min cutoff of –0.5 were used.

Using the above values, the heuristic was that even if one out of the three above values were a major cutoff, it would be a major segment change. Otherwise, the segment change is deemed minor. Applying this heuristic, major segment starts were identified at Sentences 7, 12, 18, 25, 28, 38, and 42. In addition, minor segments starts were at 5 and 34. Comparing this with Table 3, the segments identified by subjects were 7, 12, 19, 25, and 38. However, no corresponding feedback was present by subjects for the min points seen in the implementation. Let us consider each of the points that do not match.

• • For Sentence 5 (Point 3), a possible reason is low value of difference in Similarity Value, due to both incorrect semantic interpretation of Sentence 3 and the incorrect choice of synset entry for the word “household” in Sentence 4.

• • In Sentence 34 (Point 32), one direct reason we see is the choice of incorrect synset for the word “spar.” This again, is due to the problem of ambiguity.

• • Sentence 18 (Point 16) seems to have got a low value (and Point 17 a high value) due to the incorrect choice of synsets for the words “process” and “weld.” For “weld” this ambiguous choice resulted in a synset corresponding to a dye, rather than a joining of material. Similarly, for “process,” the ambiguity results in it being understood as a mental process, rather than a manufacturing one. This was also verified on the online similarity measuring tool WS4J (http://ws4jdemo.appspot.com).

• • In Sentence 28 (Point 26), one issue was that the phrase “power plant” was read as two words, and the word “plant” was interpreted as in a manufacturing plant, rather than an engine. Hence, ambiguity has again played a role in getting a low value of similarity.

Regarding the classification step, we looked at the relevant segments for the aircraft domain. Due to different segments being marked by different subjects, we looked at the aggregate numbers provided by subjects. Considering 20 Subjects as a major cutoff, and 15 as a minor cutoff, the relevant segments are marked in Table 4. The relevant semantic score as given by the subjects is also indicated.

Table 4. Relevance scores assigned by the implementation for its segments

Figure 16 shows the sentences indicated relevant by subjects, marked above the segments classified by the implementation. The bottom part of the figure shows the sentence index number. The two top rows of colored squares indicate segment boundaries and relevant sentences as given by subjects. (Red colors are major, and orange ones are minor). The two bottom rows indicate the same for the implementation.

Fig. 16. A comparison of classified segments by subjects and the implementation.

Out of 33 sentences classified by a large number of subjects, 28 sentences were classified by the program. Two extra sentences (12, 13) were included by the classification. They have not been counted in the subject's feedback because their segments were not all the same, and the counting (Table 3) was done on a per-sentence basis. Sentences 7 to 12 were classified as minor, relevant segments by subjects, but not by the implementation. It was found that although some parts of this segment did relate to the aircraft domain (such as talking about air transport), they did not contain any matching terms from the AIRCRAFT ontology. Because of this, the implementation did not identify them as relevant.