2.1. Understanding engineering project characteristics

The understanding of project characteristics has a direct influence on project success. The comprehensive understanding could help project members optimize the project structure, refine the project granularity settings, and make rational decisions on product design, manufacturing, and marketing (Cho et al., Reference Cho, Hong and Hyun2009; Li, Xie, et al., Reference Li, Xie and Goh2009).

In practice, project characteristics often have various definitions according to information needs. From an engineering design perspective, they are defined as the high-level interpretations of project features, which are associated with the project operation and management tasks (Snider et al., Reference Snider, Jones, Gopsill, Shi and Hicks2014). For certain types of projects, their characteristics can be defined in explicit forms. The characteristics of software engineering projects include, for example, the design paradigm, programming language, database type, and amount of created source code (Walter, Reference Walter2014). However, project characteristics can also be defined in implicit forms. For example, from a management perspective, project characteristics usually contain performance indicators such as elapsed time and cost (Cho et al., Reference Cho, Hong and Hyun2009). Meanwhile, the uncertainty and complexity are also considered as the key aspects of project characteristics (Ahmad et al., Reference Ahmad, Mallick and Schroeder2013). From an information management perspective, the trace of digital object creation and evolution can be considered a characteristic enabling project members to have a detailed understanding of engineering design processes (Gopsill et al., Reference Gopsill, Jones, Snider, Shi, McMahon and Hicks2014). From a human–computer interaction perspective, the change of sentiment or topic of communications provides project characteristics aimed at assisting project members to monitor work complexity and track project progress (Jones et al., Reference Jones, Payne, Hicks, Gopsill, Snider and Shi2015).

Engineering projects often involve multiple collaborators, utilize distributed resources, and handle heterogeneous and fragmented data. Consequently, their characteristics can be dynamic and time sensitive in nature. Understanding the evolving characteristics can be challenging for most project members (Engwall & Jerbrant, Reference Engwall and Jerbrant2003; Meredith & Mantel, Reference Meredith and Mantel2011). In most cases, the processing of project data, capturing, and modeling the project knowledge is often undertaken using expert effort to assess, evaluate, understand, and interpret the information into a usable form (Pascal et al., Reference Pascal, Thomas and Romme2013). This understanding and identification of characteristic is largely dependent upon the availability, capability, and experience of the project members. The following issues can occur due to these dependencies:

• • High cost on knowledge discovery and capture: In the characteristic understanding process, the knowledge concepts contained by the project data need to be discovered and captured. To perform this task, data items created during the project operation should be reviewed and analyzed. The identified knowledge concepts are often organized through the use of knowledge models, such as ontology, concept map, or semantic web. Relying on manual work means this task requires intensive efforts from the knowledge experts or experienced project members, implying it can be considerably time consuming and expensive.

• • Diversified perspectives of characteristic understanding: In order to understand the project characteristics at the appropriate level of granularity, project members are required to have the knowledge about the project components and the interrelationships between them (Shi, Gopsill, Snider, et al., Reference Shi, Gopsill, Snider, Jones, Newnes and Culley2014). Such knowledge could be the specific and precise descriptions of tasks, subsystems, project teams, the sequence of multiple tasks, the dependency of multiple subsystems, and the responsibilities of different project teams. Within a service project, numerous project members and project components are involved. Chandrasegaran et al. (Reference Chandrasegaran, Ramani, Sriram, Horváth, Bernard, Harik and Gao2013) describe how knowledge sharing and utilzation among the project teams can be difficult. Various knowledge capabilities of the project members could lead to different viewpoints/understanding for identical project components (Chungoora et al., Reference Chungoora, Young, Gunendran, Palmer, Usman and Anjum2013).

• • Low capability of processing complex data: A service project could generate various types of data, for example, e-mails, instant messages, formal reports, spreadsheets, computer-aided design models, and simulation files. During the characteristic learning process, the project members need to spend time on reviewing and analyzing such data. However, due to the restrictions on time and capability, processing large amount of complex data using manual work is not always realistic. More important, such data is likely to be generated by multiple sources, and have heterogeneous and fragmented forms, so that the manual data processing can cause the analysis results to lack accuracy or be problematic. This can be exacerbated if the project members lack certain knowledge, or when human errors occur.

To overcome these challenges, the analytical techniques such as data mining and machine learning need to be adopted.

2.2. Data mining and machine learning based models

With the application of information and communication technology, large volumes of data can be generated, captured, and stored during the project execution process. The data contains detailed information about project objectives, processes, outcomes, problems encountered, and lessons learned, so that it can be used to facilitate the understanding of project characteristics (Harding et al., Reference Harding, Shahbaz and Kusiak2006; Choudhary et al., Reference Choudhary, Harding and Tiwari2009).

To process the data in an effective manner, the use of data mining and machine learning techniques is considered to be critical. Data mining includes specifically designed computational approaches that could achieve automatic knowledge discovery, condition monitoring, pattern recognition, and association analysis. Meanwhile, machine learning includes advanced statistical methods that could achieve automatic decision making, predictive modeling, data classification, and data clustering. In practice, both techniques have been applied in various fields to create intelligent systems or analytical approaches (Chen et al., Reference Chen, Chiang and Storey2012; Kamsu-Foguem et al., Reference Kamsu-Foguem, Rigal and Mauget2013).

As stated by recent research, there is an increasing trend of using data mining and machine learning in manufacturing fields (Wang, Reference Wang2007; Köksal et al., Reference Köksal, Batmaz and Testik2011; Wu et al., Reference Wu, Thames, Rosen and Schaefer2012). Numerous applications and models have been proposed and introduced by the research community, with the aim of achieving engineering knowledge management, production process management, project characteristic identification, and performance monitoring.

To capture the process-related knowledge and identify the sequential activities of engineering projects, Shi, Gopsill, Newnes, et al. (Reference Shi, Gopsill, Snider, Jones, Newnes and Culley2014) proposed an analytical approach based on data mining and nature language processing techniques. The approach extracts knowledge concepts from engineering documents, and automatically interprets activity sequences of engineering projects. The output of this approach provides pictorial and numeric knowledge representations to the project members. The aim is to enable the project members to learn the structures of different projects processes, and also to help them understand the similarity and normality of such processes.

To understand the characteristics of the engineering design process, Gopsill et al. (Reference Gopsill, Jones, Snider, Shi, McMahon and Hicks2014) proposed an analytical tool. It applies metadata analysis and frequency analysis to identify the evolution of product design activities, that is, the creation and modification of computer-aided design and computational fluid dynamics files. By using this approach, the project members are able to assess the quantity of generated digital objects at different project stages, and to understand the dependency changes of such files over time.

For MRO activities, the analysis of service records would assess the product condition and predict the fault. Certain machine learning techniques such as support vector machine, artificial neural networks, and decision trees are considered useful. These data classification and decision-making approaches can be used to validate the product running status on a real-time basis by analyzing the operating data and historical data. Once any unexpected status of the product has been detected, these approaches can make intelligent decisions in an automatic manner (Widodo & Yang, Reference Widodo and Yang2007; Kankar et al., Reference Kankar, Sharma and Harsha2011).

To design the preventive maintenance solutions for complex engineering products or infrastructures, the integration of data mining and machine learning is necessary. For instance, to diagnose the fault of aircraft engines, Sahin et al. (Reference Sahin, Yavuz, Arnavut and Uluyol2007) proposed a model that integrates Bayesian networks, particle swarm optimization, and parallel computing. Moreover, to predict the risk of failures of a power grid, Rudin et al. (Reference Rudin, Waltz, Anderson, Boulanger, Salleb-Aouissi and Chow2012) introduced a system framework that employs supervised learning, ranking, and mean time between failures modeling.

From the examples described in this section, there are various information needs that are dependent upon an individual's role; hence, the creation of a model for learning and predicting the characteristics of collaborative and complex engineering projects may need to apply the combinations of techniques. However, presently there is still limited research describing how to select appropriate techniques to create the model and assist in the analysis of heterogeneous engineering data (Wagstaff, Reference Wagstaff2012). Hence, the adaptive capability and reusability of such models still need to be improved.

3.1. Feature modeling

During the characteristic learning process, a project is represented using a multilayer structure (see Fig. 2). From top to bottom, the layers contain project layer, characteristic layer, feature layer, and data layer. For any information elements contained in the adjacent layers, a one-to-many relationship is used to connect them. Based on this structure, a project can be represented by a set of hierarchical characteristics: each characteristic is represented by a set of features, and each feature is derived from the content of multiple data items.

Fig. 2. A multilayer project representation.

In the data layer, the data items are gathered from different project teams. To enable the application of data mining and machine learning techniques, all these data items should be stored in digital formats.

In the feature layer, all the project features are extracted by analyzing the content of data items. Project feature is the atomic information element in this model. Each project feature contains a meaningful description, and such description will be used to distinguish one feature from the others, or link the feature to the others. The fact that the feature does not have format restrictions means it can be represented by a term, phrase, image, or file.

In the characteristic layer, a project characteristic comprises a set of features with interrelationships. These relationships are identified by analyzing the feature attributes and contexts. In this work, project characteristics are defined to have three forms: pattern related, process related, and knowledge related. The features contained in a pattern-related characteristic are independent of each other. For example, the participation of team members is a pattern-related characteristic; the list of equipment names is also a pattern-related characteristic. Such features are identified by counting the occurrence of people/equipment names contained in certain data items. The features of a process-related characteristic are time dependent. For example, the fatigue test activity 1 and the fatigue test activity 2 comprise a process-related characteristic. Such features are the adjacent activities being performed during the project. The features contained in a knowledge-related characteristic are time dependent, and they also have semantic associations with each other. For example, the assembly activity 1, the assembly activity 2, the constrain of assembly activity 1, and the constrain of assembly activity 2 comprise a knowledge-related characteristic. Such features include the activities, the associations of the activities, and the constrains of the activity implementation.

Feature modeling is an automatic approach for extracting features from the project data. It contains three subtasks:

• • Feature identification: It identifies potential features from the data content, and then organizes them as a feature set.

• • Feature selection: It selects certain features from the feature set, and eliminates the ones with low significance.

• • Feature matching: It identifies the relations among the selected features, and then organizes them as feature tuples.

The feature identification task applies named entity recognition (NER) and natural language processing (NLP). NER is used to automatically identify the named entities from the project data. Named entities indicate the terms or phrases related to date, location, organization, name of people, and name of artifact. NLP contains multiple subtasks, that is, tokenization, part of speech tagging, stop words removal, and stemming. As a fundamental task of content analysis, NLP is used to covert textual information into meaningful tokens or phrases.Footnote ¹

To understand the feature identification process, a sample data item is shown in Figure 3. It is a part of the report being generated by an aircraft service project. After implementing the feature identification, certain named entities and tokens are identified from the data content (see Table 1). Three types of operations, that is, regular expression, NER, and NLP, are applied. For example, the time-related information elements with international standard date format YYYY-MM-DD can be identified by using the following regular expression:

Fig. 3. A data item generated by an aircraft service project.

The artifact name fuselage and the people name Bob Lewis can be identified by using NER. The identification methods regarding different types of features are shown in Table 1.

After this process, the identified elements are treated as the potential features of the project, which will be processed by the feature selection task in the next step. In this model, only some of the identified named entities or tokens will be recognized as project features. Feature selection is a variable selection method that is applied to select the essential variables from a given data set based on predefined rules or modeled knowledge (Gheyas & Smith, Reference Gheyas and Smith2010; Shi & Setchi, Reference Shi and Setchi2013). It could filter out the features with low significance, so that the model could avoid the overfitting problem and deal with lower dimensional data.

In practice, the implementation of service projects often requires domain-specific knowledge. To identify the significance of features, the feature selection task should also apply the same knowledge. Such knowledge also helps the feature matching task detect the relations among the features. After this process, the selected features will be organized as feature tuples. For each project, a set of feature tuples will be then generated.

3.2. Feature-based characteristic representations

In this work, feature tuples from different projects are required to be interpreted using normalized data representations, for example, vector-based representation and sequence-based representation. Let p denote a project, f denote a project feature, τ denote a feature tuple, and K denote a knowledge base. To represent pattern-related feature tuples, the bag-of-words model and vector space model are applied,

where w is a function to normalize the feature frequency.

To represent process-related feature tuples, the sequence-based data representation is applied,

where t_n is the timestamp of the feature, and t_n > t_n−1 ; g is a function to map the feature $ f_{n\comma t_n} $ to a specific activity a_n .

To represent knowledge-related feature tuples, the knowledge-based feature vector is used,

where φ is a function to map the feature $f_{n\comma t_n} $ to a set of knowledge concepts K _n , where K _n ⊆ K.

A project characteristic could include multiple feature tuples. These tuples should explicitly describe the meaning of this characteristic. Let c_N denote a project characteristic with a specific meaning, for example, task complexity level; it could be represented by a collection of feature tuples,

where ℓ _N is the label that indicates the meaning of c_N .

Based on the multilayer structure mentioned previously, a project can be represented by a combination of characteristics; that is, p = {C _pattern, C _process, C _knowledge}, where C _pattern, C _process, and C _knowledge are the sets of different characteristics.

3.3. Characteristic learning process

To understand the meaning of feature tuples, the characteristic learning approach applies two strategies: labeling the unlabeled feature tuples based on prior knowledge, for example, based on a set of prelabeled feature-tuples; and labeling the feature tuples based their similarities (it is for the situation that the prior knowledge is insufficient).

With the first learning strategy, let c _N denote a project characteristic with label ℓ _N ; c _N contains a set of labeled tuples, that is, c _N = {(τ₁, ℓ _N ) , . . . , (τ _k , ℓ _N )}. For any unlabeled feature tuple (τ _i , ℓ₀), the label ℓ _N will be assigned to τ _i , if τ _i could satisfy τ _i ∈ c _N , or τ _i and τ _j satisfy sim(τ _i , τ _j ) > α, where τ _j ∈ c _N , α is a defined threshold. In other words, if an unlabeled feature tuple is included by an identified project characteristic, then the label of the characteristic will be assigned to this tuple. If the feature tuple is not included by any identified project characteristic, but it is similar to some tuples of that characteristic, then the label of the characteristic will be assigned to this tuple.

The second learning strategy is a semisupervised approach that means it could work with insufficient prior knowledge. Given a set of unlabeled feature tuples X = {(τ₁, ℓ₀), . . . , (τ _n , ℓ₀)}, using clustering approaches, these tuples will be categorized into j clusters, that is, X = {C ₁, . . . , C _j }, where C _j = {(τ _m , ℓ₀), . . . , (τ _n , ℓ₀)}. The feature tuples contained by the same cluster often have high similarity with each other in nature; thus, all these tuples should represent an identical project characteristic. The learning process then only needs to assign a label to the cluster, that is, ℓ _N → C _j , instead of each feature tuple.

To implement the learning strategies, similarity measure approaches are required. For the vector-based representation, the similarity measure is based on,

For the sequence-based representation, an edit-distance based approach is applied (Shi, Gopsill, Newnes, et al., Reference Shi, Gopsill, Snider, Jones, Newnes and Culley2014),

where ${\rm \vartheta} $ is a cost function; d(ε, ε) = 0; and d(τ _i , τ _j ) = 0, if τ _i and τ _j are identical.

The sequence similarity is calculated by using,

where |τ _i | and |τ _j | indicate the length of τ _i and τ _j ; α and β are defined thresholds, where 0 ≤ α ≤ β × min (|τ _i |, |τ _j |), β ∈ [0, 1]. For example, when β is 0.5, the similarity of τ _i and τ _j will be considered as 0, if the edit distance between them is greater than the half-length of the shorter one.

3.4. Characteristic prediction process

In practice, the data generation of a project is a gradual process. It implies that the prediction of characteristics for an ongoing project at its early stage needs to utilize incomplete data.

Under this circumstance, the training data applied by the characteristic prediction process should reflect the dynamic relations among the time, data, and evolution of project characteristics. Assume a project p is contained by the training data, which has a set of characteristics, that is, p = {c ₁, … , c _N }, and each of the characteristics contains a set of feature tuples, that is, c _N = {(τ₁, ℓ _N ), … , (τ _k , ℓ _N )}. To perform the prediction on a real-time basis, the representation of characteristic needs to be segmented based on predefined time intervals. Consequently, a single characteristic will have multiple representations, each of which links to a specific project stage. For a characteristic c _N , its segmented representations are represented as,

where Δt is the length of time interval.

For a project, it is represented by using the time-segmented characteristics,

where k is the number of time intervals.

As an example, Figure 4a shows the process of a service project. This project has three different stages: planning stage, problem-solving stage and evaluation stage. The data items regarding each stage have various file format and storage locations. In this case, each stage takes approximately 30% of the project progress. The process segmentation therefore is implemented based on the 30% interval; that is, the project process will be segmented into three subprocesses that map to the stages accordingly. As shown in Figure 4b, each subprocess is used to represent the project characteristics.

Fig. 4. Process segmentation-based defined time interval.

In practice, the time interval could be defined as relative value, that is, percentage, or absolute value, for example, a single day or week. It can also be defined as arbitrary value; for example, if the project stage I covers the first 10 days, the project stage II covers the following 15 days, and the project stage III covers the following 30 days, then the intervals can be defined as {t ₁ : 10, t ₂ : 25, t ₃ : 55, …}.

In the prediction process, defining the time interval considers the details of the input data. Let (τ _k , ℓ₀) denote an unlabeled feature tuple with a null characteristic label. If the timestamps of its contained features are in a time range (t ₁, t _n ), then this time range will be applied to define the time interval Δt, for the purpose of segmenting the training set. In the next step, the prediction process will assign a label to τ _k , according to the time-segmented training set. The detailed process is summarized as the following steps:

• Step 1. For a given feature tuple τ _k , identifying its covered time range (t ₁, t _n ) based on its contained features.

• Step 2. Determining a time interval Δt, where Δt ≤ (t ₁, t _n ).

• Step 3. For a given training set, converting the characteristic representations into time-segmented representations by using Δt.

• Step 4. Initializing the prediction ℓ₀ → τ _k based on the time range t ₁ + Δt.

• Step 5. Finding the time-segmented characteristic representations with the time range t ₁ + Δt from the time-segmented training set, that is, p _{t 1 + Δt} = {c ₁, … , c _N } _{t 1 + Δt} .

• Step 6. Measuring the similarity between τ _k and each of the element contained by p _{t 1 + Δt} .

• Step 7. Determining ℓ₀ based on the similarity measure results, that is, ℓ _j → ℓ₀, if $\arg \max (\hbox{sim}({\rm\tau} _k \comma \, c_1 )\comma \ .\ .\ .\ \comma \ \hbox{sim}({\rm\tau} _k \comma \, c_N )) = \hbox{sim}({\rm\tau} _k \comma \, c_j )$ , where j ∈ [1, N].

According to the last step, ℓ _j is assigned as the label of τ _k ; thus, the feature tuple is predicted to be correlated to the project characteristic c _j .

4.1. Learning pattern-related characteristics

In this task, the project activity level is recognized as a pattern-related characteristic. It indicates the frequency of key activities being performed during the project execution. The calculation considers the volume of activity in the workflow. In general, the activity level could indicate the level of project input or output.

In general, the feature selection process follows the five Ws (Who, What, Where, Why, and When) and one H (How) principle. For most service cases, these features are the key elements to form the characteristics. Therefore, the feature selection process is a generalized approach that suits other types of service cases with different contexts. In this scenario, the involvement of core project members and the name of key activities are considered. Such features are extracted directly from the data content using NER.

During the learning process, the feature modeling approach analyzes the project data, and then generates a feature set regarding each project stage. The feature set includes the project member names, project activities, and timestamps. Let s _k denote a project stage that covers a range of time; where s _k = {t ₁, … , t _k }, the feature set regarding s _k is

and the vector-based representation should be

The activity level regarding s _k is calculated using

where N is the total number of the features being identified. For example, if the task planning activities have been identified 6 times in the initial project stage, and the other activities have been identified 20 times at the same project stage, then the activity level regarding task planning at the initial project stage should equal to 6/(20 + 6) = 0.23.

Figure 5 shows the activity levels of two different service projects. In general, the life cycle of both projects contains four stages: the information request stage (the initial stage), requirement finalisation stage (the early to middle stage), service design stage (the middle to late stage), and evaluation stage (the late stage).

Fig. 5. Activity levels of two service projects.

Because the projects have the same type of service requirements, that is, repairing the wing surface corrosion, similar amounts of activities regarding task planning are expected at the information request stage. According to the figure, both projects have similar activity levels at the initial stage.

Next, the cumulative activity level of Project A has a more rapid increasing trend than Project B. The reason is that Project A needs to deal with multiple corrosion locations, but Project B only deals with one. At the requirement finalization stage, Project A requires a higher level of activity than Project B, as it needs to be received and send more information. Meanwhile, at the service design stage, Project A requires a higher level of activity than Project B, as it needs to issue a higher volume of repair design solutions. At the evaluation stage, Project A again requires a higher level of activity than Project B, as it needs to perform more evaluation tasks. According to the figure, Project A has the higher activity levels at these stages than Project B.

For the reason that Project A requires more complicated engineering process and involves a higher volume of collaborations, the activity level regarding Project A could be affected more easily. According to the visualization, the change of activity level regarding Project A is clearly more frequent than Project B. This indicates the certainty regarding the execution of Project A is less than Project B, which corresponds to the facts.

Meanwhile, Project A has heavier workload than Project B; thus, it has a higher overall activity level. According to the visualization, the growth rate of activity level of Project A is clearly higher than Project B. This indicates Project A needs to complete more work than Project B within the same period of time, which also corresponds to the facts.

4.2. Learning process-related characteristics

In this task, the activity sequence is defined as a process-related characteristic. Given a project, the activity sequence is the representation of its workflow or design process. The understanding of the activity sequences regarding a type of projects could help the project members to implement the workflow standardization, design process optimization, and process knowledge reuse.

In the feature modeling process, NLP, content analysis, and modeled knowledge are used to analyze the data items on both the content and metadata levels. The activities are identified from the data content, and their activities are identified from the metadata. These activities are then organized as chronological order according to the timestamps.

Figure 6 shows the activity sequences being generated from the data set. Each row indicates the activity sequence of a project, and each Tx indicates an activity-related feature. For each activity sequence, its features are organized in a chronological order.

Fig. 6. Activity sequences of multiple service projects.

In order to compare the characteristics of different projects quantitatively, the similarity between a pair of sequences is measured by using Eqs. (6) and (7). The applied strategy is as follows: for two projects having similar characteristics, the sequence similarity between them would be high; for two projects having dissimilar characteristics, the sequence similarity between them would be low.

In this scenario, the fact is that Project P5 and P6 have the same type of service requirement, that is, corrosion damages, but Project P3 has a different one, that is, lightening damages. Therefore, the processes regarding P5 and P6 are supposed to be similar, and the processes regarding P6 and P3 should be different.

According to the learning approach, the edit distance between two sequences is equal to the number of operations to convert one to the other (see the detailed explanation in Section 3.3). For example, the number of operations (i.e., edit distance) to convert P5 to P6 (or P6 to P5) is equal to 11 (based on Eq. [6]). The edit distance is converted to a normalized similarity value with the range between 0 to 1. Based on Eq. (7), the similarity value regarding P5 and P6 is 0.656. The similarity value regarding P6 and P3 is 0.333. The results imply that P5 and P6 have similar characteristics, but P6 and P3 have different ones. In other words, the sequences of P5 and P6 have higher volume of similar patterns, but sequences of P6 and P3 have lower volume of similar patterns, which correspond to the visualizations shown in Figure 6.

4.3. Learning and predicting knowledge-related characteristics

In this task, operational complexity is defined as a knowledge-related characteristic. It covers time spending, resource consumption, and technical difficulty. Operational complexity of a service project is typically dynamic at different stages. The measure of it needs to consider the proposition of certain activities at the specific project stage. For example, the activity regarding “issuing repair instruction” has higher operational complexity than “issuing technical deposition” and “issuing answer.” If a project is at stage X and has higher proportion of “repair instruction” than another project, its operational complexity should be at a higher level than the other one. In practice, the understanding of operational complexity in an early stage could help the project members to improve the rationality of the decision making and the effectiveness of managing time and resource.

In the feature modeling process, NLP, NER, content analysis, frequency analysis, and knowledge bases are employed. The features identified from the project data include activity types, timestamps, part information, and technical terminologies. Each feature has an attribute to indicate the status of data transaction, which is identified based on the modeled knowledge. For example, initializing fatigue test is classified as an activity with internal data transaction; preparing repair instruction is classified as an activity with outgoing transaction; receiving operator damage report is classified as an activity with incoming transaction. The feature vector with such attributes is used as the data representation for the project (see Table 2).

In this scenario, the model employs the semisupervised learning. Clustering approach is used to create a training set from the historical project data. It categorizes the projects into multiple clusters. During the clustering process, Silhouette score is used to determine the number of clusters in an automated manner (Dudoit & Fridlyand, Reference Dudoit and Fridlyand2002). For the given data set, the clustering result is shown in Figure 7. The optimized cluster number is equal to six, and the projects within the same cluster have the identical color.

Fig. 7. The clustering result regarding operational complexity.

To understand the meaning of each cluster, the following rules contained in the knowledge bases are considered:

• • The projects with high outgoing, low incoming data transactions have the low level of complexity.

• • The projects with low outgoing, high incoming data transactions have the high level of complexity.

• • The projects with high internal data transactions have the high level of complexity.

• • The projects with balanced outgoing and incoming data transactions, and low internal data transactions, have the medium level of complexity.

Based on these rules, the level of complexity regarding each cluster is determined (see Table 3).

The projects with labels are treated as the training data, which will be used to predict the operational complexity of unlabeled projects. To evaluate the performance of the characteristic prediction process, 10-fold cross-validation is used. The set of labeled projects is divided into a training set (90% of the data set) and a test set (10% of the data set). During the evaluation process, the prediction algorithms, including support vector machine, artificial neural network, and random forest, are tested with the model respectively. In this evaluation, F score is used as the prediction performance measure.

In the test set, each project has a label ℓ _L being assigned by the learning process previously. Such a label is considered as the ground truth, and it is invisible to the predictors. In the training set, the label of each project is visible to the predictors. The predictors need to determine what label should be assigned to the unlabeled projects in the test set based on labeled projects in the training set. The project representations in both test set and training set are segmented by using normalized time intervals, which are 0%–30%, 0%–50%, 0%–70%, and 0%–90%. According to the training set, the prediction process will assign a label to each test data, for all the stages, for example, $\ell _1 \to $ 0%–30%, $\ell _2 \to $ 0%–50%, $\ell _3 \to $ 0%–70%, and $\ell _4 \to $ 90%. The evaluation process is then to compare whether the assigned labels {ℓ₁, ℓ₂, ℓ₃, ℓ₄} are identical to the ground truth ℓ _L . The F score of this prediction processes is shown in Table 4.

As shown in the table, the artificial neural network based model has the best performance with 0%–30% and 0%–90% time intervals, and the random forest based model has the best performance with 0%–50% and 0%–70% time intervals. In general, the prediction performance is proportional to the completion level of projects. It implies that the more data is considered by the model, the better prediction performance it will be. By using the model, the operational complexity of a service project could be predicted with a 70+% accuracy by using 30% of the project data.