1. Introduction
Nowadays, shipping technology is evolving, equipping ships with faster speed, larger size as well as more capabilities to handle dangerous goods. As a result, research on the navigation environment and ship traffic accidents has attracted increasingly more attention. When conducting accident analysis, stakeholders usually need to simulate the process of traffic accidents, analyse the cause of the accident and assess responsibility. In this process, the assessment of risk is one of the important links. Moreover, with regards to crew training, risk cognition is also an important training subject. Visual information is of great significance for human cognition, since approximately 70% of the information obtained by humans comes from vision. Based on the rapid development of computer graphics, three-dimensional (3D) simulation is used to reproduce the actual ship accident process to clarify the motion state of each participant in the accident. This method of accident responsibility judgment is becoming increasingly more recognised. Currently, ship traffic accident reproduction dynamically simulates the trajectory and motion of the accident ship based on the accident data records. Through the multi-angle observation of the entire accident process, crews and experts can assess the cause of the accident, investigate the mechanism, improve the risk cognition and provide the scientific basis for determining the responsibility. In most applications, ship traffic accident reproduction mainly includes trajectory simulation, ship motion simulation and collision simulation. Trajectory simulation is mainly based on the Automatic Identification System (AIS) data and Voyage Data Recorder (VDR) data. It usually provides the visual scene of the accident area, simulates the environment effect and provides multi-angle observation of the accident process. Based on the trajectory simulation, crews and experts can determine the ship's navigation status before the accident, analyse the ship's behaviour and risk elements, and provide guidance for avoiding accidents. In the entire process, risk analysis has been an important tool. The risk assessment model and accident scene map greatly influence the crew's cognition of accident risk. The risk assessment model assists the crew to recognise the risk from the perspective of mathematical calculation, while the accident scene map affects the crew's cognition from the perspective of scene perception.
As one of the major public hazards in the world, how to improve the risk analysis of a traffic accident has become an important research topic (Liu et al., Reference Liu, Wang, Zhang, Wang and Gong2007). Especially how to improve the crew's risk cognition is of great importance. The coupling effect between the ship and external environment is prominent during the navigation process. Restrictions and dependencies between various factors in the accident process are very important for the accident reproduction. Therefore, the analysis of ship traffic accidents needs to fully consider the influence of a ship's own factors and external environmental factors. The risk cognition model should strengthen the description of the surrounding environment, navigation marks and waterways of the accident site, and combine these factors to improve risk assessment. At present, the degree of collision risk can be used as the crew's reflection on the objective existence of collision risk. When determining the degree of collision risk, we should choose the factors that characterise the ship's collision risk, analyse the objective factors and establish a mathematical model of the collision risk. Human factors, such as lookout negligence, manoeuvring error and fatigue, are crucial causes of ship traffic accidents. When the ship passes through the bridge, if the crew is unfamiliar with the bridge area or encounters bad weather, accidents often occur. In our opinion, on the one hand, due to the unintuitive visualisation of the two-dimensional (2D) electronic navigational chart, crews have to understand the traffic state and surrounding environment by interpreting the chart (users have to generate a mental model of a map, rotate it and match it with the real world, and relate the symbols and map features to real features); on the other hand, crews are not sufficiently aware of the risks. However, a 3D model can significantly improve user's awareness of the environment and cognition of the object described by the 3D model.
At present, the construction of an accident scene map can not fully meet the needs of all-round perception of the navigation environment. Usually, one way is to simulate the accident process by constructing the accident scene based on a 2D electronic chart. However, this method loses a lot of useful scene data. Additionally, it is difficult to realise accurate and effective 3D scene perception and spatial analysis, especially in situations where different types of ship meet. Another method is real environment simulation. However, this procedure does not integrate some useful data for spatial analysis, such as a 2D electronic chart. Therefore, it is necessary to realise geometric correctness and geographic reality of the integration of multi-modal and multi-scale navigation environment data in a unified framework, and make full use of complementary multi-modal data to construct an all-round perception of the navigation environment. Compared with our previous work (Liu et al., Reference Liu, Zhao and Pan2014), data fusion is applied to the risk assessment, and 2D spatial analysis and 3D spatial analysis are integrated in this paper. In our risk calculation model, we fuse 2D spatial analysis and 3D spatial analysis together, such as yawing degree analysis and 3D collision detection calculation. We introduce a technical update of fusion visualisation, propose an efficient bounding box calculation method, design a new method for the crew to improve risk cognition of a ship collision accident and provide an intuitive visualisation for the accident simulation. Furthermore, based on the 3D spatial data representation, the utilisation rate of the screen space is greatly improved and the information that can be conveyed in a single screen is richer. We develop a trajectory risk cognition system, which can provide new cognition and analysis methods for the investigation of ship traffic accidents. This will enhance the risk cognition capabilities of crews and experts. The contributions of the proposed method are summarised as follows:
• propose a data-fusion-based method for the accurate spatial calculation in the trajectory risk reconstruction of the accident ship;
• design a trajectory risk cognition method that fuses multiple factors.
2. Related work
Based on the simulation technology and accident record data, ship traffic accident analysis first simulates the motion state of the ship involved and the accident site, and then analyses the cause of the accident. Through these analyses, the dangerous situation of the accident can be recognised. This will provide guidance for future actions. There are two tasks that are important in the cognition analysis: one is risk assessment and the other is scene map construction. However, the most current research on accident analysis focuses on risk assessment and there is not enough attention toward scene map construction in accident analysis.
In the aspect of the risk assessment research, researchers have done a lot of work. From the analysis of existing researches and the summary of the review paper (Yan et al., Reference Yan, Wu, Wang and Wei2015; Du et al., Reference Du, Goerlandt and Kujala2020; Huang et al., Reference Huang, Chen, Negenborn and van Gelder2020), risk assessment research mainly focuses on three aspects: statistical analysis methods, physical abstract models and risk assessment models. The analysis of historical accident data is of great significance for discovering the cause of the accident (Mao et al., Reference Mao, Yan, Chen, Chu and Yuan2010). After careful analysis of the accident statistics in the Gulf of Finland during the past 10 years, Kujala et al. (Reference Kujala, Hänninen, Arola and Ylitalo2009) concluded that ‘groundings and collisions are the dominant accident types in the Gulf of Finland’. Zhang et al. (Reference Zhang, Yan, Yang, Wall and Wang2013) analysed the accident database of Yangtze River based on the formal safety assessment theory. They analysed the accidents according to the designed statistical indicators and gave the proportions of different accident categories. Zaman et al. (Reference Zaman, Kobayashi, Wakabayashi and Maimun2015) used probability modelling and consequence modelling to conduct the risk analysis based on AIS data. They established the collision probability according to traffic density, head-on, crossing and overtaking conditions. Based on the analysis of ten years of ship accident data, Weng and Yang (Reference Weng and Yang2015) introduced two regression models to predict the accident probability. Using the worldwide shipping accident records from 2001 to 2011, they analysed the likelihood of fatal accidents and the number of mortalities in shipping accidents. Uǧurlu et al. (Reference Uǧurlu, Köse, Yıldırım and Yüksekyıldız2015) used fault tree analysis to investigate causal factors of the collision and grounding accidents of oil tankers based on the corresponding data stored in Global Integrated Shipping Information System. They also studied the significance degree of the initial events causing occurrence of accidents. Bye and Aalberg (Reference Bye and Aalberg2018) combined correspondence analysis and F-tests in a logistic regression model to analyse the Norwegian Maritime Directorate (NMA) accident database. Their model can predict whether the accident is navigation-related or not. Chen et al. (Reference Chen, Bian, Wan, Yang, Zheng and Wang2019) used an improved entropy weight TOPSIS (Technique for Order Preference by Similarity to Ideal Solution) model to analyse total-loss marine accident based on the total-loss marine accident data they collect. They concluded that ‘the main influential factors are foundering, stranding and fires/explosions’. The statistical method is one way to explore the factors influencing the accident. However, such a method is mainly for managers and lacks a process description of the accident from a micro-perspective and a detail analysis of the cause of the accident. Thus, it does not provide useful guidance for the crew's detail judgment. An abstract physical method has been intensively investigated by many researchers, with the aim of better probing the navigation risk. The ship domain method is commonly used (Wang et al., Reference Wang, Zhang and Hu2021). Smierzchalski and Michalewicz (Reference Smierzchalski and Michalewicz2000) treated the ship collision problem as a dynamic optimisation task. They introduced a new version of the Evolutionary Planner/Navigator to generate a safe trajectory for the ship. Safe distance and width of the dangerous area are chosen to assess the risk. Goerlandt and Kujala (Reference Goerlandt and Kujala2011) used a spatio-temporal overlap check to detect collision candidates based on a micro-simulation model and used a causation probability model to describe the ship collision risk. Their approach can provide detailed information about the circumstances in which ships encounter each other. Qu et al. (Reference Qu, Meng and Suyi2011) used three indices to evaluate the ship collision risks: time-mean speed-based speed dispersion (a macroscopic risk index), degree of acceleration and deceleration (a microscopic risk index), and number of fuzzy ship domain overlaps. Base on Lloyd's MIU AIS ship movement database and the three indices, they found the most risky legs in the Singapore Strait. Montewka et al. (Reference Montewka, Goerlandt and Kujala2012) introduced a new ship collision criterion called minimum distance to collision (MDTC) to assess the ship collision risk. They used an iterative algorithm to calculate the criterion. Kim and Kim (Reference Kim and Kim2018) introduced an improved navigation risk assessment model. The model fuses the ship's dynamic domain with a collision risk assessment formula, where the calculation of collision risk assessment formula is based on the length, speed and manoeuvring capability of a ship. Although the display of calculation results of the ship domain model is intuitive, the calculation results can not be directly mapped to actual decision-making factors (can not be directly fed back to the cognitive process of the crew). In areas with a density of ships, the ship domain method is likely to cause cognitive confusion. Moreover, it is difficult for the ship domain model to measure all dangerous situations. In recent years, researchers have begun to use the fuzzy mathematics method, back propagation (BP) neural network and grey correlation analysis to study the calculation of ship navigation risk. The fuzzy mathematics method adopts the fuzzy theory to determine the index based on the factors that affect the collision risk of ships, such as distance to the closest point of approach (DCPA), time to the closest point of approach (TCPA), relative position between ships, speed ratio of the ships, distance between ships. The fuzzy mathematics method is a widely used method for decision-making in transportation systems (Wu et al., Reference Wu, Yip, Xie and Wang2018, Reference Wu, Yip, Yan and Soares2019, Reference Wu, Cheng, Yip and Wang2020). It is easier to logically analyse the influence of various factors by using the fuzzy comprehensive evaluation method. This is closer to human cognitive law. Based on input parameters such as DCPA and TCPA, the BP neural network directly outputs the ship collision risk. It has the advantages of good self-learning ability, small calculation error and reliable result. However, this method converges slowly and it is very computationally expensive. Moreover, this method is highly dependent on samples. A set of training samples is only applicable to the corresponding sea area. Therefore, the BP neural network method is not a scalable method (Peng et al., Reference Peng, Wang, Lan, Sun and Yan2012). The grey correlation analysis method is a relatively new method. Although it does not calculate the specific value of the collision risk of a ship, it can calculate the relative risk between multiple target ships and the own ship. This is the same as the risk index in the ship collision avoidance system. It has the advantages of small amount of calculation, fast speed and accurate result. However, this method is only suitable for a multi-ship encounter. This method can quickly and accurately determine the collision avoidance sequence of multiple target ships (Li and Wang, Reference Li and Wang2011). Combining data fusion methods to study the risk of ship collision has also been a research hotspot in recent years. Based on the data processing of AIS and radar, and the analysis of ship collision avoidance situation, Wu (Reference Wu2013) established a ship collision risk evaluation model. Aiming at the main accident risks in the ship transportation system, such as collision, grounding, anchoring and drifting, Zhou et al. (Reference Zhou, Ge, Huang, Liu and Chen2013) proposed a quantitative calculation scheme for the collision risk of the marine ship transportation network system based on the system simplification idea and information fusion technology. Most of the existing researches only rely on a single or a few data sources, while there are many data sources that have to be comprehensively considered, such as AIS, radar and video. It is necessary to fuse the collected multi-source data to obtain a more comprehensive risk evaluation model.
Porathe (Reference Porathe2006) proved that compared with traditional 2D charts, 3D charts could dramatically reduce the number of human errors and improve the accuracy and efficiency of manoeuvring. In the aspect of scene map construction research, scholars have made many attempts and discussions, and the direction is gradually changing from 2D to 3D. The accident analysis relies on the construction of the accident scene map. How to extract the accident record information to obtain the information needed in the accident analysis calculation and fuse them in a unified framework is of great importance to the construction of an accident scene map. Multi-model data fusion (especially multi-model data fusion based on deep learning architecture) has become an active research area. Integrating multi-model data to analyse ship traffic accidents has become an important method. However, under-researched aspects of this work are the construction of the all-element scene and the lack of analysis research based on the all-element scene. The all-element scene will provide more comprehensive information to assist accident analysis. In recent years, with the development of theories and technologies of computer graphics and the geographic information system (GIS), 3D geospatial information applications have been rapidly developed. Information collection has become more convenient. At present, the use of 3D graphics simulation technology to construct an accident scene has become a trend, including constructing more accurate 3D models based on actual ships, waterways and other elements, and refining and perfecting the functions of user interaction modules. Under current trends, the key technologies for accident scene map construction include 3D environment modelling technology, real-time rendering technology, GIS technology, etc. Fan et al. (Reference Fan, Huang, Jiang and Xu2018) proposed a method of integrating 3D simulation with a real ship driving system and studied the ‘virtual–real’ and ‘dynamic–static’ ship navigation technologies and equipment, which effectively integrated hydrological factors, terrain features, waterways, traffic dynamics and ship driving information. Their research provides a multi-angle and multi-level visual display for the poorly visible water traffic environment to enhance the crew's ability to perceive the surroundings. Tang (Reference Tang2009) deeply analysed the key issues of the fusion of sea and land geographic vector data, including the unification of projection methods, the unification of coordinate systems and the fusion of object representations, studied the matching algorithm of similar objects and discussed the uncertainty problem of fusion results. Chybicki et al. (Reference Chybicki, Kamiński, Lubniewski, Bruniecki, Kulawiak, Da̧browski and Moszyński2009) fused AIS data, real-time radar data and remote sensing image data with electronic chart data, and analysed the system architecture and technical solution for developing a multi-source data fusion system. Kulawiak et al. (Reference Kulawiak, Chybicki and Moszynski2010) studied the fusion of hydrological data and remote sensing image data with electronic chart data, and constructed a web system for data visualisation. Fischer and Bauer (Reference Fischer and Bauer2010) proposed an object-oriented method for fusion of target observations produced by multiple heterogeneous sensors (mainly including aerial imagery, radar and AIS data), and used simulation tools to evaluate different scenarios and sensor configurations. The experimental results fully prove the importance of data fusion. Lager and Topp (Reference Lager and Topp2019) designed a remote monitoring system of an unmanned autonomous ship under limited bandwidth communication conditions. The system uses 3D modelling and virtual reality technology to provide users with surrounding environment situational awareness and collision detection functions. This greatly reduces the user's cognition load. However, multi-modal perception data fusion is not mature enough in ship traffic at present. There are few studies on the 3D multi-modal data-fusion-based trajectory risk cognition.
Liu et al. (Reference Liu, Zhao and Pan2014) proposed a 3D ship navigation and monitoring system based on multi-model spatial data fusion. The proposed system fuses remote sensing image, electronic waterway map, terrain data, 3D model and AIS data. Based on this spatial data organisation model, this paper deepens the research and applies the model to trajectory risk reconstruction of ship collision accident. First, the algorithm framework is presented. Second, the scene reconstruction method is introduced, including static information reconstruction and dynamic data reconstruction. Third, the trajectory risk reconstruction algorithm is described, including ship boundary calculation and risk calculation. We fuse 2D spatial analysis with 3D spatial analysis to provide a multi-dimensional risk assessment. Then, visualisation methods of the risk are illustrated. Finally, a ship collision accident experiment is carried out to verify the effectiveness of our approach.
3. Algorithm overview
In this paper, multi-factor analysis, multi-model data fusion and calculation, historical data mining and data visualisation method are used to explore how to improve the crew's risk cognition. Trajectory risk cognition of ship collision accident is an important research in the field of ship traffic safety. The accurate accident record information of ship collision accident is the basis of the risk reconstruction, especially ship navigation data collected in real time. The trajectory risk reconstruction scheme of the ship collision accident based on multi-model spatial data fusion and historical data mining designed in this paper is shown in Figure 1.
The trajectory risk reconstruction proposed in this paper needs to collect and process more data in the early stage, including not only the navigation data of the ship, but also the basic geographic data and waterway data, such as terrain data, remote sensing image data, 3D entity model and electronic navigational chart. First, the collected basic geographic data and waterway data are processed, and the ship traffic accident scene that fuses multi-model information is reconstructed by the method described in Section 4. Second, the ship traffic accident database is constructed. The collected ship navigation data such as VDR, AIS are extracted, filtered and stored in the database. The ship traffic accident database is used to manage accident-related data, including meteorological data, hydrological data, basic information and track state data of the accident ship, information backup and encryption, etc. It can be constructed based on a relational database or a graph database. Third, on the basis of data collection and processing, a 3D model of the accident ship is built and the spatial calculation of trajectory risk reconstruction of the accident ship is realised based on the data fusion method. Finally, visualisation methods are designed to give an intuitive result. The trajectory risk reconstruction can assist the crews and experts to recognise the risk and conduct the accident analysis. Parameter values of the method can be modified according to expert experiences. Subsequently, the results can be stored in the ship traffic accident database for later review and reference.
4. Scene reconstruction
This paper constructs a ship traffic accident scene which fuses multi-model spatial data based on 3D computer graphics technology and spatial data processing technology. The specific construction process is shown in Figure 2.
From the construction process in Figure 2, it can be seen that data involved in the scene reconstruction mainly includes terrain data, remote sensing image data, 3D entity model and electronic navigational chart. The mesh model based on multi-resolution terrain data is the basis of the simulation scene. It fuses the sounding data extracted from the electronic navigational chart and is the bottom layer of the scene (Liu et al., Reference Liu, Zhao and Pan2014). Then, the pyramid model is used to organise the remote sensing image data. Based on texture mapping and viewpoint corresponding methods, the tiled and layered remote sensing image data are mapped onto the bottom layer of the scene to form the basic scene of the ship traffic accident. The 3D entity models used in the scene are built by 3D modelling software (such as 3DS MAX, Sketchup) and oblique photography (such as ContextCapture). These data are also input to the scene (Liu et al., Reference Liu, Zhao and Pan2016). The electronic navigational chart data are divided into three categories: entity object, artificial vector information and sounding data. The entity object and sounding data are processed as described above. The artificial vector information refers to the artificial navigation vector information such as depth contour, depth area, anchor zone. We use the vector data fusion method to fuse these data in the scene (Liu et al., Reference Liu, Zhao, Pan and Bai2015).
4.1 Static information
Based on the Automotive Intelligent Chart (AIC) 3D Electronic Chart Display and Information System (ECDIS) proposed by Liu et al. (Reference Liu, Zhao and Pan2014), we conduct further improvement research. With the development of programmable rendering pipeline and graphics processing unit (GPU) hardware, we adopt the advanced GPU rendering methods in our project, such as hybrid data representation, order independent transparency and geometry clipmap. We have greatly enhanced the previous project in this paper. In terms of fusion visualisation, we mainly make the following optimisations.
• For the category of basic scene data, which includes remote sensing images, terrain data, we use the level of detail (LOD) method to organise this type of data. Terrain and riverbed data are used to build the fundamental framework. Texture mapping is used to map remote sensing image tiles onto the fundamental framework. A height map tile that contains the terrain data is also stored in the image format. The GPU-based geometry clipmap method (Losasso and Hoppe, Reference Losasso and Hoppe2004) is used to render the basic scene data. The geometry clipmap structure caches a square window of $n\times n$ samples within each level. These windows correspond to a set of nested regular grids centred about the viewer. The structure maintains triangles that are uniformly sized in screen space. This method provides a number of advantages: simplicity of data structures, smooth visual transitions, steady rendering rate, graceful degradation, efficient compression and runtime detail synthesis. To satisfy the need for texture streaming of large rich environments, we use virtual texture (Mittring and GmbH, Reference Mittring and GmbH2008) to manage the texture memory. Only the actually required portions need to be uploaded. This is very efficient.
• For the category of terrain following objects, such as country boundaries, highways, etc., we use the texturing method to accelerate the rendering of large-scale vector data. First, vector data are drawn onto texture tiles in the fragment buffer. Then, these texture tiles are mapped onto the terrain by texture mapping. This will avoid the interpolation calculation of vector and terrain and can greatly improve the calculation efficiency.
• For the category of independent objects, such as labels, 3D entities, etc., we refer to the foveated rendering method (Swafford et al., Reference Swafford, Iglesias-Guitian, Koniaris, Moon, Cosker and Mitchell2016) for rendering this type of data. We use hybrid data representation (triangle mesh and displacement map) and data compression to describe entity objects. This will reduce the memory consumption. The basic idea of foveated rendering is to render high-quality images in the foveated area and low-quality images in the peripheral area to save time and cost. Rendering methods based on gaze point information usually use technologies such as multi-texture and fragment programming to improve rendering performance.
• For the electronic navigational vector objects, the rendering method has also been improved. The order independent transparency method (DFB algorithm) (Maule et al., Reference Maule, Comba, Torchelsen and Bastos2014) is used to render this data type. The transparency attribute of these vector objects such as depth contour, depth area, anchor zone are realised more efficiently. This method can perform correct compositing for objects with efficient memory management.
In terms of data modelling, we have also made improvements. We make full use of advanced data modelling technologies, such as oblique photography and point cloud modelling technology. We select different modelling methods according to the difficulty of data acquisition. For entities that are difficult to collect laser point cloud information on the ground, oblique photography technology is used for data acquisition, and then modelling processing is performed, such as denoising, singulation and trimming. For entities that are easy to collect laser point cloud information on the ground, selection of the data acquisition method is based on the data accuracy requirements: for entities that affect berthing and unberthing, such as docks, laser point cloud technology is used for data acquisition and post-processing; for other entities, oblique photography technology is adopted. Data obtained by oblique photography and point cloud modelling technology can be reconstructed with mature software such as ContextCapture, Trimble Realworks, and fused in our system based on our algorithms.
The reconstruction effects are shown in Figures 3–5. Figures 3 and 4 show the overwater view effect of the ship traffic accident scene and Figure 5 shows the underwater view effect of the ship traffic accident scene.
4.2 Dynamic data
For the dynamic data of ship traffic, such as ship positions and accident information, we fuse these data into the accident scene. The reconstruction of trajectories of the accident ship consists of two parts: trajectory point calculation and pose calculation. Trajectory point calculation is used to reconstruct the linear motion of the ship, and the pose calculation is used to maintain the correct posture of the ship.
(1) Trajectory point calculation
The trajectory data of the accident ship stored in the ship traffic accident database are extracted to perform the reconstruction. However, during the trajectory data collection process, there are some problems in the data record of the accident ship such as large collection interval and missing data. Data jumping and unsynchronisation may occur. To achieve the synchronisation of trajectories of accident ships and avoid the impact of missing data, trajectory detection and interpolation are needed. Based on the previous position of the trajectory data, ship heading and speed information stored in the database, a data-fusion-based recursive method is designed to fill in the missing data points of the encrypted data. The calculation method is shown in Equation (1),
where $P_i$ is the position of the output point; $P_{{\rm bef}(i)}$ is the position of the previous point; $t$ is the interpolation time interval; $v$ denotes the unit vector of the ship speed; $r$ denotes the unit vector of the channel centreline or designed route segment; $P_{{\rm start}}$ and $P_{{\rm end}}$ represent the starting point and ending point of the route segment, respectively; and $\alpha$ and $\beta$ are weight coefficients. As can be seen in Equation (1), this method fuses channel information or route information and turns the reconstruction problem into a constraint calculation problem. Therefore, space and time calibrations of the data are achieved and linear motion of the ship can be realised by dynamically refreshing the 3D spatial position of the ship model according to the calculated trajectory point.
(1) Pose calculation
Pose calculation decomposes the transformation of pose change about a certain direction axis into rotation transformations about the three coordinate axes of the local coordinate system of the 3D ship model, that is, this calculation simulates the ship's roll, pitch and yaw by constructing rotation matrices about the three axes. The rotation angles are from the traffic accident database. In our system, pitch corresponds to rotation about the $X$-axis, yaw corresponds to rotation about the $Y$-axis, and roll corresponds to rotation about the $Z$-axis. The rotation transformations can be calculated according to Equations (2), (3) and (4), respectively. The rotation angle is positive when rotating counterclockwise,
where $\theta _x$, $\theta _y$ and $\theta _z$ represent the rotation angles about the $X$, $Y$ and $Z$ axes, respectively.
5. Risk reconstruction
5.1 Ship boundary calculation
Collision detection is to determine whether there is a collision between two objects. It is an indispensable part of ship accident reconstruction. The basic framework of the collision detection algorithm is to perform preliminary detection first, and then gradually to refine intersection calculations. Object space-based collision detection algorithms often use tree-type retrieval structures and bounding boxes to speed up computation. The collision detection in this paper refers to the process for detecting the collision of 3D models of accident participants in the ship traffic accident scene. Usually, the actual outlines of different objects are different. A ship has a rectangle outline. A buoy has a cylindrical outline. For different outline properties of objects, different bounding boxes are designed to accelerate the calculation of collision detection in this paper, such as rectangular bounding box for ships and bridges, cylindrical bounding box for buoys. There are two types of cuboid bounding box: axis-aligned bounding box (AABB) and oriented bounding box (OBB), as shown in Figure 6.
The construction of an OBB bounding box relies on the spatial trend of the grid points of 3D model. The OBB bounding box is a better approximation of the object and can provide more accurate results when applied in the collision detection calculation. Moreover, when the pose of 3D model changes, it is not necessary to recalculate the OBB bounding box of the model. In the field of trajectory risk reconstruction, accuracy should be the primary condition. Therefore, we adopt the OBB bounding box to accelerate the ship collision detection calculation. However, the common method of constructing an OBB bounding box is relatively complicated. Covariance matrix, eigenvalue and eigenvector should be calculated to determine the direction of the main axes. Moreover, because the distribution of grid points of the 3D ship model is irregular, it is not accurate to use the eigenvector calculated by the covariance matrix as the main axis.
We design a new method in this paper. By designing simple transformations, the calculation efficiency of the OBB bounding box is greatly improved, especially during the real-time operation of the system. Although there is no complicated mathematical formula, the clever and efficient design is very useful in the actual system operation. Based on the ship's heading angle $a$ recorded in the ship traffic accident database, we calculate the three main axes (the local coordinate system of the ship model, as shown in Figure 7) of the 3D ship model by normalisation, rotation transformation and translation transformation, and use this coordinate system to build the OBB bounding box. The specific calculation process is as follows.
First, obtain the normal direction of the main axis $Ya$ by unitising the ship position coordinate $A$;
Then, take a small step along the longitude direction to get point $B$, project point $B$ onto the tangent plane at point $A$ to form a vector with point $A$ and unitise this vector to yield vector $X'$. At last, the vector $X'$ is rotated about the main axis $Ya$ to obtain the main axis $Za$ (the rotation angle is the heading angle $a$);
Finally, the main axis $Xa$ is calculated by $Ya$ and $Za$ using the right-handed spiral rule.
Compared with the common eigenvector calculation method of the OBB bounding box, the proposed method is simple and can accurately calculate the local coordinate system characterising the 3D ship model. The calculated coordinate system fully matches the spatial trend of the model. Based on the local coordinate system, the OBB bounding box of the model can be directly calculated according to the calculation method of the AABB bounding box. On the basis of the bounding box calculation, spatial intersection calculation is performed to complete the collision detection.
5.2 Data preprocessing
Before the loading of the 3D ship model, the model needs to be normalised. The normalisation calculation scales the 3D ship model to the correct scale relative to the constructed 3D ship traffic accident scene. In this paper, we use the reference object method. The first 3D ship model is normalised according to the reference object in the scene. The normalisation of subsequent 3D ship models is based on the first 3D ship model and the same reference calculation is carried out. Based on the ship traffic accident scene and the ship size information stored in the traffic accident database, the specific calculation process is as follows.
First, calculate the OBB bounding boxes of the 3D ship model and the reference object selected in the accident scene.
Then, scale the 3D ship model according to the reference condition. The reference condition assumes that the ratio of the diagonals of two OBB bounding boxes should be approximately equal to the ratio of the physical sizes of the two objects.
After the normalisation process, the 3D ship model can be loaded into the accident scene. The rotation and translation transformations also need to be performed according to the fusing method of the entity model in the 3D scene (Liu et al., Reference Liu, Zhao and Pan2016).
5.3 Risk calculation
There are many factors that influence ship collision accidents, and it is necessary to reconstruct meaningful factors for accident risk assessment, especially the factors that affect the crew's cognition of risk and the judgment of the next operation. Research results show that more than 80$\%$ of ship collision accidents are caused by human error (Rothblum, Reference Rothblum2000). We assume that human subjective factors have the same influence in different environments. This paper focuses on the influence of objective factors. We use the fuzzy comprehensive evaluation method to construct a fuzzy evaluation model that fuses multiple factors and 2D and 3D spatial calculations. By analysing the accident data, factor extraction and historical data mining are also used in the reconstruction of the risk of ship collision accidents. This model can be mapped to the crew's cognitive process. In this paper, we have collected a total of 497 relatively complete ship collision accident reports from maritime investigation report websites. The sources of accident reports include: Maritime Safety Administration of the People's Republic of China, Shanghai Maritime Safety Administration, Shandong Maritime Safety Administration, Nanjing Maritime Safety Administration, Jiangsu Maritime Safety Administration, Australian Transport Safety Bureau (ASTB), Marine Accident Investigation Branch (MAIB), The Danish Division for Investigation of Maritime Accidents (DMA), National Transportation Safety Board of United States (NTSB) and New Zealand Maritime Bureau (NZM). Furthermore, the visualisation method of the risk cognition model designed in this paper allows the crew to better visually recognise the risk calculation process.
At present, the fuzzy mathematics methods mostly use five parameters to assess the risk: DCPA, TCPA, orientation of the target ship relative to the own ship, speed ratio of the own ship and the target ship, and distance between the own ship and the target ship. In this paper, we select several factors of which the crew have a deep understanding to reflect the risk degree of the accident and construct more comprehensive influencing factors based on the data fusion. We use five factors to evaluate the risk: DCPA, TCPA, fusion of distance and yawing degree, relative bearing $G$ of the target ship relative to the own ship, and fusion of ship density and speed ratio. Among them, DCPA and TCPA are the most important factors influencing the ship risk, which can reflect the distance, relative speed and azimuth of two ships. Smaller values of DCPA and TCPA will result in a greater collision risk. Most research in the ship collision risk area takes DCPA and TCPA as parameters. Usually, the distance between the own ship and the target ship can give a more intuitive calculation of the collision risk. We take the compound factor of distance and yawing degree as a new parameter, which can not only express the influence of distance on risk, but also the influence of yawing degree on risk. When the distance is smaller and the yawing degree is larger, the value of this factor is higher, which reflects a higher degree of danger; on the contrary, it reflects a lower degree of danger. Incoming ships from different directions pose different danger degrees to the ship. Generally speaking, the hazard degree of the starboard side is greater than that of the port side, and the hazard degree before abeam is greater than that behind. Finally, we use the compound factor of ship density and speed ratio as a new parameter. This factor considers not only the influence of the target ship, but also the influence of surrounding ships. Based on the accident sample data, the independence of the constructed influencing factors is verified through Pearson correlation analysis.
We first need to determine the calculation of risk subordinating degree of each factor in the model. In this paper, we assess the trajectory risk of a ship collision accident, without considering ship manoeuvring factors such as the latest avoidance distance. Based on the ship boundary calculation and ship domain concept, we design the method for calculating the risk subordinating degree of each factor.
(1) Risk subordinating degree function of DCPA
The risk caused by DCPA to the ship is obvious. A larger value will mean a smaller degree of risk. Risk subordinating degree function of DCPA is as follows:
where $D$ is the absolutely safe encounter distance. When a large ship meets a small boat, the distance calculation should be more accurate. We design a 3D distance fusion calculation method in this paper. Moreover, accurate distance calculation can provide better support for the calculation of collision risk. The calculation of the distance is divided into two cases based on the classical ship domain model proposed by Fujii and Tanaka (Reference Fujii and Tanaka1971). In the ship domain, the 3D OBB bounding box is used to calculate the distance between two ships. First, 3D OBB bounding boxes of two ships are calculated. Then, the principal axis projection and the minimum distance calculation are performed. Outside the ship domain, we calculate the distance between the centre points of two ships. We stipulate that when $U({\rm DCPA})$ is 1, the trajectory risk is 1.
(1) Risk subordinating degree function of TCPA
The risk caused by TCPA to the ship is also obvious. A larger value will mean a smaller degree of risk. Risk subordinating degree function of TCPA is as follows:
where $T = {\sqrt {D^2 - {\rm DCPA}^2}}/V_R$, $V_R$ is the relative speed and $D$ is the absolutely safe encounter distance. We stipulate that when $U({\rm TCPA})$ is 1, the trajectory risk is 1. Based on the changing law of this factor and the number of corresponding accidents and the experience of experts, we adopt a quadratic power function.
(1) Risk subordinating degree function of the compound factor of distance and yawing degree
At present, most inland rivers adopt the traffic separation scheme (TSS). The impact of the ship's deviation from its navigation channel is significant. Moreover, through careful analysis of the ship's navigation characteristics, the close distance between the two ships does not mean a collision risk in most cases. Our algorithm considers both inland navigation and maritime navigation. The deviation of the ship from the routing or the centreline of the channel often indicates anomalies.
The regularity of accident attributes can be found by mining and analysing a large amount of accident data. Based on the collected 497 investigation reports of ship collision accidents, we conduct an analysis of the accident distance factor and the risk degree through data correlation analysis and data fitting analysis. Through statistical analysis of investigation reports, it is found that among 497 ship collision accidents, 350 ship collision accidents relate to close distance (accounting for 70${\cdot }$4%). It can be seen that the distance factor has a significant impact on the risk of ship collision. We classify the distance and count the number of accidents within each interval, as shown in Table 1.
Through data fitting, the relationship between the distance and the collision risk is as follows:
where $d$ represents the distance between ships and $U(d)$ represents the collision risk degree. The fitting function type is an exponential function. The variance and mean square error obtained are 0${\cdot }$009 and 0${\cdot }$055, respectively. The goodness of fitting and modified goodness of fitting are 0${\cdot }$9864 and 0${\cdot }$9818. It is a good fitting function.
Two ships can be very safe even if they are relatively close in the same inland waterway lane. Based on the electronic navigational chart data, we fuse the distance between two ships with yawing degree and construct a compound factor. The risk subordinating degree function is as follows:
where $W$ represents the width of the prescribed waterway; $w$ denotes the distance between the ship and centreline of the lane; $D$ is the absolutely safe encounter distance; and $d$ is the distance between two ships. Here, $\alpha$ and $\beta$ are the normalised weight coefficients. For offshore cases, $\alpha$ is larger, while for inland cases, $\beta$ is larger. We stipulate that when $U(d,w)$ is 1, the trajectory risk is 1.
(1) Risk subordinating degree function of relative bearing
Under the same conditions, incoming ships from different directions pose different danger degrees. When the relative bearing of the incoming ship is $19^\circ$ (the most dangerous situation), the risk subordinating degree is set to 1. When the relative bearing of the incoming ship is $199^\circ$ (the safest situation), the risk subordinating degree is set to 0. The risk subordinating degree function (Zhou and Wu, Reference Zhou and Wu2004) is as follows:
where $G$ represents the relative bearing.
(1) Risk subordinating degree function of the compound factor of ship density and speed ratio
Through statistical analysis of the investigation reports, it is found that among the 497 ship collision accidents, 134 ship collision accidents relate to the ship density factor (accounting for 27$\%$). It can be seen that the ship density factor also has a significant impact on the risk of ship collision. We classify the ship density and count the number of accidents within each interval, as shown in Table 2.
Through data fitting, the relationship between the ship density and the collision risk is as follows:
where $q$ represents the ship density and $U(q)$ represents the collision risk degree. The fitting function type is a power function. The variance and mean square error obtained are 0${\cdot }$1197 and 0${\cdot }$1998, respectively. The goodness of fitting and modified goodness of fitting are 0${\cdot }$854 and 0${\cdot }$8053. It is a good fitting function.
We use the ship density $q$ to describe the traffic flow, that is, the number of ships per km$^2$. Based on the calculation method of ${\rm DCPA}$ after the ship turns to avoid collision, the functional relationship between ${\rm DCPA}$ and ship speed ratio can be obtained through appropriate assumptions and simplifications (Zhou and Wu, Reference Zhou and Wu2004). According to this, we design the risk subordinating degree function as follows:
where $k$ is the speed ratio; $C$ is the collision angle; and $\alpha$ and $\beta$ are the normalised weight coefficients.
Finally, the ship's trajectory risk is calculated as follows:
Based on the advice of experts, the normalised weight items are initially set to (0${\cdot }$35, 0${\cdot }$35, 0${\cdot }$1, 0${\cdot }$1, 0${\cdot }$1). They are dynamically adjusted based on the case feedback and expert opinions. Based on the data mining, our method is capable of self-learning.
6. Application
6.1 Visualisation method
The primary task of visualisation is to accurately display and convey the information contained in the data, and to provide effective auxiliary means to help users to understand the data. To present data in an easy-to-understand manner, it is necessary to first convert the data into visual codes that are easy to perceive. A radar chart, also called a spider chart, is one type of visual code that displays multi-dimensional data in 2D form. It can quickly express the comparison of multiple indicators in the same coordinate system. On the basis of the designed risk factors, we use a radar chart to express the contribution of each factor in the assessment calculation and intuitively compare and analyse the factors in the reconstruction process.
6.2 Application results
To demonstrate the application effect of the proposed model and system, we take the typical ship collision accident ‘ship encounter’ as an example to carry out the trajectory risk reconstruction. Based on the ship trajectory and pose data stored in the ship traffic accident database, scene and risk calculations are used to realise the trajectory risk reconstruction of the two-ship collision accident. The location of the case is the Yangtze River waterway. A ship loses control and causes the collision.
Figure 8 shows a large-scale view of the two-ship collision accident scene. The red circle indicates the 3D model of the accident ship and the red arrow indicates the trajectory. When the trajectory is selected, the system will highlight the trajectory in white.
Figure 9 shows a view of the trajectory risk reconstruction of a two-ship collision accident. The system calculates the risk in real time. Moreover, the system gives a progress indication of the simulation. It can be seen from the figure that the influence of factors is different under different conditions. As shown in Figure 9(a), the risk subordinating degree of the relative bearing is bigger than the risk subordinating degree of the compound factor of the distance and yawing degree and the risk subordinating degree of the compound factor of the ship density and speed ratio. However, as the target ship loses control and changes direction, the ship deviates from the channel and the risk subordinating degree of the compound factor of the distance and yawing degree becomes increasingly bigger. The weighted risk subordinating degree values of the four nodes in Figure 9 are listed in Table 3.
The law can also be obtained from the numbers, but it is not as intuitive as the graphical display. Our calculation model can visually present the risk change process and all-round navigation environment to the crew. The system can also give a reconstruction of the falling process, as shown in Figures 10 and 11.
To verify the effectiveness of the system, we also conduct user case experiments. We obtain a total of 100 tests from seafarers and marine technology undergraduates of Shanghai Maritime University. We compare the user risk cognition of a 2D electronic chart system, 3D navigation simulator and our system under the same accident reconstruction situation and collect users’ satisfaction with the system. By averaging the obtained satisfaction scores, the results are listed in Table 4.
It can be seen from the experimental results that this system can effectively assist users in risk cognition of ship collision accidents.
7. Conclusion
This paper studies how to improve the crew's risk cognition from the perspective of multi-source data fusion and data mining, and provides the trajectory risk cognition method of a ship collision accident. We make full use of multi-source information to provide visualisation and perform risk model calculations. Traditionally, human decision-making is multi-factorial. When the crew has an all-round perception of the navigation environment, it is easier to find the risk problem. Our method solves the visualisation limitations caused by the 2D electronic navigational chart. It can provide integrated and effective information support for experts in the analysis of ship traffic accidents, and provide a new mode for the reproduction of ship traffic accidents. We perform extensive user experiments to evaluate the effectiveness of the system. We invite a large number of industry practitioners and students of this major to use our system. Users do the accident case study through our system and give a satisfaction score based on a 100-point grading system. We count their satisfaction scores with the system. Through a large number of user experiments, we have concluded that users show great interest in the system. Compared with industry practitioners, students are more interested in the system. They are also more satisfied. We think this may be because industry practitioners have received more training and can more easily generate a mental model of a map and match it with the real world. However, although students receive less training, our system can help them recognise the navigation environment and understand risk situations faster.
However, the system for the trajectory risk cognition developed in this paper does not perform the physical damage processing and analysis. Physical-based simulation technology can be used to improve the simulation effect of the system, and the system can be migrated to a virtual reality (VR) display device. Although it is very difficult to restore the actual physical change process of the ship in the accident (complex fluid-structure coupling simulation calculations need to be solved), it can improve users’ awareness of dangerous situations. Users can observe the ship's status from any angle and understand the changes of situation, which is not possible with accident videos. Furthermore, if users can use a VR display device to immerse themselves in the accident scene, it can improve the user's experience. Users will achieve a deep understanding of the risk. Of course, this requires optimisation of the computational efficiency to support a high refresh rate, and re-design of the system. Further work has to be done in these two areas.
Funding statement
This work was supported by the National Natural Science Foundation of China (T.L., grant numbers 52071201, 61602426); the Special Funding for the Development of Science and Technology of Shanghai Ocean University (Z.L., grant number A2-2006-21-200207); the Open Subject of the State Key Laboratory of Engines (Tianjin University) (Z.L., grant number K2019-14); the Fund of Hubei Key Laboratory of Inland Shipping Technology (T.L., grant number NHHY2019001); and the Open Project Program of the State Key Lab of CAD&CG (Zhejiang University) (T.L., grant number A2107).
Conflict of interest
The authors declare that there are no conflicts of interest regarding the publication of this paper.
Ethical standards
The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional committees on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008.