2.1 Augmented reality

The beginnings of augmented reality technology can be dated back to a pioneer work in the 1960s. A see-through head-mounted device (HMD) was designed to present three-dimensional graphics (Sutherland, Reference Sutherland1968). The fundamental idea behind the three-dimensional display proposed in the work was to present the user with a perspective image which changes as he moves. As the retinal images measured by eyes are two-dimensional projections, it is possible to create three-dimensional object illusions by placing suitable two-dimensional images on the observer's retina.

Two different coordinate systems have been defined: the room coordinate system and the eye coordinate system. The viewer position and orientation is estimated at all times with a tracking device. These estimates yields the instantaneous transformation from the room coordinate system to the eye coordinate system. Reference points are described in the room coordinate system. The coordinates of these points are transformed to the eye coordinate system and then projected into the image scene with perspective projection models.

Over the years that followed, further research on the proposed concept resulted in the emergence of augmented reality as a research field. Milgram et al. (Reference Milgram, Takemura, Utsumi and Kishino1995) categorised augmented reality displays into two subclasses: see-through and monitor-based displays. See-through displays are inspired by the earlier work of Sutherland (Reference Sutherland1968), while monitor-based displays refer to those in which computer-generated images are overlaid onto live or stored video images. Furthermore, Azuma (Reference Azuma1997) published an important survey summarising the main developments up to that point. Figure 1 shows a conceptual diagram for a monitor-based system.

Figure 1. Monitor-based AR conceptual diagram – adapted from Azuma (Reference Azuma1997)

Generally, a set of virtual points to be displayed is defined with respect to a coordinate system external to the camera. For each image measured by the camera, the corresponding transformation between the camera coordinate system and this external coordinate system is applied for the set of virtual points.

2.2 Camera projection

Consider a world coordinate system described with axes $[\vec {x}, \vec {y}, \vec {z}]$ and a ship coordinate system described with axes $[\vec {i}, \vec {j}, \vec {k}]$. Each axis from the world coordinate system may be expressed as a combination of axes from the ship coordinate system:

(2.1)\begin{align} \vec{x} = x_i \vec{i} + x_j \vec{j} + x_k \vec{k} , \quad \vec{y} = y_i \vec{i} + y_j \vec{j} + y_k \vec{k} , \quad \vec{z} = z_i \vec{i} + z_j \vec{j} + z_k \vec{k} \end{align}

Thus, an arbitrary vector $\vec {a}$ described with three-dimensional coordinates $\vec {a}_w$ in the world coordinate system transforms to the ship coordinate system as follows:

(2.2)\begin{align} \vec{a}_s = \begin{bmatrix} x_i & y_i & z_i \\ x_j & y_j & z_j \\ x_k & y_k & z_k \end{bmatrix} \vec{a}_w \end{align}

Let $\vec {s}$ be the origin of the ship coordinate system described in the world coordinate system:

(2.3)\begin{equation} \vec{s} = s_x \vec{x} + s_y \vec{y} + s_z \vec{z} \end{equation}

Consider a point $P$ in the environment with coordinates $\vec {p}_w$ in the world coordinate system and $\vec {p}_s$ in the ship coordinate system. The transformation that relates the coordinates of $P$ in each coordinate system can be defined as (Hartley and Zisserman, Reference Hartley and Zisserman2004)

(2.4)\begin{align} \vec{p}_s & =\begin{bmatrix} x_i & y_i & z_i & -(x_i s_x + y_i s_y + z_i s_z) \\ x_j & y_j & z_j & -(x_j s_x + y_j s_y + z_j s_z) \\ x_k & y_k & z_k & -(x_k s_x + y_k s_y + z_k s_z) \end{bmatrix} \begin{bmatrix} \vec{p}_w \\ 1 \end{bmatrix} \end{align}

(2.5)\begin{align} \vec{p}_w & = \begin{bmatrix} x_i & x_j & x_k & s_x \\ y_i & y_j & y_k & s_y \\ z_i & z_j & z_k & s_z \end{bmatrix} \begin{bmatrix} \vec{p}_s \\ 1 \end{bmatrix} \end{align}

Further, consider a camera coordinate system described with axes $[\vec {\chi }, \vec {\gamma }, \vec {\kappa }]$. Axis $\vec {\chi }$ is oriented from left to right, axis $\vec {\gamma }$ is oriented downwards and axis $\vec {\kappa }$ is forward oriented. Let each axis from the ship coordinate system be expressed as a combination of axes from the camera coordinate system:

(2.6)\begin{equation} \vec{i} = i_\chi\vec{\chi} + i_\gamma \vec{\gamma} + i_\kappa \vec{\kappa} , \quad \vec{j} = j_\chi\vec{\chi} + j_\gamma \vec{\gamma} + j_\kappa \vec{\kappa} , \quad \vec{k} = k_\chi\vec{\chi} + k_\gamma \vec{\gamma} + k_\kappa \vec{\kappa} \end{equation}

Let $\vec {c}$ be the origin of the camera coordinate system described in the ship coordinate system:

(2.7)\begin{equation} \vec{c} = c_i \vec{i} + c_j \vec{j} + c_k \vec{k} \end{equation}

The transformation relating coordinates of a point $P$ described in the ship coordinate system or in the camera coordinate system may be similarly determined:

(2.8)\begin{align} \vec{p}_c & =\begin{bmatrix} i_\chi & j_\chi & k_\chi & -(i_\chi c_i + j_\chi c_j + k_\chi c_k) \\ i_\gamma & j_\gamma & k_\gamma & -(i_\gamma c_i + j_\gamma c_j + k_\gamma c_k) \\ i_\kappa & j_\kappa & k_\kappa & -(i_\kappa c_i + j_\kappa c_j + k_\kappa c_k) \end{bmatrix} \begin{bmatrix} \vec{p}_s \\ 1 \end{bmatrix} \end{align}

(2.9)\begin{align} \vec{p}_s & = \begin{bmatrix} i_\chi & i_\gamma & i_\kappa & c_i \\ j_\chi & j_\gamma & j_\kappa & c_j \\ k_\chi & k_\gamma & k_\kappa & c_k \end{bmatrix} \begin{bmatrix} \vec{p}_c \\ 1 \end{bmatrix} \end{align}

Thus, for a point described with coordinates $\vec {p}_w$ in the world coordinate system, the corresponding coordinates $\vec {p}_c$ in the camera coordinate system can be computed with two sequential transformations. Once a point is described in the camera coordinate system, it is possible to use camera models for its projection as a virtual point in the image scene.

Let $p = [\chi \ \gamma \ \kappa ]^\textrm {T}$ be a scene point described in the camera coordinate system and let $i = [u \ v]^\textrm {T}$ be its corresponding image projection in pixels. Ideally, the relationship between $p$ and $i$ may be expressed with the pinhole model, which is given by (Hartley and Zisserman, Reference Hartley and Zisserman2004)

(2.10)\begin{equation} \begin{bmatrix} u \\ v \end{bmatrix} = \begin{bmatrix} f \Delta_u^{{-}1} \dfrac{ \chi } { \kappa } + c_u \\ f \Delta_v^{{-}1} \dfrac{ \gamma } { \kappa } + c_v \end{bmatrix} \end{equation}

Parameters $[f \Delta _u^{-1}\ \ f \Delta _v^{-1} \ \ c_u \ \ c_v]$ are called intrinsic parameters or simply camera parameters. This model assumes a process of central projection. Light from the scene reaches the camera through a unique point referred to as the camera centre. Measurements are sampled in a plane at a distance $f$ from the camera centre. Distance $f$ is usually referred to as the focal distance of the camera. The rest of the camera parameters represents the conversion from sensor plane to pixel units. Note that more sophisticated models may be developed by taking into consideration mechanical properties of the lens and typical hardware components (Mahmoudi et al., Reference Mahmoudi, Sabzehparvar and Mortazavi2021).

From the correspondences of image coordinates and three-dimensional positions, the camera parameters can be estimated by minimising the error between image observations and their respective model projections. Simplified procedures to estimate the camera parameters are usually based on the camera observation of structures with known dimensions and easily detectable features. A common procedure is based on the observation of a known planar pattern at a few different orientations (Zhang, Reference Zhang2000). Figure 2 shows examples of such calibration patterns as measured by a camera. These planar patterns may be printed with any commercial printer.

Figure 2. Observation of known planar patterns for calibration

The size of each square from the images above is known. Thus, it is possible to define three-dimensional coordinates for each square intersection of the pattern. Calibration parameters may be estimated from these correspondences with open source libraries. The present work uses the open source library OpenCV (Bradski, Reference Bradski2000) for most implementations regarding image manipulation. Particularly, the library is used for calibrating the camera, projection of points and synthesis of graphical elements. Figure 3 shows the projection of virtual points associated with the previous calibration pattern using the OpenCV library.

Figure 3. Detections and corresponding projections of the coplanar virtual points used in the calibration

Note that all virtual points drawn above belong to a particular plane $\pi$ coincident with the calibration pattern. An arbitrary plane $\pi$ may be parametrised by a three-dimensional point $\pi _p$ belonging to $\pi$ along with two unitary orthogonal vectors $\pi _{\hat {n}_x}$ and $\pi _{\hat {n}_y}$ parallel to $\pi$. Thus, the coordinates of other points $p$ from the plane may be computed from $\pi _p$ and a linear combination of $\pi _{\hat {n}_x}$ and $\pi _{\hat {n}_y}$:

(2.11)\begin{equation} p = \pi_p + \lambda_1 \pi_{\hat{n}_x} + \lambda_2 \pi_{\hat{n}_y},\quad \lambda_1 , \lambda_2 \in \mathbb{R} \end{equation}

A particular set of points of the plane $\pi$ may be defined by iterating different pairs $(\lambda _1,\lambda _2)$. Assuming that plane parameters $\pi _p$, $\pi _{\hat {n}_x}$ and $\pi _{\hat {n}_y}$ are described with respect to the camera coordinate system, auxiliary lines can be straightforwardly projected into the scene with above camera models.

2.3 Experimental setup

Consider a video generated from a navigation experiment by the TPN-USP Ship Maneuvering Simulation Center. The TPN-USP is the largest Brazilian ship manoeuvring simulation centre, equipped with three full-mission simulators and three tug stations, as well as one part-task simulator. Tannuri et al. (Reference Tannuri, Rateiro, Fucatu, Ferreira, Masetti and Nishimoto2014) describe the mathematical model adopted in the simulator, and Makiyama et al. (Reference Makiyama, Szilagyi, Pereira, Alves, Kodama, Taniguchi and Tannuri2020) presents the visualisation framework, able to generate realistic images of the maritime scenario, in real-time. In the video from the experiment, all geometrical parameters are accurately known. Figure 4 illustrates the scene measured by the camera alongside the geometry of the ship from the simulation.

Figure 4. Ship geometry and visualisation of the simulation experiment

An estimation of the intrinsic parameters of the camera is provided. A set of points with known coordinates may be projected into the image to validate accurateness or to further optimise camera parameters. This is illustrated in Figure 5.

Figure 5. Detections in the image plane are highlighted on the left; projections from the ship geometry and the camera parameters are shown on the right

The state of the ship with respect to the world coordinate system is known from the simulation outputs. Figure 6 shows the ship position as a function of time.

Figure 6. Ship position as a function of time with respect to the world coordinate system

Each buoy has fixed coordinates in the world coordinate system during the simulation. Thus, their instantaneous relative position in the ship coordinate system may be computed with expressions from Equations (2.4) and (2.5).

These coordinates in the ship coordinate system may be transformed to the camera coordinate system with expressions from Equations (2.8) and (2.9). Figure 7 shows the instantaneous relative position of one of the buoys with respect to the camera coordinate system.

Figure 7. Buoy position as a function of time with respect to the camera coordinate system

Further, consider another video from a real experiment of a ship navigating through a channel delimited by nautical buoys with a fixed onboard camera. Figure 8 summarises the installation of the camera for this experiment alongside relevant geometrical information from the ship. The ship orientation with respect to the world coordinate system is assumed to be constant and known during the entire experiment. Note that in this real experiment, the instantaneous position of nearby obstacles are unknown.

Figure 8. Geometrical information from the ship (top picture) and installation of the camera in the ship (bottom picture)

Intrinsic parameters of the camera are known from calibration procedures, such as the implementation from the OpenCV library described in the previous section. The camera position and orientation with respect to the ship frame must be known. Thus, it is interesting to install the camera in a known location from the ship such as in its bridge or its cabin. Then, these parameters may be estimated after installation with information from the ship geometry, which are typically available in the pilot cardboard and the wheelhouse poster, as shown in Figure 8.

If there are known correspondences between camera measurements and three-dimensional coordinates, it is possible to directly estimate the camera position and orientation after installation. For example, consider that all containers from the scene have known dimensions. For each block of adjacent containers, a set of correspondences may be defined with an arbitrary origin and adjacent container points. Then, optimising each set of correspondences yields estimates for the position and orientation for each block of containers. Finally, an estimation for the orientation of the camera may be defined assuming the alignment of all containers with the ship coordinate system. Figure 9 exemplifies the procedure with visible points from four blocks of adjacent containers.

Figure 9. Camera calibration with visible points from onboard containers

Henceforth, the camera position and orientation is assumed to be known with respect to the ship coordinate system. Virtual points around the surface of the sea may be determined from Equation (2.11) by combining the ship draft with the installed camera position and orientation.

For a plane $\pi$ described with respect to a coordinate system external to the camera, the corresponding plane parameters $\pi _p$, $\pi _{\hat {n}_x}$ and $\pi _{\hat {n}_y}$ must be transformed to the camera coordinate system. Assuming knowledge of the rotation matrix relating both frames along with the origin of the external coordinate system, each vector $\pi _{\hat {n}_x}$ and $\pi _{\hat {n}_y}$ may be transformed to the camera coordinate system with a matrix multiplication, as presented in Equation 2.2. The point plane $\pi _p$ may be transformed similarly as in Equations (2.4), (2.5), (2.8) and (2.9).