I. INTRODUCTION
Extraction of human behavior from radar data is important for a wide range of security and safety applications [Reference Ram, Christianson, Youngwook and Ling1, Reference Vignaud, Ghaleb, Le Kernec and Nicolas2]. One of the reported approaches is to use high-resolution range profiles (HRRP) to analyze human range-based signatures. HRRP allows for extraction of the micro-motions induced by different parts of the human body [Reference Fogle and Rigling3]. Another approach is to use the micro-Doppler image to analyze target Doppler-based signatures [Reference Chen4]. As a tool to illustrate Doppler spectra along the slow-time axis, the micro-Doppler images have been studied widely due to their potential in human classification [Reference Chen5]. The range-Doppler (RD) images were also used to analyze distributed human scatterers [Reference Vignaud, Ghaleb, Le Kernec and Nicolas2]. Although HHRP, micro-Doppler, and RD images all show certain target information, they are restricted to observe targets in either range, Doppler or RD, and the target slow-time evolution has not been addressed in all these signals.
Slow-time behavior for human motions (e.g. walking, crawling, and running) is unique. For example, the standard walking procedure can normally be seen as a periodic movement with a certain cadence frequency. Human periodic motion has been recently analyzed in camera-based systems [Reference Cutler and Davis6, Reference BenAbdelkader, Cutler and Davis7]. Cutler and Davis [6] proposed using self-similarity matrix (SSM) of one optical image sequence to detect and analyze periodic motions. For action recognition, Junejo et al. [Reference Junejo, Dexter, Laptev and Perez8] claimed that an important structural stability of SSM can be found for a moving person observed by different cameras. Although radar signals differ significantly from optical images, in this paper we will demonstrate that the SSM theory can be extended for radar signals with some adequate modifications.
Our approach is to apply SSM to extract the slow-time feature of typical human radar backscattering (i.e. HRRP, micro-Doppler image, and RD video sequence). Range-Doppler video sequence (RDVS) [Reference He, Le Chevalier and Yarovoy9] is one sequence of RD images, as a function of slow-time. RDVS not only preserves the target range information, but also keeps the Doppler information, as a function of slow-time. The SSMs obtained from different signals will be analyzed, and their angle-invariant characteristic will be demonstrated by comparing the SSMs from radars deployed at different locations. Fourier transform-based periodicity detection method will be developed to extract the cadence frequency of the human gait from SSMs. Finally, the proposed approach will be validated on experimental data.
This paper is organized as follows. The radar backscattering from the walking human model is described in Section II. Section III discusses SSM of human backscattering in radar. Section IV addresses the gait periodicity detection from SSM. Section V demonstrates the measurement result. Section VI presents the conclusion.
II. RADAR REFLECTION SIMULATED BY HUMAN MODEL
A) Human target motion model
Two different human motion models have commonly been used in the literature: mathematical parametric model [Reference Van Dorp and Groen10] and empirical non-parametric model [Reference Vignaud, Ghaleb, Le Kernec and Nicolas2]. Although parametric models are designed for some specific motions (i.e. walking and running), empirical non-parametric models, which are based on human motion capture data, are usually more realistic and accurate. In CMU database [11], a motion capture system consisting of 12 infrared cameras and 41 markers attached to a human body, was used to record motions. Datasets of various motion types (e.g. walking, running, and jumping) are provided. For this study, we select five most crucial markers (i.e. torso, left hand, right hand, left foot, and right foot), and an equal RCS is assumed for the reflection from all the markers for simplicity. Then we construct human range profiles by coherently summing the echoes from different parts of the human body. According to the human kinematic studies [Reference Chen5], one complete human gait of our model consists of four main phases (see Fig. 1): (1) double support (both feet are on the ground), (2) right stance (only the right foot is on the ground), (3) double support, and (4) left stance (only the left foot is on the ground).
Fig. 1. One complete human gait.
B) Radar signals of the human target
Currently there are three tools to analyze radar targets, namely HRRP, micro-Doppler images, and RD images. Range profile is used as the most common tool due to its simplicity. We simulated range profiles for a walking human model (Fig. 1) using Gaussian pulses with an operational frequency from 0.8 to 1.6 GHz (−10 dB), which provides a theoretical down-range resolution of 0.18 m. The range profiles simulated show the slow-time evolution of five different scatterers (see Fig. 2). The largest fluctuations in the range profiles result from the backscattering of the feet, and the reflections of the torso and hands are heavily overlapped. Although a certain level of target signatures is preserved in range profiles, the range profiles do not show the Doppler information.
Fig. 2. Simulated human target range profiles.
The micro-Doppler of a walking human can be derived by applying time-frequency transform to the range profiles. Different transforms (e.g. short-time Fourier transform and Wigner–Ville distribution) can be chosen considering the trade-off between Doppler resolution and cross-term interference. The micro-Doppler image corresponding to the range profiles in Fig. 2, computed by short-time Fourier transform, is shown in Fig. 3. Unlike the range profiles, micro-Doppler preserves the target Doppler signature, while it neglects the range information.
Fig. 3. Simulated human target micro-Doppler image.
Similar to the micro-Doppler image, RD image is also derived from range profiles. The basic idea of RD processing is to apply Fourier transform to analyze the samples from a fixed range bin over a few successive range responses. The RD image is constructed by the spectra in all the range bins. Although the micro-Doppler image describes the target Doppler signature over a certain period, the RD image only shows the RD signature at one specific instant. The main stages of one human gait, corresponding to the phases in Fig. 1, are illustrated in four RD images (Fig. 4). A cyclic movement is constructed by the RD responses of the arms and the feet around the torso contour. Note that the arm responses are difficult to separate from the torso response, since they overlap with each other.
Fig. 4. The range-Doppler images of the simulated human gait (the person is walking toward the radar with a constant velocity).
The range profiles and the micro-Doppler image both illustrate the target movement as a function of the slow-time, whereas the RD image describes the RD pattern at one particular instant. Therefore, we proposed RDVS [Reference He, Le Chevalier and Yarovoy9] previously to describe the slow-time evolution of target RD signatures. As shown in Fig. 5, RDVS is built by successive RD frames. One pulse repetition interval (PRI) can be used as a typical interval between two frames. Only five of all the RDVS frames are shown here, and the human RD contours between consecutive frames are also demonstrated.
Fig. 5. Simulated range Doppler video sequence of the human target.
Range profiles, micro-Doppler image, and RDVS represent the radar time series based on range, Doppler, and RD signatures, respectively. As we shall see, they all exhibit unique self-similarity characteristics, due to the cyclic human movement. In the next section, the self-similarity characteristic of these three different radar signals will be analyzed.
III. SSM OF HUMAN BACKSCATTERING
A) Self-similarity matrix (SSM)
Recurrence is quite common in dynamic systems, and multi-dimensional time series is usually used to represent such systems. Eckmann et al. first proposed the recurrence plot (RP) [Reference Eckmann, Kamphorst and Ruelle12] to perform time series analysis of dynamic systems. As a variant of RP, SSM [Reference Cutler and Davis6] can be considered as a generalization of the correlation matrix, measuring the “similarity” – rather than just the correlation – between successive samples of the multidimensional time series. SSM is also an effective tool to analyze system behaviors, since the textures in SSM (e.g. the peaks and valleys, if we illustrate SSM as an image) are distinctive for different time series.
In the field of video processing, SSM has already been successfully applied for analyzing human motions in video sequences, such as multi-view human action recognition [Reference Junejo, Dexter, Laptev and Perez8], human gait recognition [Reference BenAbdelkader, Cutler and Davis7, Reference Cutler and Davis13], and video matching [Reference Shechtman and Irani14]. For the application of camera-based human action recognition, Junejo et al. [Reference Junejo, Dexter, Laptev and Perez8] claimed that an important structural stability of SSM can be found for a moving person observed with different cameras. This indicates that, while cameras observe the person from various ranges and perspectives, a similar slow-time feature (i.e. SSM) can still be captured.
Since range profiles, micro-Doppler image, and RDVS can all be seen as multi-dimensional time series, SSM should also be useful to analyze them. Although radar signals differ significantly from camera-based optical images, we will demonstrate that the SSM theory can be usefully extended to radar signals.
The range profiles, micro-Doppler image, and RDVS are represented by multi-dimensional time series S = {F 1, F 2,…,F N}. The dimension of S depends on the specific type of the radar signal. Taking the range profiles for example, they are the time series consisting of N one-dimensional vectors (pulse responses). Similarly, micro-Doppler time series are composed of N one-dimensional vectors (Doppler responses), and RDVS time series have N two-dimensional vectors (RD images). Then, the SSM of S is defined as a square symmetric matrix:
![](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:98604:20160415030944547-0741:S1759078714000087_eqn1.gif?pub-status=live)
Here d ij denotes the similarity coefficient, which measures the degree of similarity between F i and F j.
The exact form of the SSM depends on the similarity measure used for calculating d ij. Different similarity measures have already been proposed [Reference Van der Weken, Nachtegael and Kerre15]. The selection of a specific measure depends on particular applications. Mutual information (MI) [Reference Pluim, Maintz and Viergever16], a well-known statistical measure for comparing vectors, is selected to compute the SSM for radar signals. Unlike other classical measures (e.g. Euclidean distance, normalized cross-correlation) that compare intensities directly, the MI approach, which is based on Information Theory, computes the statistical dependence between two vectors using histograms. Accordingly, MI-based SSM is independent of exact target profiles (range, Doppler, or RD) and thus is less sensitive to the range alignment process – an essential step to compute SSM in the next section.
Since MI describes the shared information between two vectors statistically, the MI-based SSM does provide a simple way to quantitatively exhibit time series patterns. The MI between vector X and Y is defined by:
![](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:57350:20160415030944547-0741:S1759078714000087_eqn2.gif?pub-status=live)
where H(X) denotes the Shannon entropy of vector X, and H(X,Y) indicates the joint entropy of X and Y. Shannon entropy [Reference Shannon17] quantifies the expected information contained in a vector. It has been widely used in radar-related applications. In [Reference Li18], entropy-based approach is applied to remove noise in micro-Doppler images. The threshold used in denoise procedure is selected by maximizing the sum of the object and background entropies. Entropy is also used to assess the additional information introduced by new features in human target classification [Reference Molchanov, Astola, Egiazarian and Totsky19]. In this study, MI is defined via the entropy concept. The entropy of vector X is defined by:
![](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:83257:20160415030944547-0741:S1759078714000087_eqn3.gif?pub-status=live)
where M is the number of histogram bins used to estimate the empirical probability density. The histogram meets the following condition:
![](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:99777:20160415030944547-0741:S1759078714000087_eqn4.gif?pub-status=live)
where N p denotes the size of the vector X, and N k indicates the number of the samples whose intensities fall into the kth bin. The empirical probability density in the mth bin is given by:
![](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:64769:20160415030944547-0741:S1759078714000087_eqn5.gif?pub-status=live)
Similarly, the joint entropy H(X,Y) can also be defined as:
![](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:38379:20160415030944547-0741:S1759078714000087_eqn6.gif?pub-status=live)
where P mn denotes the joint empirical probability density of vector X and Y. More precisely, P mn indicates the probability for a given sample, which falls into the mth bin in histogram i−1, to be located in the nth bin in histogram i (it is a kind of transition probability). It is given by:
![](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:49526:20160415030944547-0741:S1759078714000087_eqn7.gif?pub-status=live)
where N q is the size of class set Q mn. The set Q mn is defined as:
![](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:75236:20160415030944547-0741:S1759078714000087_eqn8.gif?pub-status=live)
where g(i,l) is the operator to get the histogram value of vector i at sample l(1 ≤ l ≤ N p), and 1 ≤ m, n, ≤ M. Note that here we assume X and Y have the same size N p.
Then, MI can be derived by equations (2), (3), and (6) as:
![](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:78727:20160415030944547-0741:S1759078714000087_eqn9.gif?pub-status=live)
It can be analyzed as a similarity measure between the actual empirical joint probability, P mn, and the empirical joint probability in the case of independent vectors, P mP n. Finally, the entries in SSM can be obtained by:
![](https://static.cambridge.org/binary/version/id/urn:cambridge.org:id:binary:10188:20160415030944547-0741:S1759078714000087_eqn10.gif?pub-status=live)
B) Extracting SSM of radar signals
In general, extracting the SSM of the radar signals consists of the following steps. First, we track the targets in range or RD domain. Then the range profiles (or RDVS) are aligned using target trajectory information. The aligned range-profiles and RDVS can be seen in Figs 6 and 7, respectively. Note that the micro-Doppler image does not require alignment, since it naturally sums the information from all the range cells. Although this study did use the position of the torso marker as the centroid in alignment for simplicity, more details about tracking and alignment for real data can be found in our previous work [Reference He, Savelyev and Yarovoy20]. It is also worth pointing out that accurate tracking is not necessarily required for computing SSM, since MI depends on histograms, rather than the target profiles. At last, MI-based SSM can be computed for different radar signals.
Fig. 6. Alignment of the range profiles.
Fig. 7. Alignment of the range-Doppler video sequence.
The SSMs of different radar signals are shown in Fig. 8. We may note the following unique properties of SSM: (1) The diagonal shows the maximum amplitude of each row, because the MI between one image and itself is always maximum; (2) The repeated structures demonstrate the periodic characteristic of the human motion, such as the bright lines parallel to the main diagonal in (a) and (c), and the bright “square” in (b). Although the textures of (a), (b), and (c) all show some distinctive target signatures, the SSM of RDVS in (c) obviously demonstrates the target periodic characteristic in a more direct and simple way. One possible explanation could be that RDVS is more informative than range profiles and micro-Doppler image, as both range and Doppler information of the target are used. The parallel lines in (c) are experimentally found to be corresponding to the cyclic motion of the human, and in fact the distance between two parallel lines indicates the period of a half cycle of the human motion (i.e. one complete motion of one leg).
Fig. 8. Self-similarity matrix of radar signals.
C) Angle-invariant characteristic of SSM
Since SSM mainly exhibits the self-repeating information of the target motion, and MI is independent of target range/Doppler profiles, SSM is considered to be invariant to radar observation angles. Strictly speaking, SSM is not fully angle-invariant around 90° and 270°, since targets can often not be detected in these angles (detection is the pre-condition for analyzing SSM). The SSMs (Figs 9(b)–9(e)) were simulated from four different radar locations, and the simulation scenario is shown in Fig. 9(a). Results show that the HRRP-based SSMs presents weaker periodic structures, and the micro-Doppler-based SSMs are not robust at some angles (i.e. 135°, 225°). On the contrary, the RDVS-based SSMs demonstrate similar periodic textures in all aspect angles. This concludes that RDVS is able to produce robust angle-invariant SSMs. These important characteristics of SSM indicate its potential in the applications of target association, classification and recognition in multistatic radars.
Fig. 9. (a) A person is walking from P1 to P2. Radars are deployed at four locations; (b)–(e) Self-similarity matrix of different radar signals from four radar locations.
D) Component analysis of SSM
Since the RDVS-based SSM is seen to be more robust, it has been used for further analysis. The SSMs of different body parts are shown in Fig. 10. Although the SSM of the torso and hands (Fig. 10(a)) show only weak similarity along the slow-time axis, the SSM of the torso and feet (Fig. 10(b)) show clear bright lines that are parallel to the main diagonal. This indicates that the most obvious contribution for SSM is from the combination of torso and feet. One possible reason for this phenomenon could be that the responses of feet and torso are more separable in one RD image, which results in larger contour changes in the adjacent frames of the RDVS. It can also be inferred from the results that the hands movement seems less periodic than the feet movement, and thus spreads over the SSM image.
Fig. 10. Self-similarity matrix of different body parts.
IV. GAIT PERIODICITY ANALYSIS
Periodicity is an important feature of human motion. Fourier transform is a common tool to analyze the frequency components of periodic motion. To determine the gait periodicity, we estimate the spectrum amplitudes of all the row vectors in SSM. The vectors are detrended linearly, and the final spectrum of SSM is estimated by averaging the spectra obtained from all the row vectors [Reference Cutler and Davis13]. The cadence frequencies of the four RDVS-based SSMs in Fig. 9 are shown in Fig. 11. Similar cadence frequencies (around 1.7 Hz) are obtained from all the radars. This result demonstrates again the angle-invariant characteristic of SSM shown in Fig. 9. The SSMs for a walking/running person (see Figs 12(a) and 12(b)) clearly show the difference between two actions in terms of different repetitive parallel lines. Figure 12(c) points out that different human activities (i.e. walking and running) can be distinguished according to their cadence frequencies (1.7 and 2.7 Hz). Therefore, SSM can be a promising tool for human target classification and recognition.
Fig. 11. Cadence frequencies extracted from the SSMs in Fig. 9.
Fig. 12. Gait analysis for different human activities.
V. EXPERIMENTAL VALIDATION
A) Measurement set-up
The measurement was conducted in a room at Delft University of technology to provide a typical indoor environment. An m-sequence pseudo-noise radar equipped with one transmitting horn antenna and two receiving horn antennas was used for the data acquisition (Fig. 13(a)). The operational frequency band (at −10 dB level) of the radar is 0.8–1.6 GHz, which gives a down-range resolution of 0.187 m. The sampling clock frequency is 4.5 GHz and the length of the sampling window in propagation time is 113 ns, which covers an unambiguous range of 16.95 m. In the receiver, the received signal is correlated with a replica recorded from the transmitter. Thus a long m-sequence signal is compressed to a short impulse and the signal-to-noise-ratio is increased significantly via pulse compression. The PRI is 0.06 s, which leads to a maximum unambiguous Doppler frequency of 8.33 Hz for the RD processing. The measurement was carried out with one human target moving in the room from P1 to P2 (Fig. 13(b)) with a constant velocity.
Fig. 13. Measurement setup and scenario.
B) Range-doppler video sequences
The RDVS was generated by RD processing in each receiver. Each frame was computed with a CPI duration of 1 s (16 successive pulses), and the time between two successive frames is one PRI around 0.06 s. The range profiles used for generating RDVS is shown in Figs 14(a) and 14(b), and the RD frames in both channels at one specific slow-time instant are illustrated in Figs 14(c) and 14(d). For simplicity, the maximum amplitude of each RD image was selected as the centroid of the human target for target alignment.
Fig. 14. Human backscattering in multistatic UWB radar.
C) Self-similarity matrix
The SSMs obtained in different receivers are shown in Figs 15(a) and 15(b). It can be noted that they are very similar to the SSMs obtained in the previous simulation (see Figs 8 and 9). The periodic characteristic of the human motion is demonstrated clearly by the repeated lines, and the time between two parallel lines indicates the period of a half cycle of the human motion. The pixelization of these two images is due to the lower frame repetition frequency (16.7 Hz) of the RDVS, whereas 120 Hz was used in previous simulations. The cadence frequencies of the two SSMs are shown in Fig. 15(c). Similar cadence frequencies (around 1 Hz) were obtained in both receivers. This proves again the angle-invariant characteristic of the SSM. Finally, it is interesting to point out that SSM preserves rich target information, and cadence frequency is just one of the examples. Further investigation can be interesting to analyze SSM and extract more target signatures from it.
Fig. 15. Gait analysis for the measured human target data.
VI. CONCLUSION
This study introduces the SSM approach to analyze human slow-time behavior in the high-resolution radar. Motion capture data were used to model typical radar signals (i.e. range profiles, micro-Doppler image, and RDVS). The self-similarity matrices with MI as the similarity measure have been computed for simulated data. All the SSMs showed a significant level of human periodic information. The SSM of the RDVS has been further analyzed, and its angle-invariant characteristic has been demonstrated. Fourier transform-based periodicity detection was developed to determine the cadence frequency, and different human activities (i.e. walking and running) were separated from each other based on the cadence frequencies. Finally, real experiments were conducted to validate the simulation results. Using the proposed slow-time feature, the SSM-based approach has good potential for separation of multiple human targets observed by a single high-resolution radar as well as target association in distributed radar systems for area surveillance. In future, the sensitivity of SSM to noise, clutter, and multipath will have to be analyzed in detail.
Yuan He received his B.Sc. and M.Sc. degrees in Information and Communication Engineering from National University of Defense Technology (China) in 2007 and 2010, respectively. He is now working toward his Ph.D. degree at Delft University of Technology (the Netherlands). His main research interests are multistatic UWB radar system design and signal processing.
Pascal J. Aubry received the D.E.S.S. degree in Electronics and Automatics from the Université Pierre et Marie Curie (Paris 6), Paris, France, in 1993. Following his military service in the Air Force in 1994, he was a Young Graduate Trainee with the European Space Research and Technology Centre (ESTEC), in 1996, where he was involved with antenna measurements. Since 1997 he has been with the International Research Centre for Telecommunications–Transmission and Radar (IRCTR), Delft University of Technology (TUD), Delft, the Netherlands. His interests include antenna measurement techniques, radar system testing, and signal processing and analysis.
François Le Chevalier is Chief Scientist, Thales Land and Air Systems, and Professor, Radar Systems Engineering, TU Delft. A French pioneer in adaptive systems for Land and Air applications, he is now conducting research on space-time active antenna systems and Wideband radar processing and applications. He will be the Honorary Chair of the International SEE-IEEE Radar Conference in Lille, France, October 2014.
Alexander Yarovoy leads a chair of Microwave Sensing, Systems, and Signals at the Delft University of Technology, the Netherlands. His main research interests are in ultra-wideband microwave technology and its applications (in particular, radars) and applied electromagnetics (in particular, UWB antennas). He has authored and co-authored more than 250 research articles, four patents, and fourteen book chapters.