3.1. Trajectory generation for ankle joint

In the swing phase, the step length of one step is set as s, and the angle between the sole of the foot and the ground is $\theta(t)$ .

(14) \begin{equation} \theta(t) = \left\{ \begin{array}{c}0,t = kT \\ \theta_{s},t = kT + d \\ 0,t = kT + t_{h}\\ \theta_{f},t = (k + 1)T\\ 0,t = (k + 1)T + d \end{array} \right. \end{equation}

where k is positive integer, and k ≥ 0. T is time of one gait cycle.

During the swing phase, the posture of the swing leg is shown in Fig. 3(b)–(d). The center of mass (CM) of the sole was defined as the vertical point from the ankle joint to the foot. The length from the ankle joint to CM is represented by l. The length from the heel to CM is defined as l _b . The length from the toe to CM is defined as l _f .

The position of the ankle joint is

(15) \begin{equation} x(t) = \left\{ \begin{array}{c} ks,t = kT \\ ks+l_{f}\,(1 - \textrm{cos}\,\theta_{s}) + l\,\textrm{sin}\,\theta_{s},t = kT + d \\ ks + L_{h},t = kT + t_{h} \\ (k + l)s - l_{b}\,(1 - \textrm{cos}\,\theta_{f}) - l\,\textrm{sin}\,\theta_{f},t = (k + 1)T \\ (k + 1)s,t = (k + 1)T + d \end{array} \right. \end{equation}

(16) \begin{equation} z(t) = \left\{ \begin{array}{c} l,t = kT \\ l\,\textrm{cos}\,\theta_{s} + l_{f}\,\textrm{sin}\,\theta_{s},t = kT + d \\ H,t = kT + t_{h} \\ l\,\textrm{cos}\,\theta_{f} + l_{b}\,\textrm{sin}\,\theta_{f},t = (k + 1)T \\ l,t = (k + 1)T + d \end{array} \right. \end{equation}

where L _h is the distance from starting point at time t _h . H is the height of ankle joint.

Since at time kT and time (k+1)T+d, the supporting leg always touches the ground. Therefore, the constraint is:

(17) \begin{equation} \left\{ \begin{array}{c} \theta'(kT) = 0 \\ \theta'((k + 1)T + d) = 0 \end{array} \right. \end{equation}

(18) \begin{equation} \left\{ \begin{array}{c} x'(kT) = 0 \\ x'((k + 1)T + d) = 0 \end{array} \right. \end{equation}

(19) \begin{equation} \left\{ \begin{array}{c} z'(kT) = 0 \\ z'((k + 1)T + d) = 0 \end{array} \right. \end{equation}

3.2. Trajectory generation for hip joint

According to Hypothesis 2, the Z-axis coordinate of the hip joint is a constant, which is the sum of the length of the calf and the length of the thigh and the height of the ankle joint. When H is maximum, the distance traveled of the swinging leg is s/2.

(20) \begin{equation} x(t) = \left\{ \begin{array}{c} ks + s/4,t = kT \\ ks + l_{e},t = kT + d \\ ks + s/2,t = kT + t_{h} \\ ks + 1_{s},t = (k+1)T \\ ks + 3s/4,t = (k+1)T + d \end{array} \right. \end{equation}

where $s/4 < l_{e} < s/2, s/2 < l_{s} < 3s/4, l_{s}$ , represents the distance between the position of hip joint of swinging leg and the supporting leg in the initial stage of swing phase. l _e represents the distance between the position of hip joint of swinging leg and the supporting leg in the ending stage of swing phase.

According to the initial speed and the final speed of the gait cycle, the constraint conditions can be obtained:

(21) \begin{equation} \left\{ \begin{array}{c} x'(kt) = 0 \\ x'((k + 1)T + d) = 0 \end{array} \right. \end{equation}

4.1. Dynamic movement primitives

DMP was flexible representation for motion primitives with good stability [Reference Bottasso, Leonello and Savini20]. Any motion trajectory can be regarded as a combination of a second-order system and a nonlinear function. DMP can be described in the following form. According to the basic principle of dynamic motion primitives, in the process of trajectory learning, this method can obtain a motion trajectory that is the same as the teaching trajectory by providing teaching trajectory information, starting point value y ₀ , and the final point of the trajectory g.

(22) \begin{equation} {\tau}z'_{\!\!t} = \alpha_{z}(\beta_{z}(g - y_{t}) - z_{t}) + R_{t} \end{equation}

(23) \begin{equation} {\tau}y'_{\!\!t} = z_{t} \end{equation}

(24) \begin{equation} {\tau}x'_{\!\!t} = -\alpha_{x}x_{t} \end{equation}

(25) \begin{equation} R_{t} = h_{t}^{T}(\varsigma + \varepsilon_{t}) \end{equation}

where $\tau$ is a scaling factor related to time. y _t and $y'_{\!\!t}$ are joint trajectory position and velocity. x _t and z _t represent the internal state. R _t is nonlinear function which determines the shape of the trajectory. The parameters α _x , α _z , β _z, and $\varsigma$ are scale factors. $\varepsilon_{t}$ is interference term. $h_{t}^{T}$ is a linear function approximator, which contains Gaussian kernels $\psi$ ().

(26) \begin{equation} h_{t} = \frac{\sum\limits^{N}_{k = 1}\psi_{k}x_{t}}{\sum\limits^{N}_{k = 1}\psi_{k}}(g - y_{0}) \end{equation}

(27) \begin{equation} \psi_{k} = \textrm{exp}\left(-\frac{1}{2\sigma^{2}_{k}}(x_{t}-c_{k})^{2}\right) \end{equation}

where y ₀ is the initial position of the trajectory. g is the final point of the trajectory. c _k and σ _k are the center and width of Gauss function. The parameter $\varsigma$ determines the shape of joint trajectory. The parameter $\varsigma$ will be adjusted by RL algorithm to update gait trajectories from y ₀ to target g.

4.2. Stability criteria

Stable walking of the lower limb exoskeleton is the critical research content. Zero Moment Point (ZMP) is the point where the total moment of the human-exoskeleton system is 0 during walking, and its motion trajectory can be used as the basis for stability judgment. In the walking cycle, the smallest polygon formed by the contact points between the sole and the ground is called the supporting polygon. According to the definition of ZMP, if the human-exoskeleton system can walk stably, the ZMP trajectory must always fall within the supporting polygon. To prevent the occurrence of the ZMP trajectory from falling outside the edge of the supporting polygon, a certain distance is usually left at the boundary of the supporting polygon as a stability margin.

To realize the stable walking of the lower limb exoskeleton walking aid robot, this paper abstracts the stability problem as an objective function optimization problem. The objective function is constructed with the ZMP stability margin as a parameter. The shortest distance from ZMP to the edge of the stability margin is used to build the objective function. The coordinate origin is the point where the ankle joint of the supporting foot is projected on the ground. The distance between the toe of the supporting foot and the coordinate origin is D _toe . The distance between the heel of the supporting foot and the coordinate origin is D _heel . The center of stability margin of the supporting foot is x _fc = (D _toe -D _heel )/2. If the exoskeleton system achieves stable walking, ZMP needs to meet $-D_{toe} < x_{zmp} < D_{heel}$ . f(xh) is an index function reflecting the deviation of the exoskeleton from the center of stability margin.

(28) \begin{equation} f(x_{h}) = (x_{zmp} - x_{fc}) \end{equation}

In this paper, a five-segment trajectory planning method is adopted, and the walking stability needs to be satisfied in each segment trajectory. In order to improve the similarity between the generation trajectory and the target trajectory, the objective function J was constructed. The smaller the J, the more stable the walking.

(29) \begin{equation} J = f(x_{h}) + \sum^{3}_{p = 1}M_{c}g_{p}(x_{p}) + \sum^{3}_{q =1}\frac{M_{c}}{2}h_{q}(x_{q}) \end{equation}

(30) \begin{equation} g_{p}(x_{p}) = \left\{\begin{array}{c} 0,x'_{\!\!h}(t_{p}) > 0 \\[5pt] \!\! |x'_{\!\!h}(t_{p})|,x'_{\!\!h}(t_{p}) < 0 \end{array} \right. \end{equation}

(31) \begin{align} h_{q}(x_{q}) = \left\{ \begin{array}{c} 0,-D_{toe} < x_{zmp} < D_{heel} \\[5pt] \!\! |x_{zmp}(t_{q})|,x_{zmp}(t_{q}) < -D_{toe} \\[5pt] \!\!\!\! |x_{zmp}(t_{q})|,x_{zmp}(t_{q}) > D_{heel} \end{array} \right. \end{align}

where M_c is penalty factor, g_p(x_p) and hq(xq) are penalty function, p, q = 1,2,3, t_p and t_q are corresponding time consumption of penalty function.

4.3. Reinforcement learning with trajectory optimization

At present, the method of combining RL algorithm with other technologies has a good research prospect and has been widely used in many fields. Based on the stochastic Hamilton Jacobi Bellman method and path integral strategy learning method, a probabilistic RL method was proposed to solve the statistical reasoning problem from the continuous learning and training process of the sample data. The algorithm mainly uses sample data to train and learn for solving statistical reasoning problems and has the characteristics of simplicity, stability, and strong robustness. In this paper, the RL algorithm was used to update parameter $\varsigma$ , which could continuously update the shape of the trajectory to reach the target trajectory.

The cost function $S(\tau_{k})$ of DMP was defined.

(32) \begin{equation} S(\tau_{k}) = \phi_{t_{N}} + \sum_{J = K}^{N - 1}r_{t_{j}} + \sum_{J = k}^{N - 1}\left\|\frac{y_{t_{j + 1}} - y_{t_{j}}}{d_{t}} - R_{t_{j}}\right\|^{2}_{H_{t_{j}}^{-1}} + \frac{\lambda}{2}\sum_{j = k}^{N - 1}\textrm{log}\|H_{t_{j}}\| \end{equation}

(33) \begin{equation} \phi_{t_{N}} = B_{N}(y_{t} - g)^{T} (y_{t} - g) + R_{N}y^{\prime{T}}_{t} y'_{\!\!t} \end{equation}

(34) \begin{equation} r_{t} = \frac{1}{2}B_{q}y^{\prime\prime{T}}_{t} y''_{\!\!\!t} + \frac{1}{2}R(\varsigma + \varepsilon_{t})^{T} (\varsigma + \varepsilon_{t}) \end{equation}

where φ _tN denotes the terminal cost at time t _N , B _N and R _N are applied for the adjustment of terminal cost, r _t represents the current cost at time t, B _q and R are used to change the current cost. H _t is a scalar, and ${H_t} = {h_t}^{(c)T}{R^{ - 1}}{h_t}^{(c)}$ . $\dfrac{\lambda}{2}\sum\limits_{j = i}^{N - 1} {\log \left| {{H_{tj}}} \right|}$ is a certain variable, the values of all sampling paths are same, so it could be ignored.

Then, the expression of S( $\tau$ _i ) is updated as:

(35) \begin{align} S(\tau_{k}) & = \phi_{t_{N}} + \sum_{N - 1}^{j = k}r_{t_{j}} + \sum_{N - 1}^{j = k}\left\|h^{T}_{t_{j}} (\varsigma_{t_{j}} + \varepsilon_{t_{j}})\right\|^{2}_{H_{t_{j}}^{-1}} = \phi_{t_{N}} + \sum_{j = k}^{N - 1}r_{t_{j}} + \sum_{j = k}^{N - 1}\frac{1}{2}(\varsigma_{t_{j}} + \varepsilon_{t_{j}})^{T}\, h_{t_{j}}\,H_{t_{j}}^{-1}\,h_{t_{j}}^{T}\, (\varsigma_{t_{j}} + \varepsilon_{t_{j}}) \nonumber \\[5pt] & = \phi_{t_{N}} + \sum_{j = k}^{N - 1}r_{t_{j}} + \sum_{j = k}^{N - 1}\frac{1}{2}(\varsigma_{t_{j}} + \varepsilon_{t_{j}})^{T}\, \frac{h_{t_{j}}h_{t_{j}}^{T}}{h_{t_{j}}^{T}R^{-1}h_{t_{j}}}(\varsigma_{t_{j}} + \varepsilon_{t_{j}}) \end{align}

According to the stochastic optimal control theory [Reference Stulp, Theodorou and Schaal21], the result of the path integral optimal problem for DMP can be expressed as:

(36) \begin{equation} \varsigma_{t_{k}} = \int{P}(\tau_{k})\mu_{L}(\tau_{k})d\tau_{k} \end{equation}

where $P({\tau _k}) = \dfrac{\exp ( - \frac{1}{\lambda }S({\tau _k}))}{\int {\exp ( - \frac{1}{\lambda}S({\tau _k}))d{\tau _k}} }$ is probability variable, and the parameter λ was used to adjust the sensitivity of the exponential function. $\mu_{L}(\tau_{k}) = \dfrac{R^{-1}h_{t_{k}}h_{t_{k}}^{T}}{h_{t_{k}}R^{-1}h_{t_{k}}}\varepsilon_{t_{k}}$ is local control variable. And

(37) \begin{equation} S(\tau_{k}) = \phi_{t_{N}} + \sum_{j = k}^{N - 1}r_{t_{j}} + \frac{1}{2}\sum_{j = k}^{N - 1}(\varsigma_{t_{k}} + \varepsilon_{t_{j}})^{T} \times \left(\frac{R^{-1}h_{tj}h_{tj}^{T}}{h_{tj}^{T}R^{-1}h_{tj}}\right)^{T}\,R\left(\frac{R^{-1}h_{tj}h_{tj}^{T}}{h_{tj}^{T}\,R^{-1}h_{tj}}\right)(\varsigma_{t_{k}} + \varepsilon_{t_{j}}) \end{equation}

The stochastic optimal control problem is solved by Eq. (37) in the whole state space. The optimal control $\varsigma_{tk}$ can be obtained by an iterative update procedure. With an initial value $\varsigma$ , and a random parameter $\varsigma + \varepsilon_{t}$ is generated at each time step. With the consideration of Eq. (36), for the sake of the iterative updates, the update rule could be written as:

(38) \begin{equation} \varsigma_{tk}^{new} = \int{P}(\tau_{k})\frac{R^{-1}h_{t_{k}}h_{t_{k}}^{T}(\varsigma + \varepsilon_{t_{k}})}{h_{t_{k}}^{T}\,R^{-1}h_{t_{k}}}d\tau_{k} = \int{P}(\tau_{k})\frac{R^{-1}h_{t_{k}}\,h_{t_{k}}^{T}\varepsilon_{t_{k}}}{h_{t_{k}}^{T}\,R^{-1}h_{t_{k}}}d\tau_{k} + \frac{R^{-1}h_{t_{k}}\,h_{t_{k}}^{T}\varsigma_{t_{k}}}{h_{t_{k}}^{T}\,R^{-1}h_{t_{k}}} = \zeta\!\varsigma_{t_{k}} + \frac{R^{-1}h_{tj}\,h_{tj}^{T}}{h_{tj}^{T}\,R^{-1}h_{tj}}\varsigma \end{equation}

where $\zeta\!{\varsigma _{{t_k}}} = \int {P({\tau _k})} \dfrac{{R^{ - 1}}{h_{{t_k}}}h_{{t_k}}^T{\varepsilon _{{t_k}}}}{h_{{t_k}}^T{R^{ - 1}}{h_{{t_k}}}}d{\tau _k}$ , $\varsigma _{_{{t_k}}}^{new}$ is not time-independent. For each time step t _k , a new optimization parameter $\varsigma _{_{{t_k}}}^{new}$ would be obtained. However, the time-independent parameter ς ^new was needed, the parameter $\varsigma_{tk}^{new}$ was averaged over time t _k .

(39) \begin{equation} \varsigma^{new} = \frac{1}{N}\sum_{t = 0}^{N - 1}\varsigma^{new}_{t_{k}} = \frac{1}{N}\sum_{t = 0}^{N - 1}\zeta\!\varsigma_{t_{k}} + \frac{1}{N}\sum_{t = 0}^{N - 1}\frac{R^{-1}\,h_{t_{k}}\,h_{t_{k}}^{T}}{h_{t_{k}}^{T}\,R^{-1}\,h_{t_{k}}}\varsigma_{t_{k}} \end{equation}

The parameter $\varsigma^{new}$ includes two parts. The first term is the average value of $\zeta\!\varsigma_{t_{k}}$ . It reflects the useful information obtained from the mixed noise information. The second term is the interference term, which causes the loss of parameter $\zeta$ . When the average value of $\zeta\!\varsigma_{t_{k}}$ becomes 0, the convergence will be realized. Therefore, the second term does not need to further update. The second term should be eliminated. The parameter $\varsigma^{new}$ is updated to:

(40) \begin{equation} \varsigma^{new} = \frac{1}{N}\sum_{t = 0}^{N - 1}\zeta\!\varsigma_{t_{k}} + \frac{1}{N}\sum_{t = 0}^{N - 1}\varsigma_{t_{k}} = \frac{1}{N}\sum_{t = 0}^{N - 1}\zeta\!\varsigma_{t_{k}} + \varsigma \end{equation}

where $\varsigma^{new}$ is generated by the averaged value of $\zeta\!\varsigma_{t_{k}}$ , and the current value of $\varsigma$ was added in each iteration.

The update of parameter $\varsigma$ is the most important part of the improved strategy algorithm, and it will affect the shape of the trajectory. The big difference between improved strategy and path integrals is using the combination of RL-based DMP and path integrals, where RL-based DMP can guarantee the stability of the system. The update rule of strategy improvement with path integrals is as follows. The parameters $\varsigma$ is mixed with noise interference; therefore, the noise interference $\varepsilon_{t}$ was added to the parameter $\varsigma$ in DMP algorithm. While learning and updating the parameter $\varsigma$ , the updated result will lead to changes in the trajectory, and the corresponding cost function S() also changes. As the number of updates increases, the cost function S() gradually decreases and stabilizes to minimum value, and finally, the DMP system tends to be a noiseless system.

4.4. Segmented multiple-DMP trajectory learning method

When the DMP parameter values y _o and g are same, the learning trajectory is always a straight line, and the DMP method is invalid. However, the motion trajectory of the joint tends to be a sine function, and it is very likely that the values of y ₀ and g are the same. Therefore, this paper proposed a segmented multiple-DMP trajectory learning method. Combined with five-point segmented gait planning method, the learned trajectory is divided into multiple monotonic trajectories. Because the starting value and target value of each segment of the learning trajectory after segmentation are not equal, the DMP method will not invalid. The specific steps of the segmented multiple-DMP trajectory learning method are as follows: The target trajectory was divided into multiple monotonous trajectories with five key moments as the dividing points. The target trajectory is divided into four segments. To learn each monotonous trajectory, set new starting value and target value, respectively. The target value of the previous segment is equal to the starting value of the following segment of the trajectory. The initial value of each segment of the target trajectory can be expressed as:

(41) \begin{equation} Y = [y_{KT}(k = 0, y_{KT} = y_{0}), y_{KT + d}, y_{KT + t_{h}}, y_{(k + 1)T}, y_{(k + 1)T + d}] \end{equation}

The target value of the target trajectory can be expressed as:

(42) \begin{equation} G = [y_{KT + d}, y_{KT + t_{h}}, y_{(k + 1)T}, y_{(k + 1)T + d},g] \end{equation}

The values of adjacent time points must be different, so the trajectory learning problem of g = y_o is transformed into a multi-segment trajectory learning problem of g ≠ y_o .

In the multi-DMP sequence, the target parameter g of a DMP is used as the starting parameter of the next DMP, and it can affect the cost of subsequent DMP. It is equivalent to using the cost of the current DMP and the cost of the remaining DMP in the sequence to optimize the shape parameter $\varsigma$ .

The cost function of multiple-DMP is defined as

(43) \begin{equation} S\left(\Pi_{c}\right) = \sum_{j = c}^{C}S(\tau_{t_{0}}^{j}) \end{equation}

where $\Pi_{c}$ denotes the c section trajectory in the C sequences. $S(\tau_{t_{0}}^{j})$ denotes the cost function of the j-th trajectory in C sequence at time t ₀ . $s(\Pi_{c})$ denotes the total cost of the current trajectory and all subsequent sequential trajectories.

The parameter update rule of multi-DMP sequence is similar to that of single DMP, and the specific rules are as follows:

(44) \begin{equation} S(\tau_{t_{j, N}}) = \sum_{j = 0}^{N}r_{t_{j}} + \varsigma_{t_{j}}^{T}\,R\varsigma_{t_{j}} \end{equation}

(45) \begin{equation} S\left(\Pi_{c, k}\right) = \sum_{i = c}^{C}S(\tau_{t_{0},k}^{j}) \end{equation}

(46) \begin{equation} P\left(\Pi_{c, k}\right) = \frac{\textrm{exp}\left(\left(-\dfrac{1}{\lambda}S\left(\Pi_{c, k}\right)\right)\right)}{\sum\limits_{l = 1}^{K}\textrm{exp}\left(\left(-\dfrac{1}{\lambda}S\left(\Pi_{c, l}\right)\right)\right)} \end{equation}

(47) \begin{equation} \zeta\!\varsigma_{c} = \sum_{k = 1}^{K}P\left(\Pi_{c, k}\right)\varepsilon_{k}^{\varsigma{c}} \end{equation}

(48) \begin{equation} \varsigma_{c}^{new} = \varsigma_{c} + \zeta\!\varsigma_{c} \end{equation}

where Eq. (44) is the cost function of a certain trajectory; Eq. (45) is the calculation of the cost of all sequences.

5.1. Experimental platform

This paper describes a novel gait trajectory planning method based on multiple-DMP and RL algorithm for lower exoskeleton robot. In Fig. 4, the exoskeleton robot system consists of three levels of architecture, including an upper-level controller, a lower limb exoskeleton robot, and a low-level controller. The upper-level control system consists of the data processor Raspberry Pi, which is used to process data collected from encoders (AD36/1217AF. ORBVB, Hengstler, Germany). The joint movement data of the hip joint were acquired by encoder 1 encoder 2, encoder 3, and encoder 4 together. The low-level controller controls the robot to perform trajectory. The communication mode between the upper-level and the low-level controller is parallel port communication. The low-level controller is used to control the motors through the Controller Area Network (CAN) Fieldbus. The experiment was conducted in Tianjin Key Laboratory for Integrated Design & Online Monitor Center. Two healthy male subjects (subject A and subject B) were recruited for the experiment, with averaged age 24 years old, averaged height 174 cm, averaged weight 71.73 kg. The subjects were instructed to perform normal walking with exoskeleton on the ground. For safety reasons, there are two levels of safety measures taken. Firstly, if the angle of the exoskeleton is greater than the allowable value (The motion range of the hip joint is −38°–50.18°. The motion range of the knee joint is 0°–95.53°), the exoskeleton will be forced to stop moving by software. Besides, the subjects could cut off the power through the emergency switch button if they need to do it at any time, just as shown in Fig. 5.

Figure 4. Prototype of the proposed exoskeleton robot.

Figure 5. Human experiment.

5.2. Parameters setting

The time of dual support phase d is 0.2 s. The one-step length s is 0.6 m. The maximum height of ankle joint H is 0.25 m, and the elapsed time t _h is 0.5 s. The parameters l _b , l, l _s, and l _e are set as 0.06, 0.1, 0.4, and 0.2 m, respectively. The length of calf is 0.3 m. The length of thigh is 0.5 m. The height of hip joint is 0.7 m. The velocity of horizontal hip of subject A and subject B is 0.8 m/s and 0.67 m/s, respectively. The parameters of DMP equation are $\alpha$ _z = 20, $\beta$ _z = 20, τ = 0.2. The parameters of cost function are BN = $10^{-5}$ , R = $10^{-5}$ , B _N = 10, R _N = 25. The parameters of stability criteria are D _toe = 0.15 m, D _heel = 0.1 m, M _c = 300.

5.3. Experimental results

In the experiment, the trajectory learning of all joints of the lower limb exoskeleton is updated at the same time. The number of updates is set to 70. After each update, DMP generates the required joint trajectory. Then, the actual joint trajectory is obtained. The data from the actual joint trajectory are used as the value of the RL algorithm to calculate the corresponding cost, and the shape and target parameters are updated. After the shape and target parameters are updated, the multiple-DMP algorithm is used to generate a new desired joint trajectory for the lower limb exoskeleton.

The experimental results are presented in Figs. 6–8. Figures 6 and 7 show the actual learning trajectories of the two joints in the lower exoskeleton. In order to show the effect of learning clearly, six groups of data are obtained by sampling data every ten times. Each joint trajectory is composed of four sequential segments. Due to the environmental disturbance, the learning trajectory at the initial time has a larger state error. Although the subjects’ movement and speed were inconsistent, the stable convergence of the two tests is achieved through continuous learning. The experimental results show that the reinforcement learning algorithm can well suppress uncertainty and interference. After 30 iterations, with the increase of the number of updates, the amplitude is gradually stable and convergent, which verifies the learning performance of the reinforcement learning algorithm, and shows that the joint trajectory is convergent. The change in the total cost value of all trajectories is shown in Fig. 8. The amplitude tends to stabilize and converge as the number of updates increases, which verifies the validity and superiority of the proposed method. The video capture of the human experiment is shown in Fig. 9. The time of a complete gait cycle of subject A is 1.5 s. The experiment demonstrates that the exoskeleton robot can effectively assist the human body to realize stable walking.

Figure 6. Actual learning trajectories of joints of subject A. (a) Trajectory of hip joint. (b) Trajectory of knee joint.

Figure 7. Actual learning trajectories of joints of subject B. (a) Trajectory of hip joint. (b) Trajectory of knee joint.

Figure 8. Cost during reinforcement learning of subject A.

Figure 9. The video snapshot of subject A’s experiment.

5.4. Stability analysis

Based on the D’Alembert’s principle, when the exoskeleton and human body were regarded as the multi links systems, the ZMPs could be calculated by Eqs. (49) and (50) [Reference Aphiratsakun, Chairungsarpsook and Parnichkun32]. So the ZMP positions of the exoskeleton and human body in the coordinate system could be determined when walking.

(49) \begin{equation} x_{zmp} = \frac{\sum_{i}m_{i}(z''_{\!\!\!i} + G_{g})x_{i} - \sum_{i}m_{i}x''_{\!\!\!i}z_{i}}{\sum_{i}m_{i}(z''_{\!\!\!i} + G_{g})} \end{equation}

(50) \begin{equation} y_{zmp} = \frac{\sum_{i}m_{i}(z''_{\!\!\!i} + G_{g})y_{i} - \sum_{i}m_{i}y''_{\!\!\!i}z_{i}}{\sum_{i}m_{i}(z''_{\!\!\!i} + G_{g})} \end{equation}

where G_g is gravitational acceleration. (x _i , y _i , z _i ) locates the center of mass of link-i, and m _i represents the mass of link i.

Only the Sagittal plane was considered, and the position trajectories of body ZMP and exoskeleton ZMP are shown in Fig. 10. From the graph, we could see the ZMP of exoskeleton could track the wear’s ZMP well. And there is no greater ZMP gap caused by the interference. The whole system could run stably and effectively.

Figure 10. The position trajectories of body ZMP and exoskeleton ZMP.