Multilayer perceptron

ANN is a computational model for surrogate modeling aimed at reducing computational time and resources. It bypasses the requirement of lengthy and tedious analysis and mathematical calculations. NN consists of artificial neurons in multiple layers and maps the input data with the target output. Neurons of each layer are connected with each other and carry weights that are responsible to excite or inhibit the input signals. One of the widely used ANN architecture is the multilayer perceptron (MLP) consisting of an arbitrary number of hidden layers between an input and output layer. MLP is a feed-forward network and is mostly implemented for supervised learning problems. It is used to process the correlation between input and target output using backpropagation. In the forward pass, the input data flows through the intermediate layers to the output layer. Error difference between the target and predicted output is calculated. The partial derivative of the error function with respect to weights and biases are backpropagated through the network. Weights and biases are adjusted in each iteration until the network reaches a state of convergence.

The equation representing the input-output relationship of a generalized MLP is shown in Fig. 1 can be expressed as,

(1)$$y_o = f^{l + 1}\left({\mathop \sum \limits_{\,p = 1}^q w_{o, p}^{( {l + 1} ) } \left({\,f^l\left({\mathop \sum \limits_{k = 1}^s w_{\,p, k}^{( l ) } \left({ \ldots \ldots .\left({\,f^1\left({\mathop \sum \limits_{i = 1}^n w_{\,j, i}^{( 1 ) } x_i + b_j^{( 1 ) } } \right)} \right)\ldots ..} \right) + b_p^{( l ) } } \right)} \right) + b_o^{( {l + 1} ) } } \right)$$

where x_i is the input to i^th neuron and y_o is the output of o^th neuron of MLP; f¹, f^l, and f^l+1represents the activation function of the first hidden layer, l^th hidden layer, and output layer respectively; $w_{j, i}^{( 1 ) }$ denotes the connection weights between i^th input neuron and j^th hidden neuron; $w_{p, k}^{( l ) }$ denotes the connection weights between k^th and p^th hidden neuron; $w_{o, p}^{( {l + 1} ) }$ denotes the connection weights between p^th hidden and o^th output neuron; $b_j^{( 1 ) }$, $b_p^{( l ) }$, and $b_o^{( {l + 1} ) }$ represents the biases of hidden and output layer, respectively.

Fig. 1. Generalized architecture of MLP.

An MLP can be created with multiple hidden layers with a large number of neurons in each layer. However, an increase in the number of hidden layers and neurons may lead to generalization capability loss [Reference Karystinos and Pados31]. Also, few numbers of hidden layers and neurons fail to map complex input-output relationships and leads to underfitting. To achieve faster convergence, proper selection of activation function is necessary. There are several training functions used to train the MLP, which have a significant influence on its performance. Over the years, the trial and error approach is widely used to find the hyperparameters of MLP. In the proposed work, we have explored the capabilities of IPSO to determine appropriate MLP architecture (MLP-IPSO) with better hyperparameter configuration.

Design of MLP-IPSO

PSO is a search-based optimization algorithm inspired by the social behavior of swarms. An initial set of the population with random position and velocity are selected and moved around in a multidimensional search space. The movement of each particle is governed by its local best position to reach the global best position. The proximity of the particle to the global best is measured using a fitness function. The position and velocity of each particle are updated at each iteration until a global optimum solution is achieved. However, if a particle fails to achieve better fitness, then the previous velocity vector affects the next velocity vector. In order to avoid that, velocity mutation is introduced in this IPSO algorithm [Reference Zaharis, Gravas, Yioultsis, Laziridis, Glover, Skeberis and Xenos26].

Let the MLP configuration be represented as λ = (λ _N, λ _H) where λ _N denotes the network architecture and λ _H denotes the hyperparameter configuration respectively. The goal of the optimization is to tune λ _H = (λ _D, λ _C) ∈ K where K is the decision set of all hyperparameters. The hyperparameters to be optimized are the number of hidden layers, the number of neurons in each hidden layer, activation function, and training function. The discrete hyperparameters λ _D include the number of hidden layers and the number of neurons in each hidden layer. Activation function and training function are categorical or non-ordinal hyperparameters λ _C. Every λ _N is trained using corresponding λ _H in order to minimize err _MLP[(λ _N, λ _H)] which is calculated as,

(2)$$err_{MLP}[ {( {\lambda_N, \;\lambda_H} ) } ] = \displaystyle{{\mathop \sum \nolimits_{i = 1}^N {[ {\,f_p( {x_i} ) -f_t( {x_i} ) } ] }^2} \over N}\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;\;$$

where f _p(x _i) is the predicted model output and f _t(x _i) is the target output.

For MLP-IPSO, an initial population of particles with random position and velocity is selected. Each p_i particle has its positionx _i ∈ Sand velocity v _i ∈ S where S is the search space of all the hyperparameters. After population initialization, each particle p_i determines an MLP configuration, p_i = (net_IPSO, H_IPSO) where, net_IPSO and H_IPSO denotes the MLP architecture and hyperparameter configuration, respectively. The corresponding MLP model is then trained using the cross-validation technique [Reference Kwok and Yeung32]. k-fold cross-validation technique is widely applied to divide the dataset into k random independent subsets. k-1 subsets are used for training purposes and one of the k subsets is used for testing. During all the k-fold runs, each subset is selected as hold out at a time. The final performance of the model is then assessed by averaging the k recorded errors. The modeling performance of the MLP is assessed using mean square error (MSE) given by eq. 2.

The current fitness of p_i is compared with the previous best solution of the particle's position pbest and is updated if the current fitness is larger. Similarly, the global best solution of the swarm's position gbest is updated if the current fitness value is larger than the previous gbest. After updating pbest and gbest, the position and velocity of each particle is then modified using,

(3)$$\eqalign{v_i^{iter + 1} & = k\{ {v_i^{iter} + \varphi_1rand^{iter}( {\,pbest_i^{iter} -x_i^{iter} } ) }\cr & \quad+ \varphi_2rand^{iter}( {gbest_i^{iter} -x_i^{iter} } ) \}} $$

(4)$$x_i^{iter + 1} = x_i^{iter} + v_i^{iter + 1} $$

where, x_i and v_i represents n-dimensional position and velocity of i^th particle, respectively; iter denotes the iteration index; rand represents random numbers uniformly distributed in [0,1]; φ₁, φ₂denotes the cognitive and social coefficients respectively, and k is constriction coefficient calculated as,

(5)$$k = \displaystyle{2 \over {\left\vert {2-\varphi -\sqrt {\varphi^2-4\varphi } } \right\vert }}$$

where, φ = φ₁ + φ₂.

At the end of iteration iter, if an i^th particle is not able to improve its fitness, then its velocity components $v_i^{it}$ are mutated by a factor F_g, given as [Reference Zaharis, Gravas, Yioultsis, Laziridis, Glover, Skeberis and Xenos26],

(6)$$F_g = ( {0.6 + 0.1g} ) ( {2rand-1} ) $$

where g denotes the number of iterations for a particle with no fitness improvement. In this case, the velocity of the particle is updated using [Reference Zaharis, Gravas, Yioultsis, Laziridis, Glover, Skeberis and Xenos26],

(7)$$\eqalign{v_i^{iter + 1} & = k\{ {F_gv_i^{iter} + \varphi_1rand^{iter}( {\,pbest_i^{iter} -x_i^{iter} } ) }\cr& \quad+ \varphi_2rand^{iter}( {gbest_i^{iter} -x_i^{iter} } ) \}} $$

The process is continued until the maximum numbers of iterations are over. The search space of all the hyperparameters to be optimized by IPSO is listed in Table 2. The optimal solutions of the algorithm obtained are considered as the tuned hyperparameters of the MLP. Figure 2 represents the flowchart of the proposed MLP-IPSO approach.

Table 2. Search space for hyperparameter optimization

Fig. 2. Flowchart of MLP-IPSO.

EBG unit cells

Two symmetrical defected spiral lines, and two L-shaped defected lines with two defected spiral lines, are employed to design EBG ₁. EBG ₂ is designed using four symmetrical spiral lines connected from the center. Both the EBG unit cells, shown in Fig. 3, are designed on FR4 dielectric substrate with a thickness of 1.6 mm using Ansys HFSS. The geometrical parameters of EBG ₁ (a ₁, b ₁, c ₁, d ₁, e ₁, a ₂, b ₂, c ₂, d ₂, e ₂, l) and height of the dielectric substrate (h) are varied to obtain the resonance frequency and its corresponding |S ₁₁|. Extensive parametric analysis has been performed on each parameter to obtain the training and testing datasets. Similarly for EBG ₂, different geometrical parameters of the spiral line (a, b ₁, b ₂, c, d, e, f, g), and height of the dielectric substrate (h) are varied using parametric analysis to obtain the datasets. Table 3 lists out the sampling procedure for generating the datasets. A total of 1220 and 1098 datasets are obtained for EBG ₁ and EBG ₂, respectively. The simulated performance of both EBG ₁ and EBG ₂ is shown in Fig. 4. EBG ₁ gives dual resonance at 5.7 and 11.1 GHz, respectively whereas EBG ₂ is resonating at 7.5 GHz. The Brillouin-zone based dispersion diagrams of EBG ₁ and EBG ₂ are shown in Fig. 5. For EBG ₁, two bandgaps are exhibited between 5.66–5.85 GHz and 10.8–11.5 GHz, and the bandgap for EBG ₂ falls between 7.21 and 7.79 GHz. A comparative study between the EBG structures and the existing literature in terms of types of EBG design, size, and bandgaps is conducted as listed in Table 4.

Fig. 3. EBG Unit Cells (a) EBG ₁, (b) EBG _2.

Fig. 4. Simulated performance of EBG structures.

Fig. 5. Dispersion diagram of (a) EBG ₁, (b) EBG _2.

Table 3. Sampling approach for dataset generation of EBG structures

Table 4. Comparison of EBG structures with existing literature

For modeling EBG ₁, and EBG ₂, two NN models, NN ₁, and NN ₂ are proposed. NN ₁ has been implemented for obtaining 12-dimensional response [G ₁] of EBG ₁ for the 4-dimensional input [I ₁] where [G ₁] = [a ₁, b ₁, c ₁, d ₁, e ₁, a ₂, b ₂, c ₂, d ₂, e ₂, l, h] and [I ₁] = [f_{r 1}, |S₁₁|₁, f_{r 2}, |S₁₁|₂]. NN ₂ is proposed for EBG ₂, where 2-dimensional excitation [I ₂] is processed for getting 9-dimensional response [G ₂]. Here, [I ₂] = [f_r, |S₁₁|] and [G ₂] = [a, b ₁, b ₂, c, d, e, f, g, h].

EBG loaded UWB antenna

A conventional UWB antenna with a rectangular radiating surface and a partial ground plane is taken as the reference antenna. The simulated performance of the antenna is shown in Fig. 6 and it can be observed that the antenna achieved an impedance bandwidth from 2.9 to 10.5 GHz. For obtaining triple-band notch characteristics, three modified U-shaped slots as shown in Fig. 7 are incorporated onto the radiating surface. The effect of notch frequencies is validated by individually simulating each modified U-shaped slot. The slot positions are optimized in such a way that it rejects three interfering bands intended for ISM, radar surveillance, and WiMAX applications.

Fig. 6. Simulated performance.

Fig. 7. Proposed EBG loaded antenna design.

To obtain the datasets of the reference UWB antenna, a parametric analysis is performed by varying four geometrical variables, L_P, W_P, W_f, and L_g. Slot ₁ is introduced on the radiator and parametric analysis is performed by varying L ₁ and W ₁. Slot ₂ is then incorporated on the radiating surface keeping Slot ₁ at its optimized position. Parametric analysis is done on Slot ₂ by varying L ₂, and W ₂. Finally, Slot ₃ is placed keeping Slot ₁ and Slot ₂ at their optimized place and parametric analysis is performed on L ₃, and W ₃. For validating the effectiveness of EBG structures in rejecting frequency bands, the two EBG unit cells are incorporated onto the slotted antenna. As shown in Fig. 7, EBG ₁ is etched from the radiator at a distance of d ₁ and d ₂ from Slot ₁. The three modified U-shaped slots are kept at their optimized position and d ₁ and d ₂ are varied for obtaining the datasets. Similarly, EBG ₂ is introduced at a distance of p ₁ from the feed line as depicted in Fig. 7. Parametric variation of p ₁ and p ₂ is performed by keeping other geometrical variables at their optimized values.

Figure 7 is the final proposed antenna geometry, with two meander line EBG cells, EBG ₁ and EBG ₂, which produces penta notch-band characteristics. The sampling method to obtain the datasets is listed in Table 5. Figure 6 shows that the antenna satisfies the bandwidth requirement of UWB applications from f_{c 1} = 2 GHz to f_{c 2} = 10.74 GHz and the notches are achieved at 2.33, 2.83, 3.35, 3.87, and 5.87 GHz. The prototype of the optimized antenna geometry is fabricated using FR4 substrate sheet of 1.6 mm thickness for validating its performance. The fabricated prototype and the far-field measurement setup are shown in Fig. 8. The measured |S ₁₁| result of the fabricated prototype obtained using a vector network analyzer (VNA) is compared with the simulated results in Fig. 9. A good agreement between the simulated and measured performance characteristics is achieved. The slight deviations might be due to fabrication tolerances, soldering effect, or human error during the process of fabrication and/or measurement. The simulated and measured normalized radiation patterns for the antenna in H-plane and E-plane are plotted in Fig. 10. The antenna achieves an omnidirectional radiation pattern in the H-plane, and a dipole-like radiation pattern in the E-plane at the working frequencies of 4.2, 6.9, and 9.5 GHz. Figure 11 depicts that the antenna achieved stable group delay and relatively flat gain. The sudden transitions validate the radiation prohibition at the notch bands. For validating the creation of notched bands by etching slots and loading EBG structures, the surface current distribution at five notch frequencies of the antenna is plotted in Fig. 12. From the figure, it is clear that the current distribution is strongly confined around the respective slots at 2.33, 2.83, and 3.35 GHz. Besides, at 3.87, and 5.87 GHz, the surface current remains concentrated near the EBG ₁ and EBG ₂, respectively.

Fig. 8. Fabricated prototype and the measurement setup in anechoic chamber.

Fig. 9. Simulated and measured |S ₁₁| comparison.

Fig. 10. Radiation pattern comparisons (a) H-Plane, (b) E-Plane.

Fig. 11. Simulated group delay and gain.

Fig. 12. Surface current distribution.

Table 5. Sampling approach for dataset generation of EBG loaded UWB antenna

A comparative study between the presented geometry and existing literature is conducted in terms of the performance characteristics viz. impedance bandwidth, the number of notches obtained, rejected frequency, notch bands, and antenna size, as listed in Table 6. Although, [Reference Rahman, Jahromi, Mirjavadi and Hamouda33] and [Reference Yadav, Abegaonkar, Koul, Tiwari and Bhatnagar3] have proposed a more compact structure, it has achieved only single and dual notches, respectively. The suggested geometry is found to be more miniaturized than [Reference Ghahremani, Ghobadi, Nourinia, Ellis, Alizadeh and Mohammadi2, Reference Lee, Yang and Cho4, Reference Li, Hei, Feng and Shi34–Reference Iqbal, Smida, Mallat, Islam and Kim36], and has also obtained notch characteristics at five frequency bands.

Table 6. Comparison of optimized design with existing literature

For modeling the proposed antenna structure, NN ₃ has been presented to predict 14-dimensional output [G ₃] from 7-dimensional input [I ₃] where [G ₃] = [L _P, W _P, W _f, L _g, L ₁, W ₁, L ₂, W ₂, L ₃, W ₃, d ₁, d ₂, p ₁ and p ₂] and [I₃] = [f_{c 1}, f_{c 2}, f_{n 1}, f_{n 2}, f_{n 3}, f_{n 4}, f_{n 5}].

Computed performance

To verify the effectiveness of the MLP-IPSO approach mentioned in Section II, the performance of the three proposed models is evaluated in this section. The optimal hyperparameters obtained after the models are tuned using IPSO are listed in Table 7. It is observed that two hidden layers are proved to be optimum for all the models. The optimal activation function is tansig for NN ₁, and NN ₃, and logsig for NN ₂. Trainbr is proved to be the optimum training function for NN ₁, and NN ₂, whereas for NN ₃, trainlm is the optimum hyperparameter.

Table 7. Optimized hyperparameter values.

The convergence of MLP-IPSO is shown in Fig. 13. The MSE calculated using eq. 2 is plotted against the number of iterations measured on the entire training dataset, to obtain the convergence characteristics. The effectiveness of the algorithm for NN ₁, NN ₂, and NN ₃ is clearly depicted in Fig. 13, as the curve consistently converges toward the minimum. Tables 8 and 9 gives a comparison between the simulated and computed performance of all the models. It is observed from Table 8, that an error percentage of less than 1.5% is achieved for all the output parameters of EBG ₁ and EBG ₂, respectively. For the EBG loaded antenna, the error is below 1.1% for all the output parameters as shown in Table 9.

Fig. 13. Convergence curve.

Table 8. Performance comparison of EBG ₁ and EBG ₂

Table 9. Performance comparison of EBG loaded UWB antenna

For a comparative analysis, the three MLP models are also tuned using other evolutionary algorithms, viz. hybrid PSO (HPSO) [Reference Jin and Samii37], conventional PSO [Reference Sengupta, Basak and Peters38], and GA [Reference Mirjalili, Song Dong, Sadiq, Faris, Mirjalili, Song Dong and Lewis39]. The parameters of the algorithms mentioned in Table 10 are selected based on an initial parametric analysis performed with each algorithm individually. The performance of all the models has been analyzed in terms of MSE, mean absolute percentage deviation (MAPD), and coefficient of determination (R ²). The training MSE of three MLP models tuned using IPSO, HPSO, PSO, and GA are depicted in Fig. 14. Training MSE of NN ₁ and NN ₃ is the largest when tuned using PSO followed by GA, HPSO, and IPSO. NN ₂ when tuned using HPSO and PSO has achieved almost equal training MSE. It is observed from the figure, for all the MLP models, IPSO has achieved the least MSE compared to other algorithms.

Fig. 14. MSE performance.

Table 10. Settings of the evolutionary algorithms

The accuracy of the models is also evaluated using MAPD and R ²; mathematical formulae for which are given as,

(8)$$MAPD = \displaystyle{{100\% } \over N}\mathop \sum \limits_{i = 1}^N \displaystyle{{\,f_p( {x_i} ) -f_t( {x_i} ) } \over {\,f_t( {x_i} ) }}$$

(9)$$R^2 = \left({\displaystyle{{N\mathop \sum \nolimits_{i = 1}^N f_p( {x_i} ) f_t( {x_i} ) -\mathop \sum \nolimits_{i = 1}^N f_p( {x_i} ) \mathop \sum \nolimits_{i = 1}^N f_t( {x_i} ) } \over {\sqrt {\left[{N\mathop \sum \nolimits_{i = 1}^N {( {\,f_p( {x_i} ) } ) }^2-{\left({\mathop \sum \nolimits_{i = 1}^N f_p( {x_i} ) } \right)}^2} \right]\left[{N\mathop \sum \nolimits_{i = 1}^N {( {\,f_t( {x_i} ) } ) }^2-{\left({\mathop \sum \nolimits_{i = 1}^N f_t( {x_i} ) } \right)}^2} \right]} }}} \right)^2$$

MAPD measures prediction accuracy and gives relative deviation from true values whereas R ² assesses the ability of the model to adequately fit the data. As shown in Table 11, training and testing MAPD of MLP-IPSO models are less than that of MLP-HPSO, MLP-PSO, and MLP-GA models. MAPD of NN ₁, and NN ₂ is the worst when the hyperparameters are tuned using PSO, whereas, in the case of NN ₁, training MAPD is the least when IPSO is used for tuning the hyperparameters, followed by HPSO, PSO, and GA. R ² of all the models, listed in Table 11, depicts a stronger correlation between the target and predicted output. Although R ² of all the models is greater than 0.97, training and testing R ² are better for MLP-IPSO models as compared to MLP-HPSO, MLP-PSO, and MLP-GA models.

Table 11. Performance metrics comparison