2.1 Trust and reputation

Selecting the most appropriate trust and reputation model depends on what information is available, including representations of outcomes, task decomposition, agent connectivity and social connections amongst others. Additionally, some algorithms require information about the environment either in advance or at run-time, to accurately tune the parameters of the model. RaPTaR is a complimentary method to existing trust and reputation models, which improves their estimates by managing agents’ experience data to adapt to dynamic behaviours. Therefore, we include a brief review of relevant trust and reputation literature to understand the application of RaPTaR, the models we evaluate it with and why they are limited in coping with dynamic behaviours.

An agent assessing a potential interaction partner with a trust algorithm uses their direct past experiences with them to estimate how they will behave in the future (Keung & Griffiths, Reference Griffiths2010). Reputation mechanisms are similar, but additionally collect, aggregate and distribute feedback about agents’ past behaviour from witnesses (Resnick et al., Reference Resnick, Kuwabara, Zeckhauser and Friedman2000). In the remainder of this paper, we refer to trust and reputation interchangeably as we use aggregated witness reports in our trust assessments. Trust and reputation literature is extensive and addresses a variety of problems, which arise in trust and reputation assessment. Biased reputation sources can be statistically altered to account for liars or perception bias using models such as FIRE, TRAVOS and BLADE (Huynh & Jennings, Reference Huynh and Jennings2004; Regan et al., Reference Regan, Poupart and Cohen2006; Teacy et al., Reference Teacy, Patel, Jennings and Luck2006). Similarly, HABIT is a Bayesian inference system, which addresses the need for a general behaviour representation and improves computational efficiency (Teacy et al., Reference Teacy, Luck, Rogers and Jennings2012). Stereotyping techniques improve assessments of newcomers or agents for whom there is no past experience (Burnett et al., Reference Burnett, Norman and Sycara2010; Sensoy et al., Reference Sensoy, Yilmaz and Norman2016). More recently, trust assessment has been viewed as a machine-learning problem with a trust model including a learning component and a predictive component (Lu & Lu, Reference Lu and Lu2017; Taylor et al., Reference Taylor, Barakat, Miles and Griffiths2018).

There are no existing reputation systems that focus on statistically identifying whether witness reports are representative of the target agent’s current behaviour given that it may have changed. The main contribution of RaPTaR is to provide this feature to existing trust and reputation models, and in this paper we select Beta Reputation System (BRS) (Jøsang & Ismail, Reference Jøsang and Ismail2002), Dirichelet Reputation System (DRS) (Josang & Haller, Reference Josang and Haller2007) and TRAVOS (Teacy et al., Reference Teacy, Patel, Jennings and Luck2006) as representative trust mechanisms.

A trustor agent, tr, using BRS to assess a trustee, te, can perceive interaction outcomes as good or bad and collects the sum of good and bad experiences with te, $r_{te}$ and $s_{te}$ , respectively. All the past interactions tr has had with trustees are stored in a history, $\mathcal{O}_{tr}$ . These are the input parameters to a beta probability density function (PDF), whose expected value is the estimate of the trustee’s behaviour, trust(te). An agent can aggregate witness reports from available agents to improve the accuracy of the assessment. For now, we assume that these values are relevant data about the current behaviour of te. The belief, $b_{te}$ , in te is their expected behaviour based solely on their previous interactions, as calculated below:

(1) \begin{equation}b_{te} = \frac{r_{te}}{r_{te} + s_{te} + 2}\end{equation}

BRS weights older interactions less according to a forgetting factor, described below in Section 2.3. To account for uncertainty in this belief, an a priori, a, is weighted by an uncertainty factor, $u_{te}$ :

(2) \begin{equation}u_{te} = \frac{2}{r_{te} + s_{te} + 2}\end{equation}

which decreases as more interaction experiences are collected, signifying a higher confidence in the belief factor. The default value for the a priori, a, is 0.5, which assumes that an agent is neither good nor bad, unless the default can be replaced by the output of another model:

(3) \begin{equation}a_{te} =\begin{cases} x, & \text{if a model is available} \\[5pt] 0.5 & \text{otherwise}\end{cases}\\\end{equation}

This is discussed further in Section 2.2. The overall expected behaviour according to BRS, trust(te), is calculated using subjective logic, namely:

(4) \begin{equation}trust(te) = b_{te} + u_{te} \times a_{te}\end{equation}

The values of $r_{te}$ and $s_{te}$ are an aggregation of good and bad interactions with te from all available trustors known as opinion providers, $op \in \mathcal{A}_{tr}$ , defined as:

(5) \begin{equation}r_{te} = \sum_{op}^{\mathcal{A}_{tr}}{r_{te}^{op}}, \quad s_{te} = \sum_{op}^{\mathcal{A}_{tr}}{s_{te}^{op}}\end{equation}

In BRS, the witness reports are assumed to be honest, and therefore both direct and indirect experiences are weighted equally (Burnett et al., Reference Burnett, Norman and Sycara2010; Jøsang & Ismail, Reference Jøsang and Ismail2002). If it is not possible to assume that reputation providers will be honest or unbiased, then TRAVOS can be used. TRAVOS expands on BRS, by altering witness reports to weight opinions which have statistically proven to align with the trustor’s true experiences in the past. The accuracy of a witness is calculated from tr’s perspective by comparing the beta distribution of the opinion provider’s reports about any te, and the beta distribution of the actual outcomes tr ultimately received. However, this is computationally expensive because it requires collating witness reports from all trustors whenever reputation information is needed.

DRS is a generalization of BRS, which divides interaction outcomes into k possible values. While BRS is a prominent, mathematically rigorous model, the limitation of only two possible outcomes from an interaction can be unrealistic. For example, if $k=5$ , interactions might fall into one of the following categories: 1-bad, 2-mediocre, 3-average, 4-good or 5-excellent (Josang & Haller, Reference Josang and Haller2007). When tr calculates the reputation of te at time t using DRS, the set of tr’s past interactions with te, $\mathcal{O}^{te}_{tr}\subset \mathcal{O}_{tr}$ , are each labelled with one of k values to become the input parameters to a multivariate Dirichlet PDF. From a Dirichlet distribution, a reputation value can be calculated using point estimate representation, which is easily computable and human understandable. Generalizing storing interaction outcomes from BRS, agents have an evidence vector, $\overrightarrow{R}_{te} = (R_{te}(i) | i = 1...k)$ , where $R_{te}(i)$ indicates the number of interactions from $\mathcal{O}^{te}_{tr}$ that were rated in the $i^{th}$ category. Each category i is given a point value, $v(i) =\frac{i-1}{k-1}$ , such that the values are evenly distributed in the range [0,1]. An overall reputation value is a weighted average calculated as $reputation(te) = \sum_{i=1}^k v(i)S(i)$ given that:

(6) \begin{equation}\overrightarrow{S}_{te} : ( S_{te}(i) = \frac{R_{te}(i) + Ca(i)}{C + \sum^k_{j=1}R_{te}(j)};|i=1,...k)\\\end{equation}

where each element, $S_{te}(i)$ , in the vector $\overrightarrow{S}_{te}$ describes the multinomial probability reputation of the $i^{th}$ element given the aggregate reports in $\overrightarrow{R}_{te}$ , and the value of C weights the importance of the a priori. The literature this work is proposed in, recommend a value of $C=2$ , but optionally a higher value for C will lower the importance of the a priori (Josang & Haller, Reference Josang and Haller2007). The a priori, a(i), is the prior probability of the $i^{th}$ category. Without any other information, all k intervals are given an equal probability of the a priori. Stereotype techniques can provide a more informative a priori value, $ a \in [0,1]$ for an agent, which is substituted into DRS by assigning $a(i)=1$ to the $i^{th}$ category that a falls into, i.e. $\lfloor a \times k\rfloor$ , and all other $a(i)=0$ :

(7) \begin{equation}a(i) =\begin{cases} a(i) \in \overrightarrow{a} \text{ if \emph{a priori} model available} \\[5pt] \frac{1}{k} \text{ otherwise}\end{cases}\end{equation}

A disadvantage of using point estimates is that no standard deviation is calculated, and so an agent who behaves well half the time but badly otherwise, will have an equally good reputation compared to an agent who consistently performs averagely.

2.2 Groups

Trust and reputation models are data-based approaches that may perform poorly when assessing a new agent for which there are no previous interactions. This is a reoccurring problem in dynamic groups where agents may have had past interactions with other agents who were previously in their neighbourhood, but none with agents who they are now connected to. One possible solution is to provide the trust algorithm with an a priori value for behaviour, until the agent can collect direct experiences to learn from Burnett et al., (Reference Burnett, Norman and Sycara2010), Teacy et al., (Reference Teacy, Luck, Rogers and Jennings2012), and Nguyen & Bai (Reference Nguyen and Bai2018). This may be a base value representing average performance, or an inferred value based on interactions with other agents. Grouping agents together by how they appear and behave was initially proposed in the Swift Trust algorithm (Debra et al., Reference Debra, Weick and Kramer1995). This method is known as stereotypes and uses the assumption that agents who are observably similar may act similarly. For example, a decision tree can be trained on past experience data, learning correlations between agents’ observable features and the trust they have in that agent (Burnett et al., Reference Burnett, Norman and Sycara2010). The a priroi is given less precedence as direct experiences become available. HABIT uses the average value of the first interactions with other agents in the group the trustee is associated with. This represents the notion that if previously interacting with an agent in that group for the first time was typically bad, the trustee might also be bad. StereoTrust assigns agents an a priori assessment of their behaviour based on the trust they have in other agents who are observably similar, weighted by the extent of that similarity (Liu et al., Reference Liu, Datta, Rzadca and Lim2009). If agents of a group can be assumed to have the same static behaviour, we extend this to assume their behaviour at any time point is similar, given that it can change.

2.3 Dynamic behaviour

Dynamic behaviour is a challenge to trust assessment because behaviour can change at a variety of speeds and times for the agents in an MAS, rendering traditional methods of forgetting old data at a constant rate ineffective. Despite this limitation, prominent trust and reputation models still rely on techniques such as sliding windows and forgetting factors (Huynh & Jennings, Reference Huynh and Jennings2004; Jøsang & Ismail, 2002; Liang & Shi, Reference Liang and Shi2005; Regan et al., Reference Regan, Poupart and Cohen2006; Teacy et al., Reference Teacy, Patel, Jennings and Luck2006). While other literature mentions the importance of coping with dynamic behaviour, it remains an open challenge (Anders et al., Reference Anders, Seebach, Steghöfer, Reif, André, Hähner, Müller-Schloer and Ungerer2016).

A sliding window of size n retains the most recent n experiences. The intuition is that the most recent n instances capture an agent’s current behaviour, and older interaction records that are no longer relevant are deleted. For a new instance, $o^t$ , at time t and a window, W, of size n the window is updated as follows.

(8) \begin{equation}W \gets (W/W[n]) + o^t\end{equation}

A forgetting factor, also referred to as a longevity factor, forgets at a constant rate with all instances being retained and more recent interactions weighted higher. An interaction from time t at the current time, t’, which can be used in trust assessment is weighted as:

(9) \begin{equation}o^{t,t^{\prime}} = \lambda^m \times o^t\end{equation}

where m is the elapsed time since $o^t$ was recorded, that is, $m=t'-t$ . Forgetting factors can be implemented recursively by storing one value which is updated over time instead of storing all values, reweighting and collating them at every time point. This improves on both time and space efficiency as older interactions do not need to be searched or saved. However, not saving old data loses potentially useful information, especially about behaviour change over time.

Unfortunately, several issues impact on the generality of using sliding windows or forgetting factors to aggregate information. First, choosing the window size or the rate of forgetting is problematic. If an agent forgets old instances too quickly, they will lose relevant data that could help them make more accurate calculations. Conversely, if too much information is retained, then agents will make assessments based on data that no longer represents current behaviours. The optimal values will depend on the application. Second, the optimal values for window sizes or forgetting factors can vary over time. For example, behaviours may change frequently for a period and then remain static, and this would require using very recent data initially and then a larger window size. Third, agents may not change their behaviour at the same rates, and so a global window size or forgetting factor will not suit all agents. Finally, sliding windows and forgetting factors are only effective in coping with gradual change rather than sudden changes, which may occur at different times for different agents.

Existing MAS literature outside of the scope of trust and reputation has looked at identifying the best behaviour policy to use from a set of policies, by identifying other agents’ behaviours and responding to that. Part of this task involves identifying when other agents have changed their own policy (Hernandez-Leal et al., Reference Hernandez-Leal, Taylor, Rosman, Enrique Sucar and De Cote2016). One reason we do not compare against policy change literature is in our context, agent behaviours are not a series of static policies that are swapped between, but rather are continuous values that can change gradually or suddenly. Agents need to learn behaviours through outcome rewards and cannot identify discrete action choices of other agents immediately. Identifying agent behaviour changes through analyzing a continuous variable is a different problem to policy detection, and this depends on the information available in the environment (Hernandez-Leal et al., Reference Hernandez-Leal, Zhan, Taylor, Sucar and de Cote2017). Finally, as discussed with all existing techniques on estimating agent behaviour, work on reacting to policy changes also does not discuss the probability of an upcoming change.

In this paper, we propose a method that uses concept drift techniques to identify changes in behaviour. Machine learning models find correlations between a feature set, X, and a class value, Y, typically assuming that the joint probability of the features and the target remains static over time, that is, $P_t(XY) = P_{t^{\prime}}(XY)$ for all times t and t’ (Webb et al., Reference Webb, Hyde, Cao, Nguyen and Petitjean2016). Concept drift techniques aim to detect when such relationships change over time, meaning $P_t(Y|X) \neq P_{t^{\prime}}(Y|X)$ (Tsymbal, Reference Tsymbal2004). In stereotyping, agents are grouped based on their observable features, and the correlation between those features and agent behaviour is learned. In this paper, we address the problems that arise from agent behaviour changing over time meaning that the correlation between features and behaviour may change.

A concept is a learned correlation between the features and the target, or class value, which in our context is the trust value. If the correlation between a set of feature values and a class value instantly changes to a new class value, then sudden drift has occurred. A slow progression of one concept becoming less prevalent while another becomes more prevalent is known as gradual drift. Gradual drift is harder to detect, especially if the differences between the two concepts are not empirically large but are important (Hoens et al., Reference Hoens, Polikar and Chawla2012). Sliding windows and forgetting factors will have an optimal size or value to handle only one speed of change; however, agents may change behaviour at different times and rates and a single window size or forgetting factor may be insufficient. The method proposed in this paper handles agent behaviours changing at different speeds or at different times without requiring parameter tuning.

In our approach, we draw upon concept drift techniques that can identify the point of change in the data, allowing the agent to update its interaction history. The Adaptive Windowing (AdWin) model uses a variable window size which expands with new, incoming data if that data is statistically assessed to be drawn from the same distribution as the rest of the data in the window (Bifet & Gavaldà, Reference Bifet and Gavaldà2007). If there is a split of the window such that it is believed the two subsets come from different distributions, then change has been detected and data from the oldest subset is forgotten, which is synonymous with shrinking the window. The method we propose in this paper uses a similar adaptive window as part of its reactive component. Once AdWin deems a concept to have ended, the data from that concept is forgotten forever. RePro retains such information with the aim of improving predictions in an environment where concepts might reoccur (Wu & Zhu, 2005). Once a concept has changed, RePro remembers the concept and records the transition to it from the previous concept. If the concept reoccurs in future, it can be proactively predicted given the concept which came before it, without being relearned from scratch, therefore saving time and improving predictive accuracy. RePro was designed for categorical predictions; however, we present a simple adaptation for continuous predictions. RePro can also only predict one concept at a time, as the proactive mode selects the concept that most frequently follows the current concept and there is no consideration of unknown concepts. The proactive mode requires a trigger, such as detecting a small amount of error, to determine whether a different concept has taken over. Work on detecting concept drift in the error of agent behaviour estimates exists (Hernandez-Leal et al., Reference Hernandez-Leal, Zhan, Taylor, Sucar and de Cote2017); however, in the context of trust and reputation, estimates of behaviour can be highly inaccurate but not affect the ability to choose good partners. We find it more prudent to analyze interaction outcomes for concept drift, which have not been biased by agents’ potentially inaccurate trust estimates, which allows RaPTaR to detect changes effectively regardless of which trust algorithm is being used.

3.1 Learning component

The learning component comprises two parts: the first detects changes in agent behaviour using interaction outcomes and the second learns patterns in changes of behaviour.

RaPTaR maintains a window, W, of outcomes from interacting with the agents of a group, G, such that after an interaction the outcome, $o^t$ , is appended to the end of W. For every split of W into $W_0$ and $W_1$ , we perform the K-S test to determine whether the null hypothesis, ‘the data in the two sets is drawn from the same distribution’ can be rejected, that is, there has been a change in the agents’ behaviour. The K-S test is appropriate because the data is continuous (Press et al., Reference Press, Teukolsky, Vetterling and Flannery2007). There is only one parameter in the K-S test, $\alpha \in [0,1]$ , where the confidence in the decision of accepting or rejecting the null hypothesis is $1-\alpha$ . Therefore, for a high level of confidence in this decision, $\alpha$ should be a low value. We show in Section 5, that in our results the efficacy of RaPTaR does not fluctuate with differing values for $\alpha$ .

In the original AdWin algorithm, W is iteratively split into all combinations of $W_0$ and $W_1$ starting from isolating the oldest instance and incrementally moving the oldest element of $W_1$ into $W_0$ , until either change is detected or all the instances in W are assumed to be from the same distribution. We identified that there is a performance difference in the AdWin algorithm if W is split into $W_0$ and $W_1$ in reverse order, such that the most recent instance is isolated first into $W_1$ with all other instances in $W_0$ , then iteratively moving the last instance of $W_0$ into $W_1$ . To keep the retained data as relevant as possible, conducting the split starting at the most recent instance is more effective. This is because if there is a gradual change in the distribution over data, which does not have one clearly defined time point of change, the null hypothesis could be rejected at any of several consecutive split points. By identifying the most recent split point, the most relevant data is retained in $W_1$ . Alternatively, starting the tests from older instances will result in the change being detected closer to the start of the window and thus retaining more, possibly irrelevant, data in $W_1$ . Algorithm 1 describes how RaPTaR learns behavioural patterns, splitting W into $W_0$ and $W_1$ by starting to check for behaviour change at the end and then working backwards.

If there exists a split of W into $W_0$ and $W_1$ such that a change in behaviour is detected, RaPTaR stores information about the change for future predictions about behaviour. When a change is detected, the instances in $W_0$ can be learnt as a concept. The currently active concept, $c_c$ , which best estimates $W_1$ , is still active and may still change, so we cannot record a transition to it yet. Therefore, transitions are recorded between the two concepts that were active prior to $c_c$ . Concept $c_y$ represents the behaviour in $W_0$ and $c_x$ was the concept learned before $c_y$ , learned from data which has since been removed from W. Concept $c_x$ is initialized to the unknown concept, and then updated at the end of this process as $c_x \gets c_y$ , ready to record the next transition. The form of the transition matrix is depicted in Table 2. The first column of TM, index 0, is for the unknown concept, to show how frequently other concepts are followed by a new concept. When a change in behaviour is detected, as described above, the concept $c_y$ that describes the behaviour in $W_0$ needs to be identified as either a concept seen before or be learnt as a new concept. The value of all the known concepts in $\overrightarrow{C}$ is compared to the mean of the instances in $W_0$ , $\mu_{W_0}$ , and if there exists a concept $c \in \overrightarrow{C}$ such that $|\mu_{W_0} - \mu_{c}| < cet$ , where cet is the conceptual equivalence threshold, then $c_y \gets c$ . Storing mean values is a simple approach, and future work might consider storing more complex agent policies. If no concept is equivalent, a new concept is learnt with the value $\mu_{W_0}$ . If the length of $W_0$ is less than a stable learning size, s, then it is not considered substantial to record a transition for. The previous concept, $c_x$ , is remembered even though the data for it has since been deleted, and we can now record the transition between $c_x$ and the newly learned $c_y$ . The list, $L_{[c_x,c_y]}$ , records the length of time spent at $c_x$ , $tl_{cx}$ , before moving to $c_y$ , for every occurrence of the transition. The time at $c_x$ is initialized to 0 for the first recording. The reactive and learning component is summarized in Algorithm 1.

Table 2 Transition Matrix

Algorithm 1 Learning Concepts by Updating TM

3.2 Predictive component

For any trust and reputation model that accepts an a priori trust estimate, RaPTaR calculates an expected utility based on the value of each concept, $\mu_c \in \overrightarrow{C}$ , in the concept history for a group, and the probability it may currently be active, p(c). The expected utility draws upon the information recorded in TM by the learning component and is calculated as:

(10) \begin{align}EU &= \forall_{c\in\overrightarrow{C}} \quad \mu_c \times p(c)\end{align}

First, tr identifies if any of the known concepts, $c \in \overrightarrow{C}$ , could be the currently active concept, $c_c$ , by evaluating the difference between the concept values, $\mu_c$ , and the mean of the values currently in W, $\mu_W$ , if $|\mu_{c_c} - \mu_W |< cet$ . The trustor assesses the length of time this concept has likely been active, $t_{c_c}$ , as the current time, t, minus the time the first outcome in W was recorded:

\begin{equation*}t_{c_c} = t - t_{W[0]}\end{equation*}

The first known occurrence of the currently active concept should be in the first element of the adaptive window, W[0], as the reactive component statistically readjusted the window to contain data believed to be generated by the same underlying distribution.

The next step is to calculate the probability, p(c), that each concept $c \in \overrightarrow{C}$ will succeed $c_c$ given that $c_c$ is assumed to have been active for $t_{c_c}$ . A PDF is built with a Gaussian Kernel Density Estimator using the lengths of time spent at $c_c$ in the past before it was succeeded by c. These times were recorded in TM, illustrated in Table 2, $\forall tl_{c_c,c} \in L_{[c_c,c]}$ . A PDF interprets the probability of the length of the concept, which is advantageous because the reactive component may not identify the time of change perfectly, but the PDF will combine all the times to give an estimate. As $|L_{[c_c,c]}|$ increases, the PDF will get more accurate at pinpointing the time. The probability, $\dot{p(c)}$ , that c succeeds $c_c$ after $t_{c_c}$ time at $c_c$ is:

\begin{equation*}\dot{p(c)} = PDF(t_{c_c}) \times |L _{[c_c,c]}|\end{equation*}

where $|L _{[c_c,c]}|$ is the length of the list, and therefore the number of times c has succeeded $c_c$ . Multiplying by the frequency of transitions turns the probability into a frequency distribution estimate which will give precedence to the concepts which more frequently succeed $c_c$ when the probabilities are normalized.

To normalize the probabilities of each possible next concept (including the possibility that $c_c$ is followed by $c_c$ at this time), the probabilities must sum to one, $\sum_{c \in \overrightarrow{C}} p(c) = 1$ . Therefore, p(c) is given as:

(11) \begin{equation}p(c) = \frac{\dot{p(c)}}{\sum_{c_n \in \overrightarrow{C}} \dot{p(c_n)}}\end{equation}

If no known concept is active, the currently active concept is assumed to be new and it takes on the value $\mu_W$ and probability 1. No other concept has a probability of being active as we have no information about what or when another behaviour might follow the currently active concept.

3.3 Integrating RaPTaR with trust and reputation models

RaPTaR has two points of integration with trust and reputation algorithms. First, RaPTaR detects changes in the behaviour of agents in a group, G, and deletes any records believed to be irrelevant from $\mathcal{O}$ . The effect is that the trust and reputation model draws upon records in $\mathcal{O}$ which are believed to be representative of current behaviours. This contrasts with use of a fixed size sliding window that can only delete the oldest interaction record to accommodate space for the newest interaction record, regardless of the relevance of either the newer or older record. Second, RaPTaR outputs an expected utility which can be used as an a priori estimate in trust and reputation models which incorporate such a value when there are no, or few, direct experiences with an agent on which to otherwise base a decision.

4.1 Dynamic behaviours

The trustor, tr, receives an outcome as a result of interacting with a trustee te, $o_t \in [0,1]$ , dependent on the behaviour of te at time t. This value can be seen as the extent to which, or the probability that, te cooperates or defects. Static agent behaviour, b, is typically defined within a range of values where the bounds are application dependent. Trust and reputation algorithms then normalize these bounds such that $b \in [0,1]$ , where 0 indicates defective behaviour, whereas a value tending towards 1 indicates a cooperative agent. We extend this to define dynamic behaviours, $b(t) \in [0,1]$ , as a function dependent on time. These functions are randomly generated in each experiment, so that the length of time concepts are active for, and the speed of transitions vary. Our evaluation then shows how each method copes given there is no advance information about agent behaviours. The outcome of an interaction is then drawn from a Gaussian distribution with mean b(t) and standard deviation 0.2, to represent that an agent will not always perform exactly the same, or that agents sharing the behaviour from that group behave uniformly. Examples of dynamic behaviours are given in Figure 1. These functions could be repetitive patterns over time as seen in Figure 1(a), or be random dynamic behaviour as depicted in Figure 1(b). An agent’s behaviour is defined by the group they belong to.

Figure 1 Example of groups’ dynamic behaviours

4.2 Groups

In the model assumptions stated in Section 3, agents are identifiable by group, and their behaviour is dictated by their group. Figure 1 illustrates how a groups’ behaviour can change over time. Groups may be explicit, as we have assumed in this paper, but they may represent profiles of entities in the application. For example, Smart Grid users might be classified into groups including a single person living alone, or large houses with multiple occupancy, or industrial offices. As smart meters become more advanced and prevalent in the electricity grid, they can provide more information to improve these classifications using stereotype techniques when group identities are not explicit. For the evaluation in this paper, we use 3 groups, and agents do not change between groups. Agents are randomly assigned to a group at the beginning of their lifetime.

4.3 Network topology

To demonstrate that RaPTaR is appropriate in many domains, we consider agents being connected in different network structures (Anderson et al., Reference Anderson, Lee and Menassa2013). The network topology affects which agents are connected to which other agents, and the consequences to trust and reputation models include limiting available interaction partners and witnesses. Our evaluation considers three common structures. The Kleinberg small-world network where agents are grouped in hubs with few links connecting each hub, the Barabási-Albert scale-free network which uses the preferential attachment property, often used to represent social networks, where agents have a higher probability of being attached to an agent who already has a higher degree, and a fully connected network. When we use the small-world or scale-free networks in our evaluation they are populated with 100, 200 or 500 agents, but the fully connected network is only populated with 20 or 40 agents as the increased number of edges means it is significantly less scalable and simulations become too long to run. We include results with a fully connected network because this represent a portion of the network or a group rather than the whole population. Additionally, a fully connected network represents a scenario where agents have sparse data because of the high degree but many possible interaction partners. Small-world networks in the experiments are generated with a clustering coefficient of 0.5.

Figure 2 RaPTaR more quickly adapts its trust estimates (EU) to reflect the true behaviour compared to other methods. Agents use BRS in a fully connected network with an exploration rate of 0.2

4.4 RaPTaR variables

The conceptual equivalence threshold, cet, is set to 0.1 in our evaluation. Trust models can have low accuracy and so a value of cet as high as 0.1 helps to group together interactions with trustees whose outcomes were generated by the same behaviour but have a margin of discrepancy between their assessment. Additionally, this is low enough that any true behaviours that are different but are assessed to be the same concept are not too different. Previous experiments show that perfect accuracy for estimating agent behaviour is not important but that the ranking of agents needs to be correct (Player & Griffiths, Reference Player and Griffiths2017). If $cet = 0.1,$ this bins trust into 10 categories (assuming that trust is in the range [0,1]), providing enough granularity to rank behaviour from good to bad. The stable learning size, s, is set as low as 3 to encourage learning brief periods of behaviour which occur during transitions between longer periods of static behaviour. We do not consider this an input parameter to RaPTaR because we recommend the value to be small in a dynamic noisy agent-based environment.

Table 3 Average utility per agents per time step in network topologies using different trust models with either a fixed window size (ws), forgetting factor ( $\lambda$ ) or RaPTaR, averaged over 1000 rounds and 100 runs.

Figure 3 Quick adaptations to behaviour change lead to short term large utility benefits which are substantial enough to be significant improvements to average utility over 100 runs