2. Samples and Criteria

The investigation used two sets of samples. There is a set of 35 larger samples. Each of these contains 1000 words. There are 12 such samples from Matthew, 10 from Mark and 13 from Luke. These 35 samples are divided by genre and by proposed source attribution.Footnote ¹³ The tests on the set of smaller samples used counts of the 5 most frequent words in the 60 samples of 250 words each drawn from Matthew. Tests using these explore more precisely whether or not there are differences within Matthew between samples attributed by the 2ST to Mark or to Q or to M, especially between the latter pair.

The aim here is not to attempt to test all or even most of the theories, but to look specifically at the choice between two of the most widely supported rivals: the 2ST and the Farrer–Goulder theory (FGT) that Luke made use of Matthew.Footnote ¹⁴ If there are significant differences between the relevant sets of samples, then that favours the 2ST against one of its main rivals. If there are no such differences, then that will favour the rival theory that Luke used Matthew. The main task is to examine the stylistic evidence to see what results actually emerge from a fresh scrutiny of the evidence using different methods.

The size of the second set of samples is small. At only 250 words it is one quarter the size of the samples more commonly used. It is not the smallest size that some stylistic studies have used. Henri H. SomersFootnote ¹⁵ used samples as small as 80 words on Paul, Philo and the LXX. H. H. GreenwoodFootnote ¹⁶ used samples the size of each chapter in the epistles. (For Romans the average is 445 words, for Titus 220 words.) Then Forsyth and HolmesFootnote ¹⁷ showed 200-word samples to be viable in benchmark tests. If it were the case that very rare words were being used as the criteria then huge amounts of text might be needed, but this is not the case. The tests do involve small samples of text but the criteria to be used will be the most frequent items in those samples. So the plan is to use items which are very frequent, and the 5 most frequent words in Matthew are the article, καί, αὐτός, δέ, and ἐν. In stylometric work in the last twenty years on a range of different literatures it is very common to use high-frequency function words such as theseFootnote ¹⁸ and that is the preferred method here.

The choice of smaller samples, and the use of only the most frequent words, is also the result of other considerations. Many studies have shown texts to vary in style between speech and narrative.Footnote ¹⁹ It is therefore essential to allow for genre, and to partition the samples accordingly. Here they are classified either as samples of speech, or of narrative or of mixed genre. The latter includes parables and apophthegms or pronouncement stories. Obviously the texts also have to be divided by possible source. With three genres and three possible sources the 18,346 words of Matthew would in theory yield 9 samples of 2038 words each, but in practice rather less. Further, each of those 9 blocks has to be partitioned into several smaller samples in order to obtain a mean and standard deviation. Only in this way can proper statistical assessment be made both of differences between groups and also of the coherence or diversity of style within each group. This last point is a very crucial one.

3. The Statistical Methods Used

Most literary statistics are now performed using multivariate methods which have the considerable advantage of using several variables, or key word criteria, in combination in order to analyse and display the profile of sections of text. The specific methods used here were Correspondence Analysis (CA), the General Linear Model (GLM), two forms of Discriminant Analysis (Discrim and Candisc) and Cluster Analysis. Each method will be discussed in turn together with the results derived from it.

4. Using Correspondence Analysis

The first of these methods, CA, is neutral in that it does not know the groups to which the samples are attributed. It looks at all the samples, analyses the frequency of all the key words in these samples and then proceeds to plot the main differences between these sections of text in such a way that the samples can be located in two or three (or more) dimensional displays. (On the method see Greenacre.Footnote ²⁰) The extent to which the texts do, or do not, fall into groups is then sometimes very evident. So in Figure 1, using large samples of 1000 words from all the texts, it is very clear that the narrative samples are almost all on one side of the plot, and the samples of sayings are almost all on the other side of the plot. The difference of genre between speech and narrative is one of the most fundamental explanations of differences of style,Footnote ²¹ and appears on the main (horizontal) axis.

Figure 1. Plot showing samples from the Synoptic Gospels

The output in Figure 1 shows the location of 35 large (1000 word) samples drawn from all three Synoptic Gospels. The criteria or variables used here are 30 words of fairly high or high frequency in Matthew. These 35 samples are shown with different fonts to identify three genre categories. Each Synoptic Gospel is identified as Mt or Mk or Lk and then narrative samples are in bold face, speech samples in italic font and samples of mixed genre in normal font. Source attribution in Matthew is indicated by showing the 3 M samples in a square surround, Q mixed genre in a diamond, and the 3 samples of Q sayings in Matthew in oval surrounds.

In CA the first major contrast in the data is shown on the main axis, normally the X or horizontal axis. Here it is very obvious that narrative samples are those in bold face on the left, and speech samples are those in italic font on the right. Samples of mixed genre are more central, range from −0.25 to +0.2 on this axis and overlap the others only slightly. Genre is evidently a strong factor in the style of the Synoptics. Other output (not printed here) shows that narratives use words such as καί and δέ (and, but) and genitive endings. The speech samples are characterized by words such as εἰ, γάρ, ἀλλά, μή, οὖν (if, for, but, not, so then).

The second contrast on the Y (vertical) axis captures the next most important feature of these texts. The lower third of the plot contains 8 of the 10 samples from the Gospel of Mark, whereas all three samples from M are in the top section of the plot. The third dimension (not shown) captures the differences between Matthew and Luke. CA therefore picks out the most prominent differences one might expect to find: narrative v. speech, Mark v. M, Luke v. Matthew.

5. Using High Frequency Words in 60 Smaller Samples

As the material had to be divided both by genre and by source it was quickly apparent that there were insufficient samples if the sample size is set at 1000 words. The amount of text cannot be increased, so each sample from Matthew was divided into four samples of 250 words, and some additional sections, previously too small, could now fit. A consequence of this strategy is that only the most frequent words in the text will now be used as variables. At the same time a change of method will permit samples of more than one genre to be tested simultaneously, as there is a method which allows for genre, before testing whether an overall source effect is unlikely to be due to random variation.

This also allows the statistical significance of the results to be assessed, something which is not the case with the simple numerical counts based on vocabulary in many commentaries on Synoptic texts. There are studies which do use statistics which include, or are amenable to, significance tests. Works by Lloyd GastonFootnote ²² and Raymond A. MartinFootnote ²³ and a number of studies of patterns of verbal agreements in the Synoptics cited by John PoirierFootnote ²⁴ provide examples of this.

Matthew yields 60 samples of 250 words (when one containing the genealogy is omitted). The samples are classified by genre and by proposed source. The key word variables are now the 5 most frequent words in Matthew. These, with the totals for each in Matthew, are: the definite article (art) (2790), καί (1178), αὐτός (aut) (922), δέ (494) and ἐν (293).Footnote ²⁵ Where subsidiary tests use only four criteria it is ἐν which is omitted. There is still insufficient narrative for tests on Q narrative material, but Markan and Q and M material can be tested on the sayings and mixed genre groups. The latter include parables and apophthegms. In order to eliminate possible bias from using variable selection methods, these were avoided. The 5 most frequent words were used as standard for all the main tests, though some subsidiary tests were used with the 4 most frequent words and/or the removal of samples suspected of being problematic. These subsidiary tests were strictly limited to the exploration of supplementary hypotheses, and only the standard tests were used for the main conclusions.

6. Using GLM to Test for Source Effect while Allowing for Genre

The method used in the next section is part of the SAS statistical suite and is known by the abbreviation GLM. It is their version of a more widely used method known as the General Linear Model. In the work reported here GLM was run on the variation in the use of the 5 most frequent words in the 250 word samples from Matthew. The null hypothesis to be tested is that, when allowance has been made for a genre effect, there is no overall source effect. If the resulting p value is very low, then the conclusion is that the difference indicates that a variation in the word counts correlates with a change of source, and is unlikely to be due to random variation. Normally a p value of 0.05 (5%) or less is regarded as significant, and one of 0.01 (1%) or less as very significant. (More loosely researchers tend to say that a low p value indicates that a given effect is significant, but that should be interpreted in the manner described above.)

The results from GLM normally indicated a significant genre effect, but also a significant source effect, even when allowing for the genre effect. There are two types of potential error here. Accepting a value of 5% may be too lenient, in which case the researcher might infer a significant source effect when that is not the case. Alternatively, however, insisting that only a p value of 1% or less is to be regarded as significant could lead to the opposite error, and cause the researcher to reject evidence of a significant source effect, when it does indeed exist. One cannot reduce the risk of one of these errors without increasing the risk of the other. That is the more serious calculation which lies behind the looser explanation that a 5% p value is significant and a 1% p value very significant. The decision that an effect is not random does not exclude an objector appealing to some further factor. Such an appeal would, however, need to present an auxiliary hypothesis, and itself need to be tested.

The tests carried out using GLM were on samples divided into 3 genre categories (narrative, sayings and mixed) and allow for 3 possible sources (Mark, Q and M). In some tests only some of these categories were available, or selected even if available. GLM can be asked to divide the material by source and genre, calculate the relevant means, analyse the variance, give a p value to the hypothesis that there is no overall genre effect and then do the same for the hypothesis of no overall source effect. It will also, on request, plot the actual and predicted (mean) values for each sample in each group.

For tests on Matthew GLM was run by a SAS statement providing a model of the following kind:

The model is designed to test whether the frequencies of the items listed on the left vary in response to the effect of source, genre and the interaction between source and genre. In other words the model is testing the view that a change of genre in a text results in a change in the use of the most frequent words in the text. It is also testing the view that, when allowance is made for genre, a change of source can also result in a change of use in these very high frequency words. Sometimes there can be an interaction between such factors, and that should be considered as well. In most cases, in this study, the possibility of interaction can be checked and then excluded as not significant.

What this procedure does is to formalize what literary scholarship often does intuitively. The careful reader of a text might well expect to find that style changes, and that the use of particular words changes when there is a change of genre in a long text, or when there is thought to be a change of source. Technically the terms on the left are potential response variables. The model sets up a test of the hypothesis that the items on the left do not vary in response to changes of genre or source as specified on the right. This null hypothesis is rejected if a sufficiently low p value results, as discussed above.

The tests do not examine every possible solution to the Synoptic problem, but have the more limited aim of testing the null hypothesis that there is no overall source difference between Q and M material in Matthew. If the Q theory ought to be rejected, and the view that Luke used Matthew ought to be accepted, then there should not be significant differences in Matthew between Q passages and M passages. If the null hypothesis stands, then that favours the FGT, if it falls, that favours the 2ST. Faced with a complex set of problems it is sometimes advisable to aim at a more modest ‘crucial experiment’ to test between two rival views, rather than to try to find a method to solve everything at once. (It is not unreasonable to hope that others might consider adapting the methods to resolve other aspects of the overall problem.)

Some of these tests do also include comparisons of Matthean material held to derive from Mark. By comparing Q and M with such supposed Markan material it is possible to put the comparison of Q and M in a slightly wider context. It also helps to establish that the method used is also identifying other differences which are to be expected on the theory under test.

6.1. Using GLM on Markan, Q, and M Material

A series of tests were run and these are summarized in Table 1 below. Each test is first given a brief account and then the table of results follows the last of these accounts. The first six tests all use the standard variables consisting of the 5 most frequent words in Matthew. The remaining tests examine a number of minor variations in order to look briefly at various alternative ways of viewing parts of the evidence.

Table 1. The Main tests with GLM

The first test includes text attributed to Markan, Q and M material. Narrative cannot be tested as there is insufficient pure narrative in the Q material. There are 22 samples of mixed genre, and 22 samples of sayings. Allowing for genre the p value for no overall source effect is 2.73% and so significant at below 5%. If only the saying samples are tested, the p value is 0.69% and so very significant at below 1%. It is reasonable to infer that there is a difference between the sources, and that it is even more evident in the sayings material.

7. Using Discriminant Analysis on Single Genre Comparisons

Discriminant Analysis is a further resource which can be deployed where only one genre is involved. SAS provides two forms of this method. Discrim supplies not only details of p values, but also Mahalanobis distancesFootnote ²⁹ between groups, and error rates where cross-validation assigns a sample to a group other than that expected. In this paper the words attribute and attribution are normally used to indicate the allocation of texts to sources by conventional literary scholarship. The terms assign and assignment are used to denote the outcome of cross-validation tests that test each sample in turn against all the others in two (or more) groups.

Candisc gives some of the same figures, but will also plot the resulting locations of the samples and so the groups. This allows a visual check on the extent to which the samples do, or do not, form groups whose separation is clear and distinct. The 5 most frequent words were once again used as variables, and the variable selection method (Stepdisc) was not used as this can introduce bias. The aim of the next series of tests is to look at the grouping of sayings attributed to Q material and sayings attributed to M material. The hypothesis to be tested is that these groups are not distinct, and that samples cannot be correctly assigned by this method to the group to which they are supposed to belong. This hypothesis should be rejected if the p values are low, if the Mahalanobis distances between groups are significant and if there are few classification errors. Again rejection of the null hypothesis would favour the 2ST over the FGT; acceptance of it would favour the FGT.

The first test was to see whether there was clear discrimination between the sayings samples from the three proposed sources in Matthew: Mark, Q and M. These provide 22 samples of sayings in all. There are four different ways of calculating the multivariate p value which results. The simplest way to report this is that using the 5 variables on three groups p was very significant at 0.69% or less. (The lowest value was 0.07%.) The Mahalanobis distances measure the separation of the three groups and the p values for these were between 1.11% and 1% in all three cases (Mk/M, Mk/Q, Q/M), signalling a significant distance between all three groups. The resulting plot (see Figure 2) sent M samples to the upper left quadrant, Q samples mainly to the lower left quadrant, and Markan samples to the right quadrants. (The plot is extended to the right.) Most of the samples are in three separate groups, but with three samples overlapping in the centre of the plot.

Figure 2. Plot showing Candisc on sayings samples from Matthew (Mk, Q, M)

The error rate from cross-validation was 5/22 (22.7%) with 77.3% assigned to the expected group. Each sample in turn was stripped of its source identifier, and classified into whichever of the three groups it matched on the basis of the statistics. When each sample had been separately assessed against all the others, the number of incorrect assignments was tallied. Success rates of around 73% for function words and 79% for strings were regarded as quite normal in stylometric studies of other literatures performed by Forsyth and Holmes.Footnote ³⁰ Results for word frequencies by GrieveFootnote ³¹ in 2007 reached 80–90%, but when testing only two authors with a large function word profile. It requires very voluminous texts, such as lengthy novels,Footnote ³² to obtain success rates above 90%.

The overall multivariate p value of 0.69% (or less) did produce an outcome in the very significant category, though the error rates suggest slightly more caution. These results indicate that there probably is a stylistic difference between the three groups of samples attributed to the three different proposed sources. It is not possible, however, to assign every single sample to its expected group; this can be done only with about 77% of them.

Three-way contrasts are always more problematic than binary comparisons. So the next test focused on the relation between samples attributed to Q and those attributed to M. Whether or not there is a difference here is central to the purpose of this investigation. There were 12 samples of Q sayings and 5 of M sayings, making 17 in all. The 5 most frequent words were again used as variables. This time the p value was 1.58% and so well below 5% and clearly significant. Cross-validation gave 3 errors out of 17 samples. With 14 samples out of 17 correctly assigned the success rate was 82.3%. Candisc produced the plot which follows (see Figure 3).

Figure 3. Candisc on Sayings samples from Q and M material in Matthew

The plot shows 12 samples of Q material towards the left, four samples of M material to the right, but one M sample nearer the centre of the plot on the right-hand edge of the Q group. The three errors involve three samples in the centre of the plot (S, R and 8). Cross-validation assigned S and R to M, and sample 8 to Q. This clarifies the issue in that it can be seen that, in the main, the two groups are distinct,Footnote ³³ but there is a small area of overlap between them. The method can correctly distinguish Q sayings samples from M sayings samples in just over 4 instances out of 5.

Some additional tests were carried out in order to explore just which samples were most prone to being assigned to a group other than that to which they are attributed by the 2ST. These were not intended to amend the findings of the main tests, so they are omitted at this point but reported in note 34 below. As the evidence of the main tests is sufficient on balance to reject the null hypothesis of no significant difference in Matthew between Q material and M material, then that favours the 2ST as against the FGT. This should make it clear that the purpose of these additional tests is not to make any special pleas in relation to the main argument, but rather to explore, elsewhere, what factors might be at issue in those samples which differ from the main trend of the results, and whether fluctuations observed in the main results might match one or another of the variations adopted by proponents of that theory.

Table 3 provides an overview of the main tests involving Discriminant Analysis.

Table 2. Subsidiary Variations on the Main Tests

Table 3. Results from Discriminant Analysis

The second result should be regarded as the main one and the supplementary explorations as interesting additional information which is not to be given the same weight. The main result was a p value of 1.58% for Q versus M sayings, three errors out of 17 and a discrimination rate of 82.3%.

There were, however, two further tests. These paired up 250 word samples to form samples of 500 words. Sample 9 had no partner so was first doubled and included, then omitted. In the first case 8 of 9 samples were assigned to the expected source (88.9%), in the second 7 out of 8 were assigned to the expected source (87.5%). In each case sample N + O was wrongly assigned.

8. Using Cluster Analysis

Another method which can be used here is Cluster Analysis. This method is not given prior knowledge of the groups to which samples are attributed. It simply joins the two samples closest to each other into a cluster, then joins the next two nearest items together, and so forth until eventually all the samples are in a single cluster. It is then possible to see if the resulting pattern matches expected groupings in the data, or reveals unexpected patterns deserving further exploration. Earlier tests reported in this paper had first used samples of 1000 words, then much shorter ones of 250 words, and briefly a set which combined pairs of 250 word samples into samples of 500 words. These latter provide a suitable set for Cluster Analysis, with one drawback that sample 9 had no partner and so was simply doubled. This solution might accentuate any wayward tendencies in sample 9, so as a safeguard Cluster Analysis was also run without this sample.

When the 9 sayings samples of 500 words were run with Cluster Analysis, using counts of the 5 most frequent words, Ward linkage and Euclidean distance, the 6 samples from Q clustered together and the 3 samples from M clustered together. The resulting chart shows the Q samples on the left and the M samples on the right. It is worth noting once again that Cluster is not told whether samples are attributed to Q or to M—it links samples by assessing the similarity of their usage of the 5 most frequent words.

The plot (see Figure 4) is in the form of a tree, or dendrogram, showing clustering starting near the left edge of the plot (where LM joins PQ). More pairs of Q samples join this cluster until all 6 are there. Meanwhile the M samples 56 and 78 join up, eventually followed by sample 99 (the doubled 9). The end result is remarkably close to the expectations of the 2ST. (The dendrogram of the run without sample 9 is very similar, apart from the absence of that sample.) This produces an initially puzzling result, that Cluster Analysis should not reveal variations which Discriminant Analysis identified. It is probably correct to regard cross-validation as more rigorous, but it is also worth noting that even this did give a higher assignment of samples to expected source when the sample size was 500 words. It is also worth noting that sample N + O was late joining the Q cluster here, though it did do so. When Cluster Analysis was run on the 250 word samples it assigned 14/17 to the expected source (82.3%), putting 6 & 8 with Q, and S with M.

Figure 4. Six Q samples cluster on the left and three M samples cluster on the right

The overall result from Cluster Analysis supports the main conclusion that these tests can assign about 80% of the sayings samples attributed to Q or to M to the expected source. When the sample size is increased to 500 words this figure rises, but we are using fewer samples. GLM, Discriminant Analysis and Cluster Analysis allow a cautious judgment that Q material and M material are not indistinguishable, even if it is not possible to effect a clear separation of every last sample in every test.Footnote ³⁴

Stylometric work can find clear and distinct differences between major known authors of lengthy texts, but when dealing with texts much shorter than modern novels it is unreasonable to expect results with accuracy close to 100%. Forsyth & HolmesFootnote ³⁵ give a figure close to 73% for success rates using function words on small samples in this kind of work. In the case of the proposed Synoptic sources the difficulty of achieving clear discrimination is greater. If there are sources it is likely that they have been adapted to a greater or lesser extent by the authors who included them. Where a proposed source still exists, or where there are two adaptations of a proposed source, estimates of the extent of that adaptation can be attempted. The aim of the current work is slightly different from that. Here the aim was to investigate the theory that if samples of text are taken from two proposed sources, and allowance made for genre there is still sufficient stylometric evidence to reject the view that there is no significant difference between them.