Subject: Publishing statistical results

From: "Vinton A. Dearing" <[email-masked]>
Date: Fri, 29 Aug 1997 11:19:53 PST

Bob Deardorf asked for the full citation of an article I 
remembered reading in Literary and Linguistic Computing. I was 
unable to find it from the tables of contents in my back issues, 
but I ran across something else during the search, "Proposed 
Criteria for Publishing Statistical Results," by D. Ross (Univ. of 
Minnesota) and B. Brainerd (Univ. of Toronto), in the 
Association for Literary and Linguistic Computing Bulletin, 
VI (1978), 233-234. The authors gave criteria for what they thought 
was a minimally adequate presentation, but the editors of the 
Bulletin said they would accept articles that did not meet them. I 
quote here as part of my campaign for more information in our 
communications with one another.
  "1. The numerical data from which the statistics were derived, for 
example, the counts of frequencies of literary and linguistic 
properties, should preferably be listed in the article or in an 
appendix [or otherwise made available for examination].
   2. When statistical assertions are made, they should be 
accompanied by adequate supporting information.  
      a. When sample means (averages) are obtained, the sample size 
and standard deviation should be routinely included.
      b. Comparison between sample means from a common normal 
population should present the t statistic.
      c. Correlation and regression analysis should present the value 
of r and the coefficient of the relationship.
      d. When the F-test, the t-test, or the chi-squared test is 
used, it should also have complete information about the degrees of 
freedom and the level of significance.
   3. When any other than the most common statistical tests or 
methods of estimation are employed, a reference to an appropriate 
statistical textbook or article should be given. [Strike the words 
"When . . . employed," say I. Who knows what the t statistic is?]
   This does not exclude the presentation of other relevant 
parameters for the tests listed, nor does it imply that these are 
necessarily the best procedures for all bodies of data." [Perhaps 
statisticians have devised other and better tests in the last 
twenty years. In any case, say I, make my day: explain why you think 
your procedures fit the circumstances you are reporting on.]
     Vionton A. Dearing

---

From: Jim Deardorff <[email-masked]>
Date: Fri, 29 Aug 1997 15:31:40 -0700 (PDT)

On Fri, 29 Aug 1997, Vinton A. Dearing wrote:

> remembered reading in Literary and Linguistic Computing. 

I presume you are addressing this primarily to me, namely to Jim
Deardorff?

> I was

That's indeed necessary.  That's in the portion of the paper I said I'd
e-mail to any individuals on the list who may be interested enough
to request it. 

> freedom and the level of significance.

These tests (t, chi-squared..) as I've seen them described seem to apply
to data that tend to fit a normal (or Gaussian or bell-curve)
distribution, which the geometric-progression or exponential-decline
distribution is not.  Not being a statistician, I'd need to be told what
test for what property could be best applied to data that appears to
follow the geometric-progression distribution (follow it up to a point,
that is, and then perhaps deviate from it). 

When you count the number of consecutive duplicate words between parallel
passages that each contain I=10 words, say, you get just one number.  That
is your mean for that value of "I".  There is no standard deviation there
(unless you count them over and over and derive somewhat varying results
each time, due to mistakes in identification and counting; this latter
does of course happen to a small extent, which is one reason why my
results could stand checking by someone else; another reason is that I
used N-A 21, though there were no differences between its results and N-A
26 for the 9 longest strings of Mt-Lk data I checked; but a check using
N-A 27, or maybe TR?, would be an obvious thing to do).

> "When . . . employed," say I. Who knows what the t statistic is?]

Again, not being a statistician, I don't know any more than you if some
standard test can be applied to check out whether a given observed
frequency distribution follows a geometric progression better or worse
than some other mathematical distribution.  If I had known of an
understandable reference textbook on this, I would have cited it.

However, by graphing the data and comparing it against the geometric
progression or exponential, it's obvious by eyeball that these do well
within their range of applicability.  In e-mail form, Part II of my paper
gives the data in Tables rather than graphs, however.  

There's probably a simple way to best-fit an exponential or
geometric-progression curve through a histogram of observed data that
scatters about such a curve, in analogy to least-mean-squares for linear
data.  I've sent an e-mail to a statistician at my university in case he
has time to advise me what it is.

> your procedures fit the circumstances you are reporting on.]

In my writeup, I do present the odds in several instances that an observed
data point (a long duplicate string of words -- 15 or 20 in the cases in
question) could occur due to chance, or due to independent
translation/editing.  Odds like 1/300  or 1/500 occur in the Gospel
parallels; rare odds like these did not occur for the data of the test
case.

Now, to make what's left in your day, you need to hear again why the
geometric progression is the frequency distribution to be expected for
independently translated or edited works.  Let's take the case of an
independent translation. A translator (T2) is editing a Hebrew text into
Greek, and is unaware that another translator (T1) has already done this,
or else T2 does not have T1's text available to him. 

At each step in T2's translation, he chooses a word that has a certain
(conditional) probability of being the same word that T1 had chosen at the
same point within the parallel.  If it is a proper noun that is needed
there, the probability is pretty high, say 0.9, of being the same as what
T1 had chosen.  But it's not unity, since T2 may have chosen to express
the noun in a different case (nom. or accusative...) than did T1 (and same
with any article or preposition that may have preceded the noun).  If it's
an ordinary (non-proper) noun, the probability is somewhat lower, say 0.8,
since T2 may choose a different synonym for the noun than did T1. 
Following this noun, the next word could be a verb or adjective or
preposition..., and the odds are still lower that it would be the
identical word T1 had used within the same parallel, say 0.3.  (That is,
if there were 10 independent translators, only 3 out of 10 would have
picked the same word as had T1 at that point.)  Perhaps T2 chose to put in
a prepositional phrase there, where T1 had placed it elsewhere in the
sentence.  Nevertheless, this could cause a 3-word string (prepositional
phrase) to be duplicated within the parallel passage, if the words T2 used
at each end of the phrase were different than what T1 had used.  But
perhaps T2 used a "kai" at the end of the prepositional phrase and T1 had
also, then this would constitute a 4-word duplicate string (unless T2's
"kai" started a different phrase or clause than what T1's had); the
conditional probability may have been 0.5 for a duplicate "kai."  Continue
this reasoning through a parallel text containing some 10,000 words.  I
contend it leads to a geometric progression for the frequency distribution
of length of duplicate word strings, with the average ratio of occurences
of each word string to that of the 1-shorter word string being the same
(within the scatter of sampling error) as the average conditional
probability involved in all the word choices. 

If instead of two translators two editors independently edit the same
text, the same sorts of arguments exist.  Each editor has different
reasons for wishing to omit certain material and add some of his own, for
making improvements in the original text, and for adding in his own
theology.  These cause the finished works to again exhibit the property of
the longer the duplicate string, the rarer its occurrence. 

I've already explained how I've simulated the T2 case by the numerical
experiment of drawing a ball from a mixed bag of 1000 black balls and red
balls, and keeping track of how many times in a row that you draw a black
ball.  In the latest set of experiments, I allowed the conditional
probability of drawing the black ball to vary randomly on each draw,
between 0.254 and 0.900, like in the discussion above.  Again, on the
average, a geometric-progression frequency distribution was obtained for
the number of black balls drawn in a row, with the ratio of successive
frequencies of occurrence being 0.58, i.e., the average of .254 &
.900.

But of more importance than computer simulations is to try to discover
cases of apparently independent translation or editing of the same basic
document and examine them for their duplicate-strings frequency
distributions. 

So as I see it, when you run into data that depart radically from the
exponential frequency distribution for large word length, not just once
but several times or many times within a given data set, it indicates that
one translator or editor was aware of the other's text -- unless tc can
explain it as due to subsequent scribal harmonistic corruptions. 

Jim Deardorff

---

From: "Robert B. Waltz" <[email-masked]>
Date: Fri, 29 Aug 1997 20:03:50 -0500 (CDT)

I'm going to add a few mathematical comments to this discussion.
This is a matter where proper technique is vital, and rarely
practiced.

On Fri, 29 Aug 1997, Jim Deardorff <[email-masked]> wrote,
in part:

[ ... ]

>e-mail to any individuals on the list who may be interested enough

As a general goal, one should *always* include as much raw data as
possible. Jimmy Adair made the suggestion to me that the World Wide Web
makes it possible to supply all source data, as well as interpreted
results.

Also, if you use the computer for anything, you should either list
the programs you used or provide source code of any custom software
you wrote. If possible, the code should be a "plain vanilla"
dialect of a common language (BASIC, Pascal, C, Perl), so that it
can run on any platform.

>that is, and then perhaps deviate from it). 

More important, I would say, is point (a): Give sample size and
standard deviation. This tells you how "scattered" the data is.

Regression (curve fitting) is a bit of an art; you have to "guess"
the form of the function (typically by looking at a graph), then
use a least-squares fit to get the best approximation.

>N-A 27, or maybe TR?, would be an obvious thing to do).

Since I haven't been observing this discussion, I may have missed something.
But why should each sample be ten words? If you are insisting on ten word
parallels, what about the distance between such parallels?

I am inclined to say that a data set for which you cannot offer a
standard deviation is not a statistic, it's an appeal to faith....

[ ... ]

>has time to advise me what it is.

I think I've covered this above. For curve-fitting, you always
have to "guess at" a solution and a list of parameters.

>parallels; rare odds like these did not occur for the data of the test

I have to ask: How do you estimate such odds?

[ ... ]

>frequencies of occurrence being 0.58, i.e., the average of .254 &

I don't entirely buy this analogy. Drawing balls is a random process.
Constructing a sentence isn't, or the first sentence of this paragraph
could read

    buy analogy entirely don't I this...

or something worse....

***

My apologies if some of this re-covers ground already gone over. But
as a mathematician, I hate to see math abused. :-)

-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-

                        Robert B. Waltz
                     [email-masked]

Want more loudmouthed opinions about textual criticism?
Try my web page: http://www.skypoint.com/~waltzmn
(A site inspired by the Encyclopedia of NT Textual Criticism)

---

From: Jim Deardorff <[email-masked]>
Date: Fri, 29 Aug 1997 23:05:29 -0700 (PDT)

> On Fri, 29 Aug 1997, Jim Deardorff <[email-masked]> wrote,
    [Vinton wrote:] 

> .... 

If the reviewers of my paper should request the source code (in C++), I
could include it.  However, the editor was wishing I could shorten the
paper, not lengthen it.  I could, however, send the program via e-mail if
one of you should request it.

> >> ...

> use a least-squares fit to get the best approximation.

It didn't take much guessing to see the obvious relationship of the
frequency-distribution function.  But as to least-squares fit, I need to
figure out how to scale the function or normalize it first so that
sampling errors where the function is large (though sampling errors are
relatively small there) do not swamp the sampling errors at large "I"
where the function is small (though the sampling errors are relatively
large there).

 
> standard deviation is not a statistic, it's an appeal to faith....

The sample of 10-word strings (I=10) was an example.  There are also
samples of 2-word strings, 3-word strings, 4-word strings... out to 15- or
20- or 30- word strings.  For each data set, you have but one sample for a
given value of "I".  That's it.  In my Mt-Mk Gospel-parallel data set, I
found 321 duplicate 2-word strings, 195 3-word strings, 122, 4-word
strings, 54 5-word strings (sampling error showed up biggest there), 52
6-word strings , 33 7-word strings, 28 8-word strings, 13 9-word strings,
9 10-word strings, etc.  (No two people analyzing it will come up with
precisely the same numbers, I venture to say, but it will be very
similar).  For each of these frequencies of occurrence, there is but one
number that one derives for that value of "I".  Hence, no standard
deviation.

Similarly, for the Gospel parallels between Mt & Lk's "Q" verses, each
value of "I" yields but one number giving the frequency of occurrence of
strings of that length.  This is perhaps a simpler problem that what
you're used to.  If I were to average this in with the data set for Mt-Mk
and the one for Mk-Lk, I would have three numbers for each value of "I"
from which to calculate a standard deviation.  But this would be mashing
together data sets of different size and different authorship for which
the degree of dependence of one writer's work upon the other's (or degree
of independence) is likely different. That wouldn't be logical at all.  So
no standard deviation is involved. 

I could break a given data set into three parts and then calculate a
bare-bones standard deviation for each "I" value.  But then each data set
would be woefully small in total numbers of words.

If you had ten data sets of the same size wherein each writer had
translated the same extensive Semitic document into Greek, independently
of each other, and also ten sharp graduate students eager to work on the
problem, then you could form the standard deviation about the mean number
of duplicate word strings each of length I = 2, 3, 4,.....20 or 30.

Since we don't have such available, it is only natural to turn to a
numerical program that can simulate the same distribution, in order to
explore the sampling error of its results, for example.  From such
numerical results, one could also derive standard deviations for each
value of "I".  But this may not be of much interest to biblical scholars
who are interested in the real thing, not in numerical models

> For curve-fitting, you always have to "guess at" a solution and a
> list of parameters.

So that's where I'm at.  The guess is obvious from the test data, despite
its sampling errors.  For values of "I" from 4 up to about 10 (for the
size data sets available) the geometric progression also holds for the
cases of interest: Mt-Mk, Mk-Lk and Lk-Mt parallels.  For longer word
strings, the results of dependency become pronounced.

> >In my writeup, I do present the odds in several instances that an
> >observed data point (a long duplicate
> >string of words -- 15 or 20 in the cases in 

> I have to ask: How do you estimate such odds?

The exponential (or geometric progression) curve one fits through the data
points from I=4 to I=9 or 10 can be extended on outwards to larger values
of duplicate word-string length, "I".  For I=11, the curve derived from my
test set predicted only one such string should occur on the average, and
one did occur there (though it could easily have been 0 or 2 due to
sampling error).  For I=13 the curve predicted 0.4 of a string of that
length, which has to be zero or one in any one realization.  One occurred. 
For I=14 the curve predicted 0.23 of a string.  None occurred.  For I=15
the curve predicted 0.13 of a string, none occurred.  Etc., no more
strings occurred for larger "I".  Had a string occurred at I= 20, say (as
in the Gospel-parallels data), the predicted number of strings there would
have been only 0.0085.  I take this to mean that if 117 realizations had
been available (1/0.0085), in only one of them would a string this long by
chance have occurred, on the average.  I'd like to know if you would place
a different interpretation than this on it. 

This seems to be the obvious interpretation, and I utilized the numerical
program to check that it seemed to work, as there I had 100 realizations
available, using different seeds with the pseudo-random number generator.

>>I've already explained how I've
>>simulated the T2 case by the numerical experiment of drawing a ball
>>from a mixed bag of 1000 black balls and red balls, and keeping track
>>of how many times in a row that you draw a black ball.  In the latest
>>set of experiments, I allowed the conditional probability of drawing
>>the black ball to vary randomly on each draw, between 0.254 and 0.900,
>>like in the discussion above.  Again, on the average, a
>>geometric-progression frequency distribution was obtained for the
>>number of black balls drawn in a row, with the ratio of successive 

> I don't entirely buy this analogy. Drawing balls is a random
> process.  Constructing a sentence isn't, or the first sentence of this
> paragraph could read > > buy analogy entirely don't I this...  > > or
> something worse....  
> My apologies if some of this re-covers
> ground already gone over. But  as a mathematician, I hate to see math
> abused. :-)  

You may need to analyze a test case of your own choosing, if you were to
get that interested, to see that once you get out past I = 3 the
randomness sets in for a large data set.  In your example, there were no
duplicate word strings of length 2 or more.  In a long works in Greek,
arrangements like that can happen once in a while within a parallel
sentence, as much or more by different choice of words than by
different order of words.  More often, though,
two different translators wouldn't translate that sentence in Hebrew into
Greek without one or maybe two duplicate word strings of I=2 or 3
occurring, especially where theological topics are covered and informal
language like "buy" is not used.  So in numerous other sentences there
will be duplicate word strings of somewhat greater length occurring, and
occasionally of length 15 words or so, which evidently can occur within
the last part of one sentence and the first part of another, etc. 

Look at it this way: if you were to translate 100 pages of Greek Eusebius
into meaningful English, without looking at any existing English
translation, would you want to bet your N-S 27 that you wouldn't end up
with a lot of 2, 3, and even 4-word-string duplications in comparing
against what you read in the Kirsopp & Lake English translation, e.g.? (No
cheating allowed!)  And also some strings of I=5 and 6?

And don't you think the higher you go in "I" the smaller the number of
strings you'd find?

And would you be so surprised if a smooth curve through your data obeyed
the exponential distribution, for I>3 ?  Y(I) = A exp(-bI) .

It's that simple, really.

Jim Deardorff