So in this last module, since we're talking about the writing process. I want
to make sure to just give a few last tips for the very final draft,
Before you send something off to your editor before you send something off to be
peer reviewed, there's a little check list you ought to go through on that very final
draft. Once you revise it and the prose is
sounding good, there's a few other things that I want to make sure that you think
about and check. So my little checklist for the final
draft, got to check for consistency and in particular for numerical consistency.
You also want to check and make sure that your references are not what I call
references to nowhere. And I'm going to talk a little bit more about. So those are
kinds of things that kind of a high level that you want to make sure you've, you've
checked off your list before send something out.
So, checking for consistency just means, make sure that you don't have things that
are totally contradictory in different parts of the manuscript, and, and this
happens often. I was recently editing somebody,
somebody's work, and in the methods section, they said, we followed
participants for a minimum of two years. And then later on, I went over to the
results section, and I read, the average follow up time was 1.5 years.
So I was going, huh. If every participant was followed for more
than two years, how could the average be 1.5 years?
It didn't make any sense. So they probably had an incorrect word
choice there. They didn't need a minimum of t2wo years.
They actually were. Their, their goal was two years.
So that was a little inconsistency. So try to pick up on those, because
reviewers will look at those in with, with great, you know, it raises a lot of
questions, a lot of red flags. Even more importantly. well I think it's
more important cuz I often focus on the statistics and the numbers of papers is
make sure that you check for numerical consistency.
So do the numbers in your abstract. Match the numbers in the rest of the
paper. And I put that one up first because it's a frequent thing that I see.
The papers I'm reviewing, or even in papers that are actually published in
literature, that sometimes for some reason the abstract numbers don't end up matching
things in the rest of the paper and I don't know why that's such a frequent
occurrence. I think it's possibly that people write
the abstracts first for like a presentation before the data analysis is
really final. And then, they end up cutting and pasting
that abstract and they forget to change all the numbers or maybe there's a cutting
and pasting error just from going from the tables and figures in the text to the
abstract. Or, I don't know why it is, but there,
there seems to be often a lot of inconsistencies there.
I've seen many examples also, where the numbers and the text don't match exactly
the numbers in the tables and the figures. So for example, I was doing a statistical
review of a paper just this week. And, I got very confused because I read
the numbers in the text and then I went to table two, where they told me to go,
And the authors had basically subtracted column one from column two to come up with
the numbers in the text. They had just done a subtraction.
Well, I subtracted those two numbers and I didn't get the numbers that they had in
the text. I didn't get anything close.
So, it was like, huh, Well I, I'm totally confused here.
And then even worse, they calculated percentages also from table two and, and
they came up with totally different percentages than I did.
So since it's simple arithmetic, I'm thinking that they used different numbers
to calculate the figures and text than they did for the tables and figures and
that raises a big red flag. Why are those numbers, why don't they
match? And then I've also seen the case where so
numbers in one table are figured don't match, this, this what's suppose to be
similar number in another table, I figure. So, the very common case of that you will
have some numbers at a table and then somebody will also present those numbers
in a graphic. And if I kind of look at the graphic and
try to figure out the exact values in the graphic and then I look at those numbers
in the table, those often don't match. So again, it raises all sorts of red flags
and I don't know exactly why these kinds of, things happen.
It might just be human error. It might be that people are using
different iterations of the data set, But all of those things, raise real red
flags for a reviewer, especially, a statistical reviewer.
So you want to make sure that you are consistent throughout your paper and that
those things match. And then, the final thing to think about
is your references and you wanna find out and kind of nature you pay attention to
whether or not you have what I call references to nowhere.
And these are references that when you go to the reference,
If you're reading the paper and then you go to the reference to try to find the
information that the authors indicate is available in that reference.
In fact, it turns out that, that information is not there.
And I have to tell you that I do a lot of this for various reasons. I end up wanting
to get the information from the original source.
And I go back from a reference from a paper I'm reading.
I go back to the paper they've indicated is the citation and it's more often than
not the case that the reference does not if fact provide the indicated information
or fact that what the authors are saying in their paper it provides.
So this is the rule, not the exception. So you should know that,
So that you don't end up accidentally citing somebody's mistaken citation.
So there's a number of reasons this might happen.
So often times authors will sort of misinterpret or exaggerate the findings
from the original source. If you go back to the original reference,
it turns out that they were selective in the information that they chose to cite in
their paper. Or they're, they're referencing a paper to
back up a particular statement and that statement is not exactly supported by the
original reference. It might be supported in some roundabout
way but not directly. So oftentimes, those sources really don't
back up the statements that the authors are writing in their paper where they are
citing the source. Another really, really common problem is
that the reference that the authors have cited actually is a secondary source and
it's not the primary source. I call this citation propagation and I'm
going to talk about a little more. So somebody, you know, Jones et al, Smith
et al does the original study, They come up with a number of figure or
statistic. Jones et al cites Smith et al.
Okay, now Beery et al goes to give the statistic in their paper.
Well, they found that statistic in Jones et al,
They read it there, so they don't bother to go back to Smith et al and get the
original reference, they just reference Jones et al.
Okay, well then what happens, well. James et al now wants to say Beery et, you
know, it reads the statistic and Beery et al, Now, they say Beery et al. So, now
this citation, this statistic has gone through multiple, it's been propagated
across the literature. And it reminds me of the game telephone,
that children play. So, if you're not familiar with that game,
that's the game where the children sit in a circle. And the first person who comes
up with a sentence. And they whisper that sentence in the ear
of the next child. And then, that child whispers into the ear
of the next child, and so on and so forth until you go around the circle.
You get to the last child. And that child said, says out loud what
they heard. And it's always something that's very
garbled and funny that has little resemblance to what started the statement
that started the whole game. And this is just what happens in the
scientific literature when you propagate citations.
You get, you lose important pieces of information from the original source down,
down the chain. And so be, pay attention to that.
I'm going to talk a little bit more about that give you an example in a minute.
And then, sometimes others just miss number references.
A way to avoid this is if you have available to you a reference management
program like EndNote. It's highly recommended that you use that cuz that can
avoid these kind of mistakes. But I'll give you an example of a paper I
was working on a few years ago. There were multiple authors in the paper.
We were not using a reference manager since there were multiple authors across
different institutions looking at the paper.
There was, and the writing had some problems in it, so a number of authors
rewrote the paper, did some heavy editing on the paper.
And in that process, a few of the references, the reference numbers got
garbled. So, reference what we were calling
reference three on draft one, you know, became reference five on draft two and
nobody made the correct change. So, when that paper went out for a review
with misnumbered references, because all sorts of confusion among the, one of, in
particular, one of the reviewers, in terms of they were confused.
And I initially thought that the reviewer just didn't know the literature, but
actually when I looked carefully, I finally realized that the reason that the
reviewer's confused because our number references are misnumbered.
And so luckily we've caught it but it can happen pretty easily, especially if you're
not using an automatic reference management program.
So, watch out for that kind of thing, especially the multiple people have looked
at the draft and edited the draft, references tend to get lost in the
shuffle. So here's an example of the first kind of
thing I was talking about. So you, some authors give a reference to
back up a statement. You go to the original reference and it
just doesn't back up the statement. So here's an example.
I'm going to pick on a particular example from something I was reading lately but
again, These authors are by no means the only
ones who do this. This is very, very common.
So, I read the sentence. These data are particularly disturbing as
the UVC emission is even larger than ambient sunlight on a mountain.
Now, I was writing about this for the lay public and so I thought, well great, here
I've got a comparison that anybody can understand.
The amount of UVC, if you're exposed to sunlight on a mountain.
Well, that's a great way to put this in my piece.
So I wanted to get more information about this.
I thought that was an interesting and cute fact, a good fact to use for a piece for
the lay public. And I also had a question, because when
they say it's larger than ambient sunlight on the mountain, they didn't give any time
frame. So, is it larger than the amount that
you'd get in one minute? Is it larger than the amount that you'd
get in an hour? So there was no reference right there.
So I needed to go back to the original references to get more information.
And again, I, I'm doing this often for various reasons like, like we've been
writing something for the lay public and I see an interesting statistic or fact that
I wanna trace back. Alright.
So, to go to reference thirteen, Well, it's, it's you know, a URL.
It's, it's a website link and it was broken.
So, the, the link doesn't work. I get, you know, an error message.
It does bring me to a site. So I search on that site and there's no
relevant information about UVC emissions on a mountain on that site that I can
find. Maybe I didn't do a great search.
So maybe, you know, the URL, link got broken, you know, between the time the
authors wrotethe paper and, the time it was sent, it was published.
Could have happened, But anyway, it's a problem because again,
I'm not able to trace that reference. Alright. So I said, well, well good.
They gave two references, so I have hope here.
So I went to reference fourteen. Pulled up, the paper. It was a paper.
Scanned through the paper, did a search on the paper.
It did not contain the words the ambient sunlight, the word mountain, or the word
UVC and I didn't do UVC lighting, I just did UVC.
So there was no mention of any of those things in that paper.
So, it was a reference to nowhere, because what the authors were giving me as their
backup for that statement, it just didn't exist at least not in those references.
So I wasn't able to ever get that information, cuz it wasn't at least where
the authors told me it was. So that's an example of references nowhere
and again, I think this happens a lot. So, you know, if you can avoid it in your
own writing, hopefully, by doing a good prewriting step, by carefully organizing
your information before you start, and keeping your, you know, that information
with the references early on you won't make these kinds of mistakes where you
just have references that don't actually back up what you're saying.
And then finally, the citation propagation concept like telephone I was, I was saying
before. This was one of my pet peeves and so I
just want to point out an example of this and caution you to avoid citing secondary
sources. So, when I was a graduate student, I
worked on, something called female athlete triad.
And one of the components of that triad was something we called disordered eating,
which itself is kind of poorly defined, but in any case, a part of the triad is
disordered eating. So people would always want to say, well,
how common is disordered eating in female athletes?
And at the time that I was a graduate student and writing out papers on this,
In the late 90's or early 2000's, there was Hallmark statistic that everybody
cited. It's, the statistic was that fifteen to
62% have disordered eating. And that was just in every paper on female
athletes triad eating disorders in athletes.
That's just the one that everybody gave. However, it wasn't like they all cited the
same source. If you, I was trying to trace back at one
point where that statistic came from and I found about 50 different attributions for
where that statistic came from. So one paper would cite Smith et al.,
another paper would cite Jones et al and they all cited different things.
Well obviously that statistic came from somewhere.
So, everybody's citing secondary sources. So, I want to give you some examples.
So, this was from a paper in the Journal of General Internal Medicine.
They say it has been estimated that the prevalence of disordered eating in female
athletes ranges from fifteen percent to 62 percent and they give two references
for that. One is a book and one is an article.
These are articles in 1996. That article is not the original reference
for that statistic. That book, of course, is a review and is
not also the original reference. So those are the citations given, but
again, they're secondary sources. So notice that they referenced all the way
back to this was a paper in you know, like the early 2000's, they referenced back to
1996. The original references go back further
than that, as I am going to show you in a minute.
Here's another example. There was a fact sheet on eating disorders
I found on the internet and it gave a citation.
Well, that citation was also a book. So they were citing a book rather than
again an original an original source. Here's another 2000 review in American
Family Physician. They give the statistic, although, the
exact prevalence of the female athlete triad is unknown, so they have reported
disordered eating behavior at fifteen to 62% of female college athletes.
That's very specific, female college athletes.
We'll, have to evaluate whether or not that was actually where that statistic
came from. They didn't give a citation on this one.
It was just so widely used that everybody just threw it around.
No citation was given. Another example, a 2004 paper they cited
as the source of that statistic variant [inaudible] 2000.
Studies report between 52% and 62% of college women engaged in problematic
weight control behaviors. Okay so I, I pulled that citation today,
Beery, you know out of curiosity and looked at the paper.
It doesn't contain any mention of that statistic.
So it's not even it's definitely is a reference to nowhere because not only is
it not the original source but it doesn't even talk about that statistic in that
paper. So I don't know, you know, it might have
been a misnumbered reference or something. So it was, it's just all over the place.
In fact now, if you look at the literature now in this area I think people have
finally stopped using this statistic. Which is a good thing.
But for a long time it was widely used with all massive different citations.
Never back to the original citation. Interestingly,
As I was Googling this, I found a 1999 New York Times article and
actually they got the closest to getting this right.
So they said, but informal surveys suggest that fifteen percent to 62% of female
athletes are affected by disordered behavior that ranges from a preoccupation
with losing weight to anorexia or bulimia. So that informal survey is actually right
on a money, so I know that some fact checker at the New York Times actually
went back and found those original sources.
So, so they did a good job, You know, publications like The New York
Times has a fact checking department. Unfortunately, a lot of journals don't
have a fact checking department, so, you know, these kind of things slip through.
So out of curiosity I got interested in where that statistic came from.
So I went out and found the original source, and this took some digging, this
took some sleuthing on my part. So where did those statistics actually
come from? So the fifteen percent actually came from
1987 paper. The 62% came from a 1988 paper and there's
a two part, cuz actually it was three papers in a series,
So that was something in between fifteen percent and 62%.
That came from a 1986 paper and all of these papers had one author in common.
So there was an author common to all three of those.
So that's the, where the fifteen percent to 62% came.
It was all, it was over two decades old. It was still being cited in, you know,
2000, 2006, 2007, 2008 sometimes even.
So that, those references were, were, two decades old.
That, that statistic was two decades old. Of course, nobody every cited the 80, the,
the original reference. So these are really old studies.
If you look carefully at the studies, they're very limited in their study
design. So their cross-sectional surveys, they
don't have a non-athlete control group. So there's no control group, it's just a
survey, it's self-report Where, what was the populations?
They were, these aren't random samples of athletes. These were just convenient
samples. So, in one of their studies we got a bunch
of athletes from some Midwestern universities,
Across a whole bunch of sports. So those were college athletes but, okay.
Now the next one, the 1987 study actually was nine to eighteen year old swimmers at
a swim camp. So it wasn't college athletes although
everybody later cited it as college athletes just some swimmers at a swim
camp. And then another was also college athletes
college gymnasts, just 42 though, college gymnasts from just five different teams
who happened to be at an athletic conference.
So these are totally non-representative samples across a wide range of athletic
groups with no controls. And, if you look at the measurement, I
don't want to go into a lot of detail. But if you're curious, you know, what they
were calling disordered eating was you had to have done one of the following for at
least once a month. Well some, for some of the sur, surveys,
you had to do it twice weekly, sometimes they had fluid restriction in there.
So there was really a wide range in what they were defining as disordered eating.
You know, so here were the findings that 1987 study found that fifteen percent of
swimmers at that swim camp used at least one of these behaviors.
So that was the source of the fifteen%. The nineteen 86 paper found 32%.
Overall, used at least, you know, had this disordered eating according to their
definition, and that ranged from eight percent among basketball players to
74% among gymnasts. And then the 1980 study, 1988 study, found
that 62% of the gymnasts at the conference, you know, had this, had this
behaviors. So that's the source of the statistics.
But yet, this statistic was used as if it was the gold standard.
It was used everywhere. But it basically had no meaning, because
it came from studies that really couldn't address this very well.
It had nothing inherently wrong with the studies, they were just very, very
limited, and they were old. So my take home message is, you always
want to cite, You go back to the primary source.
Bother to dig up that primary source. Don't trust that just because another
author says the statement or the fact that gives a citation,
That, that author has done it right, because more often than notm those authors
have made errors in their citing. So you don't just want to rely on somebody
else. You wanna dig back to the original source and actually get all the details,
Find out what all the actual information was, and let your reader know what that
actual original source was rather than doing this kind of propagating citations
through the literature. The preceding program is copyrighted by
The Board of Trustees of the Leland Stanford Junior University.
Please visit us at med.stanford.edu.