This is a response from the first author of an article that suggest the possibility of lack of replication in imaging studies to my message suggesting that they try to reproduce their results at another site. My comments below.

*********************************************************


From: "Michael Miller" <miller_psych.ucsb.edu>
To: "Yehouda Harpaz" <yh_maldoo.com>
Cc: <todd.c.handy_Dartmouth.EDU>; <scott.grafton_Dartmouth.EDU>;
<michael.s.gazzaniga_Dartmouth.EDU>; <george.wolford_Dartmouth.EDU>
Sent: Thursday, November 07, 2002 4:24 PM
Subject: Re: Reliability of individual differences


Dear Yehouda Harpaz,

Thank you for your suggestion. I was going to refer you to a paper by
Casey et al., 1998, that looks at spatial working memory activations
across 4 institutions, but then I read your website listed below and
saw that you've already given this topic considerable thought,
including a nice critique of the Casey article. Another paper you
might consider is McGonigle et al 2000 in Neuroimage. They don't look
at reliability across institutions, but they do a nice job of
reliability in individual subjects across multiple sessions on simple
cognitive tasks. I see you've also considered the use of the fMRI data
center as a source for reliability studies. We are currently working
on such a project using the analysis that we developed in our most
recent paper. I think it's important not to rely on comparisons of
statistical maps, which can have relatively arbitrary thresholds, but
to rely on quantitative comparisons of signal intensities. My main
focus right now is to look at reliability across a variety of memory
tasks. My experience has been that more constrained memory tasks like
working memory are quite reliable across subjects while more
strategically -oriented memory tasks like episodic retrieval are quite
variable across subjects. Anyway, I appreciate your interest.

Mike Miller

> On Thursday, November 7, 2002, at 02:31 AM, Yehouda Harpaz wrote:
>
> > Dear Michael Miller,
> >
> > I just read your artiacle "Extensive Individual Differences in Brain
> >  Activations Associated with Episodic Retrieval Are Reliable Over Time",
> > And I think it is very interesting.
> >
> > One suggestion for future studies: get _another research center_ to do
> > the
> > same
> > experiments on the _same individuals_, to see if they can reproduce the
> > results.
> > Brain imaging is notorious for being irreproducible across research
> > centers,
> > (presumably because of individual variability), so it is going to be
> > useful
> > to establish
> > that what you got is reproducible across research centers.
> >
> > Yehouda Harpaz
> > http://human-brain.org/imaging.html

*****************************************************

Things to notice are:

  • This guy, for a change, actually read at least part of my page.

  • He was going to refer to Casey et al, which I discuss on my page and show that even though they claim to find "reliable activity", they clearly did not. He doesn't actually say whether he was intending to give it as an example of replication across sites, or as an example of failed replication. However, if he agrees with me that it was a failure, there is no reason why he shouldn't say it, so it is more likely that he thought it was an example of replication.

  • In his future research, he is going to look for replication across memory tasks. Since any lack of replication in such test is naturally attributed to the differences in the memory tasks, this research is "safe" from the possibility of uncovering lack of replication across sites or subjects. Thus for him, the possibility of lack of replication across sites and subjects is not something worth pursuing.

    I tried to ask him whether he agrees with my discussion of Casey et al, and about cross-site reproduction, but he didn't answer.