Single-Subject Experimental Designs: Measuring Treatment Fidelity

Single-Subject Experimental Designs: Measuring Treatment Fidelity


Well, it has number of terms that are being
used. One of them, as you know, is treatment fidelity. But there are other synonyms or
more or less equivalent terms such as “procedural integrity” or “treatment integrity” or “procedural
reliability” so there are four or five different terms that are being used. What it means though, as a concept related
to treatment is that you measure that the intervention is implemented as you had planned.
You have a treatment protocol, and you want to make sure the treatment is carried out
as you had intended. Why is that important? It’s important because
if you don’t know how the treatment was implemented, it becomes very difficult to know what the
causal factor in the change was. So you reach a certain outcome, but you cannot really attribute
it to something concrete because you don’t know how well the treatment was implemented.
It affects internal validity. It affects external validity. It’s a very important aspect of
treatment research. You have to think about how the treatment
breaks down into steps. you want those steps to be reflective of what the active ingredient
of the intervention is. Sometimes you see in the literature there
are steps laid out and they are measured, everything with good reliability, good treatment
integrity, but the steps may not necessarily be reflective of what the key construct is. It’s important that those steps are related
to what your theory of change is. You have a certain theory, and you want to make sure
those steps are reflective of it. That’s one aspect. The other challenge is that you have to think
about what method are you going to use to evaluate treatment fidelity. You can use self
monitoring. That is when the experimenter him or herself basically does check marks
or takes notes. So that’s one method. The second method is when you have a second
observer, and the second observer basically takes notes or records how well the experimenter
does. That’s the second method. And the third method is when you have the
experimenter take notes, and the second observer, and then you compare. And you derive what
is called interobserver agreement on treatment fidelity. So the first and the second step
are not mutually exclusive, you can do both. So that’s a big thing to think about in terms
of measuring treatment fidelity. Another challenge is: How many sessions do
you need? Do you need 100%, or is it okay to have maybe 20% to 30%? There are no clear-cut
rules about this. But in general, journal editors and reviewers like to see above sort
of 20%, at least, of the observations. In the beginning when I started doing this
kind of work, I was under the illusion that I could design this at the desk, and then
the experimenter will implement it as I had planned. When you design an intervention basically
at your desk, it all looks very deliberate and concrete and logical. Then you actually
ask somebody to implement it, and there are steps missing, unforeseen circumstances happening,
and the whole thing will fall apart. And you have to start from scratch. It’s an iterative
process to develop a good treatment fidelity protocol. I have learned the hard way that
you always have to pilot. For multiple reasons pilot studies are really important, but related
to treatment fidelity, it is essential. You don’t know if this is actually doable. You
prepare a data collection sheet, and the observer says, “This is too cumbersome. I couldn’t
keep up.” Especially if it’s done live. If you have recorded — you might want to
think about that. Do I do live? Do I do video recording? These are pros and cons too. Live
observation requires somebody to be there right then. And the video recording has added
flexibility. You can watch it any time, it can be replayed. So you have to think about
that, too. What I’ve observed in the real implementation
is sometime the experimenter feels very conscious about being watched. They might feel anxious.
“Am I doing the right thing?” The researcher has to be careful about how to approach this.
This is not about “big brother is watching you.” It’s more about, “We’d like for you,
as the experimenter to do the best job possible to deliver the intervention. So, we’d like
to work with you and give you feedback, as you proceed with doing this. And we can help
you do better, if needed.” You kind of phrase it like, “We are in this together. We’re trying
to delivery the best intervention we can. Let’s make it happen.” So that kind of anxiety
goes away. There’s a progression of research. There are
many people who have written about this in our field. Initially, you want to have really
good control, and really good treatment integrity. So that’s the primary objective — you want
to have it as perfect as possible. But then, as you go into real practice, there are all
kinds of constraints imposed on implementing an intervention that, it’s different from
a study, as you know. There, it becomes sometimes important to do a study with less integrity.
The treatment fidelity sort of becomes the independent variable that you’re manipulating.
Can the same treatment outcomes be obtained by having less perfect implementation? Because,
assuming the clinician is not a robot, and has to be responsive to what happens with
the clients, you want them to be more flexible. But can you still obtain the same outcomes?
You should follow this progression in measurement. Do multiple studies, and hopefully get to
the point where we can reach outcomes in real-life settings with real-life expectations with
what is reasonable in terms of fidelity.

Leave a Reply

Your email address will not be published. Required fields are marked *