critical appraisal session – reflection

for 2 x librarians, 4 x Dr, 1 x OT, 2 x lay members of ethics committees


To provide participants with a structured basis for critically appraising quantitative research papers.


By the end of the session participants will be able to:

•Understand the principles of critical appraisal

•Know about the resources that will help you

•Critically appraise a piece of research as a group

Plan of the  1/2 day:

    • Start with introductions & expectations – a bit of history about me, and a clarification that I’m not a statistician and that if it takes a statistician to understand clinical research then I bet there’s more than just me in hot water!, also disclaimer that I’m not a clinician, so I might skip over some aspects of a paper that are terribly interesting to the group from a clinical view point, but that from the point of view of the paper are perhaps not that important.
    • Go round table asking about experience, and expectations for the day (mix of drs & OTs & lay ethics group members)
    • Start with a quiz – mind gym after lunch to get them refreshing/thinking about research methodology and terminology
    • Work through quiz, getting people to suggest reasons for correct answers, and to suggest reasons why an answer might be incorrect. For the OR (odds ratio) section, draw a blank forest plot and ask group to draw on diagram where the CIs would lie, and therefore why a result might be significant or not. Trying to encourage theose with stronger skills to explain to those with more rudimentary understanding – helps them refresh, and encourages idea that I don’t have all the answers, and that colleagues and books might help
    • Break
    • Actually do the critical appraisal of a paper using CASP RCT checklist – small enough group that we worked through it as a whole group, rather than dividing into groups. They were already well socialised, and were happy to speak up and raise areas where they were unsure or unclear. We spent quite a long time on the first 3-4 questions, and so had to rush a bit through the final questions, but did manage to squeeze in final round up
    • Info on the other sources of help – websites, books , practice!

Reflections – 2nd half was very rushed – I perhaps spent too much time raising areas for them to consider. Perhaps the group hadn’t read the paper so very thoroughly in advance, and certainly one had only received it on the day (late applicant)

Good things

  • quiz
    • raised lots of issues early on, and encouraged group to share and discuss reasons for correct answers with each other. Lay member raised issue that sometimes the quantitative slant of the questions wouldn’t be the most important sort of research to help/influence patients – the qualitative stuff was just as important (sometimes more so) when considering hierarchy of evidence.
    • Interactive and engaging but not confessional – nobody knew who answered what, so it allowed people to discuss without shame/ embarrassment
  • Critical appraisal
    • Again, very group oriented – lots of discussion and explanation from the group,
    • Questions raised from lots of different perspectives
    • Point raised later that having a mix of “dispassionate” reviewers plus clinical reviewers helped to tease apart some issues, helped focus the mind on the paper as opposed to the clinical issues which might actually just be a difference of opinion
  • Flipchart
    • Liked that I drew diagrams to give visual presentations of some of the stats, rather than relying on verbal explanation
    • Liked that members of the group drew too, and contributed.

Bad things/ areas for improvments

  • time – very rushed at the end – I’d spend too long asking them to look in more depth at the population
  • fairly stuffy room – glad I put in a break half way through, but it could have been a little shorter, which would have given more time
  • should have perhaps done a little more introduction to some of the main themes/terminologies before the quiz to (could use a prezi??) to soften the blow for those for whom it really was almost the first introduction
  • should have remembered the clickything to move the powerpoint on, which would have meant I could have moved around or been sitting more (less of a teacher?), rather than standing
  • didn’t refer to the books or get any of the group to look up the “answer” in the books, though a couple of the doctors were interested in the books we’d brought along
  • given that some of the group were very much more “begininer” than I’d realised, so it was pretty harsh to drop them in quite so hard – so need to get much better understanding of who’s coming along to be able to pitch it at the right level – bit it did make it an opportunity for the rest of the group to test/demonstrate their understanding of p-value/confidence intervals etc – if you have to explain what it means, then it can highlight just how well you understand the topic!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: