A clearly written and well argued account of an appropriate application of the scientific method of enquiry to an emotionally charged area of our lives (and possible avoidable death!).
I found the examples of (type 1 and type 2) errors – in understanding the results of flawed research – chilling.
As a result I hope, in the future, to be even more aware of the possible pitfalls and the potential for bias in both the design and reporting of health related research. Thank You.
This book is an essential building block for health literacy everywhere.
It can be a powerful catalyst for fruitful conversations in medical consultations.
I hope that many of its readers will describe its effects on their health care – on this site and elsewhere.
We welcome all of your comments and feedback!
Please tell us about useful resources that help explain the importance of testing treatments.
I like the pictures of content that change on the home page. Is there any way to make these clickable so someone could click through to something that catches their interest as it flashes up?
Thanks Amanda, if you click on the image below the text it links you to the relevant TTextra
I can’t find the link to download the 2nd edition. can you please post it? All I can find is links to the hypertext edition.
Thanks for point this out JD, I have now rectified matters.
The PDF of Testing Treatments is now available to download here: http://www.testingtreatments.org/the-book/download-the-book/
Thanks for the link to the download!
Nice to hear the Testing Treatments book recommended by Ben Goldacre on Radio 4 today:
Thanks for this great link, Andre. Interesting to see how the same book can get such a difference reaction! (For anyone else who wants to listen to the review but not the whole programme the discussion starts at 20 minutes.) This website addresses many of the comments of Michelle Paver.
This is a great initiative. Like Andre, I heard about it on Radio 4 yesterday. Ben’s recommendation, although audacious, made a great change from the obscure & tedious books brought to our attention by some of previous guests over the years.
If you are getting the right treatment you feel effective you will probably not be here. It is those who need research & more assistance who are found on Forums such as Healthunlocked & one linked to care 2 USA [ Inspire?]
Helplines of Charities run by the non medical on corporate remits or a Nurse are a complete waste of time -assistance needs to be given often urgently .
Just a few comments in response to a recent query regarding the apparent discrepancy in length between the Chinese and English versions of Testing Treatments (1st edition):
(1) The Chinese version does not include the cover, Contents and Acknowledgement pages and the foreword by Ben Goldacre (which constitute the first 11 pages of the English version), nor does it include the References, Additional resources and Index (which constitute the last 21 pages of the English version).
(2) In addition to the difference in font size, the written Chinese text may be more concise (space-wise, although I’m certainly not a linguistic expert…) than the written English text. For these two versions, one page of text in Chinese can cover approximately two and a quarter pages of text in English without losing information. Therefore, taking into account the omission of contents at the front and end of the English version from the Chinese version (as stated above), the number of pages for the Chinese version seems reasonable.
(3) The only things that appear to be missing from the main text of the Chinese version (that I could identify after a page-by-page comparison) are the text boxes ‘The Which? verdict on health screens’ in Chapter 2 and ‘Two cheers for uncertainty’ in Chapter 4 of the English version. Other headings, text boxes and texts seem to agree well between the two versions. Like any translated work, the Chinese version may have minor loss of fidelity in some sentences, but overall I would have no hesitation to recommend the Chinese version to a Chinese reader, who can always consult the original English version if/when necessary.
I hope this helps.
Thanks for your advice. The second edition(Chinese) is going to come up, which will cover all contents that appear in English version.
That’s great news for Chinese readers. Look forward to seeing the new edition!
The Chinese version is now available. Please select “Chinese” from the drop-down menu under “Translations of this website”, above right.
Just a couple of suggestions:
1. Please change the name from TT extras to “blog”. I don’t think anybody knows what TT extras means. Everyone knows what a blog means.
2.I would love to see an RSS feed fr the articles. So it is easy to find updates in my google reader. More like a pull technique.
3.Also a facebook page where people can “like”.So any artcil or updates can be seen in their facebook feed.
Thanks for all the hard work!
2.I would love to see an RSS feed fr the articles. So it is easy to find updates in my google reader. More like a pull technique
Many thanks for these suggestions, which we will take into account. Another think we are looking at is making the linkage between TTextras and the book content a little more direct.
However, I think it’s a great idea to explicitly provide a blog. The problem is getting people to write it!
I can do one per month on the topic of your choice in a public friendly format using TT as a base. Please let me know if this would be useful
Comment for James McCormack. A good video “Some studies…”. One of the studies cited, the one about salt restriction, could be improved (or removed). In fact in that review, patients were given such an amount of furosemide that everyone would bet that with a salt restriction regime patients will suffer -for sure- a hiponatremia. The authors of the trials included in this review gave in fact 3 or more times the recommended amount of furosemide!!. Aplicability of the conclusions are then flawed. The methodology of the SR is ok, they follow PRISMA, they do anything for a sound paper, except the usual dosage of diuretics…
Hi Jose – thanks so much for your comment – I think about it this way.
1) There really is no recommended dose of furosemide – the “recommend” dose is the dose that, via titration by the clinician or patient, controls excess fluid/weight – Regardless, they state “approximately two-thirds of patients received furosemide 50 mg twice daily (comparable with a commonly seen 40 mg twice daily dosing in the community)”
2) Let’s assume that the doses in these studies are higher than those typically seen – how does that “flaw” a study – in my opinion it doesn’t. All that issue does is limit it’s generalizability to people who are on those doses.
3) Unless there is a relatively well described reason why these specific doses of furosemide somehow were the reason why the people in the salt restriction group had the magnitude seen for increased all cause mortality (RR 1.95, 95% CI 1.66 to 2.29), sudden death (RR 1.72, 95% CI 1.21 to 2.44), death due to HF (RR 2.23, 95% CI 1.77 to 2.81) and HF readmissions (RR 2.10, 95% CI 1.67 to 2.64) I’m not sure that I’m too concerned about the dose of furosemide.
3) What is presented in this review is the “best” available evidence that we have – “flawed” as it is. Unless there is RCT evidence that salt restriction actually improves outcome I would lean towards using this evidence rather than ignoring it because it is “flawed”. The bottom line is that anyone who recommends salt restriction in HF is doing it based on opinion – the lowest form of evidence and ignoring much higher levels of evidence.
This is a very interesting and useful resource – any plans to introduce it as part of the teaching materials at universities? Medical students may find this very enlightening – get ‘em while they’re young!
Many thanks for your kind words.
I completely agree with you, students are absolutely a key group and we’d love to hear your (or anybody else’s) ideas on what we can to do to get TTi onto their radar.
At the moment we are talking to a broad range of potential users to find out more about how to make TTi useful to them, and this includes educators as well as health professionals, information workers, journalists, bloggers and other groups.
So now’s the time to tell us what you think!
Science AAAS is using Google video and Discus for interactive panel workshops. ThinkWell could perhaps work with you to set up an interactive Public Led Online Trial to road test learning. The public often gets complex papers thrown at the activists to shut them up and they have no way like CASP or TTI to vet the paper. People learn by doing. Recently I had a trust contact me who had no idea what an SR really was!
Hi, I love this Testing Treatments website, it has great content and I would blog post excerpts and links on our website. I can’t because older really valuable posts are lost. Same when I want to email someone so they can check it out. For instance I looked for one Post on the benefits of brain training but could not find it even through search. There is very little information via the menus so it discourages repeat visits. You could fix this with a location page where you list everything and not just latest blogs or even with a visible sitemap. Hope this can be fixed but great book to download in multiple languages, thanks for this!
We don’t have any posts about brain training; that’s probably why you didn’t find it!
Thanks for the suggestions though. We are listening to what people say and doing hands-on user research at the moment, so we will take this into account.
It was actually a post that included a reference to using steroids post brain truama. I wanted to find this paper and use it as I know physicians that have done this research since the study which is a waste of resources . The patient opinion data that no one took any account of reported this intervention as unhelpful. I was tying this in with a large brain training study that was done with inadequate not appropriately targeted training which received great press for its not significant findings but if you use inaccurate tools the results are not going to be stellar. If they had done an unbiased review they would have seen this. Some research does more harm than good….a link to the paper would be great!
Ah OK. Is this the one you mean?
It’s in the part of the site that deals with sample size. Both references are on that page.
Search for “steroid” found it.
LOL Helps to have the right word! Thanks very much. I think I came across it while looking for sample size info and lost both again. I got to comments again by clicking feedback as clicking on your comment took me your profile. The entire article was excellent and easy reading.
Excellent book. Currently integrating it into our new pharmacy curriculum.
Thanks Anthony. We’d be really interested to hear how you get on with that, and any suggestions you have to make the content more useful for your students.
I wonder if the trials do not automatically help “the majority” who react positively to a treatment. Every person and his/her metabolism is different so maybe big trials disadvantage the “rare metabolisms” as for them no treatments are developed? Thanks for comments.
I believe this is a great resource for patient and public to understand the importance of the clinical process
This website is really helpful, but it is not sufficiently on the radar. I have only stumbled upon it. I have been struggling with doctors fobbing me off with “It will be fine. The risk is low.” and refusing to go into detail. I then have to go off and spend quite some time googling relevant research (although I’m not a scientist, let alone a medical person) to make up my own mind and satisfy myself that the supporting evidence is there. Unfortunately I am now faced with a proposed treatment where it seems to me that the uncertainties and controversies have not been resolved. Indeed I have found out that a research review is underway hoping to clarify things. The doctor is playing it as if there is no uncertainty at all. I have printed off “Addressing uncertainties about the effects of treatment” and “Providing treatment as part of a fair test” as a structure for discussion next time, which will hopefully be a more intelligent conversation. I do hope your website will help doctors to have more “grown up” conversations with people who are intelligent enough to handle it and express an interest (by asking too many difficult questions!). More needs to be done to identify us as a subset who need a different sort of conversation. I know lots of people need the “touchy-feely” support that is widely available, but medical practitioners are actually making the stress worse for those of us who don’t need that, but do need a grown up conversation that isn’t afraid of discussing risk and uncertainty and guiding us to an independent rational decision, mindful of the evidence and its limitations, and not just going along with “doctor knows best”. Keep up the good work!
Many thanks FTF! Let us know how you get on; we’re delighted to know that our materials are helpful to you and will be very interested to hear how they went in your consultation.
What do you suggest if you find misconduct within a Cochrane group ? I’ve read ‘Bad Pharma’ and am ploughing thru’ Deadly Medicines but there’s been no suggestion as to who watches the watchers.
I would document it and contact the Cochrane team http://www.cochrane.org/contact/. Have you done this?
In terms of “who watches the watchers”, we all do. Labels and authority in and of themselves don’t guarantee quality. We should always think critically about any health claims.
In the case of Cochrane Collaboration, their methods are transparent as are their findings (or they are supposed to be anyway). This makes it easier to think critically about their output than pretty much any other source of information I can think of.
Douglas, I’ve received no reply regarding concerns raised offline to Minervation so they’re now publicly posted at FnMyalgia.com
Marzipan, did you contact me at Minervation? I didn’t receive anything from you.
Dear Douglas, Iain, Amanda and Ben
We are holding an afternoon workshop for consumers and parents at Westmead Hospital in Sydney, Australia on Saturday 13 June. If anyone has any friends or colleagues who may happen to be close by on that day, they are welcome to come along, free. Extracts from the flyer appear below. Please tell your friends in Australia. We’ll let you know how it goes!
CLINICAL TRIALS ARE AMAZING !
An Intelligent Person’s Guide to Fair Tests of Any Treatment
Trials have transformed millions of lives and saved billions of dollars – by showing which treatments work and which treatments don’t. Over the last 30 years, the chances of healthy survival for premature babies – and for people with asthma, heart attacks, cancer and many other diseases – have dramatically improved, thanks to vital evidence from hundreds of trials.
Yet, trials are usually an ‘optional extra’ – not part of routine care. Too often no trials are available for patients to join.
AS A RESULT, LESS THAN 1 IN 20 PATIENTS TAKE PART.
You can help change this at a free workshop at Westmead Hospital on Saturday afternoon, 13 June. It’s an entertaining introduction to the amazing world of clinical trials – see program.
For background, see http://www.testingtreatments.org/
To register visit page http://bit.ly/1S3RZoB
or Google ‘Westmead International Update 2015’ and click on the link
You can download the program at http://ctc.usyd.edu.au/wiu
Tell your friends in Australia. We hope to see some of them there!
All the best
WINNER Centre for Newborn Research
NHMRC Clinical Trials Centre, University of Sydney
Thanks for letting us know. The event looks great and I hope it goes well.
Some of your participants may be interested in Generation R, a UK network for involving young people in research: http://www.generationr.org.uk
We’d be particularly keen to share any resources you have that others could use to improve public understanding of clinical trials.
Keep us posted!
What does “The sample of patients can be referred to a known population” even mean ??
It’s about whether the group of people who take part in a study (the sample) is representative of the wider population in whom the treatment would be used in real life.
Check out the Jargon buster on the right for more info.
But we’ll have a think about rephrasing that question in the light of your comment!
Thanks for the speedy and helpful response. I agree that your interpretation is the most plausible; that’s what I presumed, and was surprised to see the first offered answer marked incorrect.
The problem is in the ambiguity in the question. If you want to leave the first answer as being incorrect, the question needs to state specifically that patients are allocated “…either to receive the new treatment or the comparison…” at random. In that case, the randomisation does indeed only deal with bias in allocation to the trial arms, and may suffer from generalisability issues, thus making the first answer incorrect.
But as it stands, the question suggests randomisation has been applied to the RCT as a whole, including selection for participation as well as allocation to trial arm. Such randomisation is specifically designed to achieve generalisability to the wider population, making the first answer correct.
Anyhow, thanks for a very entertain quiz. Am sure a lot of people will be surprised by some of the (undoubtedly) spot-on answers – like “don’t trust results just because they appear in a prestigious journal”.
Good point. The problem seems to be the term “allocation” really. It’s easier to think it as assigning *exposures* to people at random, but you can think it as assigning people to exposures (other way around) (as in Jargon buster).
Anyway, the standard usage is that “allocation” is not about sampling but what happens after that, and randomization *of sampling* is not that common terminology, I guess, either but randomization refers also only to exposures.