An information reel every Australian needs to see.
The 13-minute information reel below reveals details of how Australia’s peak medical research body, the National Health & Medical Research Council (NHMRC), set about reviewing the evidence on a natural therapy it had no prior subject, clinical or research expertise in.
This is not a story about homoeopathy – the topic could be anything. It’s a story about truth an integrity, how a government agency used taxpayer money and leveraged its statutory power and reputation to “assure” the public that it conducted a “transparent” process using “standardised, accepted methods”.
You be the judge.
A formal investigation into the NHMRC Homeopathy Review has uncovered multiple issues associated with bias, conflicts of interest, unreported expert advice, misreporting and procedural and scientific irregularities – many of which are not mentioned in the information reel above. These have been referred to the Commonwealth Ombudsman as a formal Complaint, with the Your Health Your Choice petition calling for a Senate Inquiry.
The Executive Summary to the Ombudsman Submission of Complaint can be accessed here.
Between 2012 and 2015 the NHMRC conducted reviews of 17 natural therapies. These reviews concluded there was ‘no evidence’ for any of these therapies and were used by government to justify removing Private Health Insurance rebates for natural therapies.
This was part of a political process, instigated in 2012 by the then Federal Treasurer, who flagged potential ‘savings’ by removing rebates for natural therapies. The Department of Health was charged with overseeing the Natural Therapies Review to justify the policy and subcontracted the NHMRC to conduct the reviews. The reviews did not involve any research or subject experts for any natural therapy, did not retrieve or assess any original research papers and did not conduct any targeted cost-effectiveness, safety or quality assessment.
The NHMRC had already initiated the Homeopathy Review in 2012 and incorporated it into the Natural Therapies Review ‘to avoid duplication of effort’. This review was the first cab off the rank and informed the approach then taken to reviewing 16 other natural therapies in 2013.
Ten reasons why there is a Senate petition & Ombudsman Complaint:
1. Review published in 2015 was the SECOND attempt:
The NHMRC has not disclosed to the Australian public that it produced two reports, one in July 2012 and the one released to the public in March 2015.
A first review conducted between April and August 2012 was terminated within days of the contractor submitting their final Draft Report. The existence of this taxpayer funded review, its findings, methodology and reasons for termination have not been disclosed. NHMRC remains tight-lipped.
Around the world, people are demanding that NHMRC release the first taxpayer funded report. What’s there to hide?
2. First review was conducted by a reputable & highly experienced reviewer:
The first review was conducted by a highly experienced and reputable reviewer and a principal author of NHMRC’s own ‘best practice’ guidelines on how to review health evidence. The reviewer was also a key developer of NHMRC’s internationally recognised ‘dimensions of evidence’ methodology, widely recognised and used by NHMRC and other groups to assess health evidence.
3. First review was of good methodological quality:
Freedom of Information (FOI) documents have revealed expert peer reviewer advice praising the high standard of the first review. In July 2012, Homeopathy Working Committee member Prof Fred Mendelsohn advised NHMRC,
“I am impressed by the rigor, thoroughness and systematic approach given to this evaluation [….] Overall, a lot of excellent work has gone into this review and the results are presented in a systematic, unbiased and convincing manner.”
The reviewer’s contract was terminated less than three weeks later. Why?
4. Second review – unreported, midstream changes to the research protocol:
The research protocol (the rules) for the second review was agreed and finalised in December 2012, but was never published – only revealed through FOI.
The second contractor, OptumInsight, completed its literature search and evidence review in March 2013 according to the agreed research protocol.
Between April and July 2013, the NHMRC established a special ‘Sub-Group’ that step-by-step re-created the research protocol. None of the changes made were reported or disclosed, despite the Optum Overview Report including a ‘Changes to the Research Protocol’ section (Optum Overview Report, p.20). The existence of the Sub-Group process was also not reported.
5. Use of arbitrary criteria – created & applied retrospectively:
NHMRC used a method that has never been used in any other review, before or since, by any other research group or agency. NHMRC decided that for trials to be ‘reliable’ they had to have at least 150 participants and reach an unusually high 100% threshold for quality (5/5 on the Jadad or equivalent scale). The following was also not disclosed or reported:
- These rules were applied in July 2013 – seven months after the original protocol had been ‘agreed and finalised’
- These changes were not reported – even though ethical scientific processes must always report all changes to the research protocol, to safeguard against bias and manipulation of the results
- The impact of these changes on the Review’s findings were also not reported: together, they dismissed the results of 171 out of the 176 included trials from contributing to the Review’s published finding of ‘no reliable evidence’. That is, the published findings are based on only 5 trials
- One of the 5 ‘reliable’ trials was still positive, but was excluded from Table 1 of the NHMRC Information Paper that presented the findings to the public. Instead, it is substituted with a negative trial in the same condition (lower back pain) that was not one of the 176 trials
- NHMRC itself routinely conducts studies with less than 150 participants and no other NHMRC-published evidence review has ever applied a minimum trial size threshold for ‘reliability’. The cut-off is unprecedented and without any scientific justification; why no scientific standard or guideline exists to authenticate it.
The NHMRC revealed under Senate scrutiny in 2018 that, “At the time this work was underway there was no relevant guidance or standard endorsed by NHMRC, or a relevant international organization, on the development and content of evidence statements” and that the criteria it used were, “drafted over a number of months following the completion of the overview search for literature”.
6. No original studies assessed & ‘assumptions’ made about missing data
The Optum Overview did not retrieve or assess any original studies – unprecedented in NHMRC reviews. Instead it relied exclusively on secondary sources (systematic reviews), most of which (88%) were judged to be poor quality due to factors such as missing data (e.g. unreported study quality) and inaccurate reporting (e.g. study sample sizes). Why? To save time and money (see NHMRC FAQ document & Administrative Report).
Where a study’s quality was not reported, instead of reading the original study to check NHMRC created the special rule: “If quality of the included studies was not reported in the systematic review then those studies were assumed to be poor quality”(Information Paper, p.37).
This ‘assumption’ impacted a substantial proportion of the 176 studies – but how many is not reported. Instead, the 40-page NHMRC Information Paper definitely concludes there is ‘no good quality studies with enough participants’ multiple times and refers to ‘quality’ over 100 times.
7. No subject or research experts:
NHMRC guidelines and standards mandate that its review and guideline development processes involve subject and clinical experts in the topic under review – for obvious reasons. Despite having no prior knowledge, experience or expertise in homeopathy, no research or subject experts were included on the expert committee.
NHMRC actively resisted and ignored protests and repeated requests from expert groups that at least one expert be included.
8. Hidden expert peer reviewer advice:
In 2013, the NHMRC contracted the Australasian Cochrane Centre (ACC) to peer review the methodology for the second review. The ACC identified a number of unaddressed methodological issues and advised:
“when a substantial proportion of small (but good quality) studies show significant differences, […] ‘no reliable evidence’ does not seem an accurate reflection of the body of evidence.”
NHMRC also did not disclose further key expert peer reviewer advice received in 2014, despite publishing an ‘Expert review comments’ document. For example (revealed through FOI):
“The dismissal of positive systematic reviews compounded with the lack of an independent systematic review of high quality randomised controlled trials leaves me uncertain of the definitive nature of the Report’s conclusions.”
“High quality RCTs with narrow confidence intervals … should have been searched for and included in this review. Systematic reviews (SRs) have considerable weaknesses as reliable sources of evidence.”
9. Conflicts of interest:
In 2012, the NHMRC appointed a member of Australia’s leading anti-natural medicine lobby group, Friends of Science in Medicine (FSM), as the Chair of the Homeopathy Working Committee (HWC) to oversee the review. He did not initially declare his conflict, which when discovered was not formally managed.
NHMRC Council (chaired by the NHMRC CEO) and the Health Care Committee overseeing the work of the HWC contained a number of FSM members during the course of the review. No conflicts were declared or managed at any stage.
In 2014, NHMRC appointed a FSM member as a contractor to review additional evidence submitted during public consultation. The conflict was also not declared or managed.
In 2011, before any evidence had been reviewed, the NHMRC Chairman publicly declared,
“Let me assure you I am no supporter of homeopathy. As Chairman of NHMRC I can also assure you that NHMRC does not support homeopathy.”
In 2014, the former NHMRC CEO publicly declared his ‘annoyance’ that complementary and alternative medicine treatments, which he referred to as ‘unethical’ and a ‘waste of money’, attracted health fund rebates. In April 2015 the CEO publicly declared:
“But it’s an entirely different matter when people who are ill with a treatable illness are pushed therapies that don’t work and, in fact, are often implausible, pushed by practitioners who we must assume either believe in magic or perhaps are just dishonest. Ill health has attracted charlatans since time immemorial. Snake oil merchants wanting to take your money by promising false hope.”
A year earlier, FSM had written to the CEO urging him to, “ensure that Australians are not sold snake oil”.
The Homeopathy Review was the first of 17 natural therapies reviews conducted by the NHMRC and informed the approach taken to all of them. These reviews have been used to justify removal of Private Health Insurance rebates for natural therapies – impacting all Australians.
Sign the petition below calling for a Senate inquiry and help protect integrity in government and right of choice in healthcare.