A reminder on standards of evidence

[John]

The purpose of this post is to remind the CCC community of the standards for evidence in medical outcomes research, because IMHO these same standards should apply in some present discussions.

The standards of outcome research in medicine require that investigators (1) specify in advance the outcome criteria, (2) specify in advance the evaluation methods, (3) apply the methods to multiple cases simultaneously, (4) use more than one evaluator, (5) blind the evaluators to the identity of the cases, and (6) publicly disclose the criteria and the results.

All of the above are essential ... they serve to restrain the (inevitable) prejudices and passions of investigators ... and the medical literature shows clearly that to ignore even one principle leads swiftly to pseudoscience.

That is why, in outcome studies where these principles are not followed, there is a presumption that the results *are* pseudoscience.

These principles are not easy to follow ... they are a *lot* of extra work ... but it *is* possible to follow them ... and in some of present CCC discussions, these principles *should* have been followed in the past ... and they *should* be followed in the future.

The literature on Bible Codes provides a lesson in the disastrous consequences of ignoring these principles.

[bob]

The purpose of this post is to remind the CCC community of the standards for evidence in medical outcomes research, because IMHO these same standards should apply in some present discussions.

The standards of outcome research in medicine require that investigators (1) specify in advance the outcome criteria, (2) specify in advance the evaluation methods, (3) apply the methods to multiple cases simultaneously, (4) use more than one evaluator, (5) blind the evaluators to the identity of the cases, and (6) publicly disclose the criteria and the results.

All of the above are essential ... they serve to restrain the (inevitable) prejudices and passions of investigators ... and the medical literature shows clearly that to ignore even one principle leads swiftly to pseudoscience.

That is why, in outcome studies where these principles are not followed, there is a presumption that the results *are* pseudoscience.

These principles are not easy to follow ... they are a *lot* of extra work ... but it *is* possible to follow them ... and in some of present CCC discussions, these principles *should* have been followed in the past ... and they *should* be followed in the future.

The literature on Bible Codes provides a lesson in the disastrous consequences of ignoring these principles.

What does that have to do with the current discussion? We are not doing a clinical trial, nor a double-blind drug evaluation. We are trying to decide if B was copied in part from A. There is but one way to pull this off, and that is to compare A with B to see if there are too many identical parts to explain away as "pure luck".

[John]

The purpose of this post is to remind the CCC community of the standards for evidence in medical outcomes research.

What does that have to do with the current discussion?

Bob, when a person's reputation and livelihood are stake---when accusations of scientific misconduct and business misconduct are made---the similarity to medicine is strong indeed.

Many of the people involved in present CCC discussions are highly intelligent and immensely experienced. But soberingly, the medical literature suggests that neither intelligence nor experience provides reliable protection against pseudoscientific reasoing. If anything, high intelligence makes both physicians and patients *more* prone---not less---to self-delusion.

In medicine, all students learn that nothing is more valuable than well-designed protocols, strong evidence, solid documentation, and careful peer review. And nothing is more harmful than ad hoc protocols, retrospective assessments, scanty or delayed documentation, and evaluation by public opinion.

[chrisw]

The purpose of this post is to remind the CCC community of the standards for evidence in medical outcomes research.

What does that have to do with the current discussion?

Bob, when a person's reputation and livelihood are stake---when accusations of scientific misconduct and business misconduct are made---the similarity to medicine is strong indeed.

Many of the people involved in present CCC discussions are highly intelligent and immensely experienced. But soberingly, the medical literature suggests that neither intelligence nor experience provides reliable protection against pseudoscientific reasoing. If anything, high intelligence makes both physicians and patients *more* prone---not less---to self-delusion.

In medicine, all students learn that nothing is more valuable than well-designed protocols, strong evidence, solid documentation, and careful peer review. And nothing is more harmful than ad hoc protocols, retrospective assessments, scanty or delayed documentation, and evaluation by public opinion.

Absolutely correct. And, considering the danger to the accusers of having their wallets surgically removed, one would have thought they'ld like to apply higher standards themselves, too.

[Terry McCracken]

The purpose of this post is to remind the CCC community of the standards for evidence in medical outcomes research.

What does that have to do with the current discussion?

Bob, when a person's reputation and livelihood are stake---when accusations of scientific misconduct and business misconduct are made---the similarity to medicine is strong indeed.

Many of the people involved in present CCC discussions are highly intelligent and immensely experienced. But soberingly, the medical literature suggests that neither intelligence nor experience provides reliable protection against pseudoscientific reasoing. If anything, high intelligence makes both physicians and patients *more* prone---not less---to self-delusion.

In medicine, all students learn that nothing is more valuable than well-designed protocols, strong evidence, solid documentation, and careful peer review. And nothing is more harmful than ad hoc protocols, retrospective assessments, scanty or delayed documentation, and evaluation by public opinion.

Absolutely correct. And, considering the danger to the accusers of having their wallets surgically removed, one would have thought they'ld like to apply higher standards themselves, too.

This is the height of absurdity. Also there is no danger that Dr. Hyatt et al can be sued.

What colour is the sky in your world?

[bob]

The purpose of this post is to remind the CCC community of the standards for evidence in medical outcomes research.

What does that have to do with the current discussion?

Bob, when a person's reputation and livelihood are stake---when accusations of scientific misconduct and business misconduct are made---the similarity to medicine is strong indeed.

Many of the people involved in present CCC discussions are highly intelligent and immensely experienced. But soberingly, the medical literature suggests that neither intelligence nor experience provides reliable protection against pseudoscientific reasoing. If anything, high intelligence makes both physicians and patients *more* prone---not less---to self-delusion.

In medicine, all students learn that nothing is more valuable than well-designed protocols, strong evidence, solid documentation, and careful peer review. And nothing is more harmful than ad hoc protocols, retrospective assessments, scanty or delayed documentation, and evaluation by public opinion.

Absolutely correct. And, considering the danger to the accusers of having their wallets surgically removed, one would have thought they'ld like to apply higher standards themselves, too.

This is the height of absurdity. Also there is no danger that Dr. Hyatt et al can be sued.

What colour is the sky in your world?

dark, dark brown.

[bob]

The purpose of this post is to remind the CCC community of the standards for evidence in medical outcomes research.

What does that have to do with the current discussion?

Bob, when a person's reputation and livelihood are stake---when accusations of scientific misconduct and business misconduct are made---the similarity to medicine is strong indeed.

Many of the people involved in present CCC discussions are highly intelligent and immensely experienced. But soberingly, the medical literature suggests that neither intelligence nor experience provides reliable protection against pseudoscientific reasoing. If anything, high intelligence makes both physicians and patients *more* prone---not less---to self-delusion.

In medicine, all students learn that nothing is more valuable than well-designed protocols, strong evidence, solid documentation, and careful peer review. And nothing is more harmful than ad hoc protocols, retrospective assessments, scanty or delayed documentation, and evaluation by public opinion.

It doesn't fit here. We are in the hospital, the patient is in the bed, we are trying to figure out what to do. Clinical trials and such do not apply. We are looking at the evidence as it is presented. The only benefit we have is that nobody is going to die if we don't reach a conclusion quickly. But this is the _right_ place to have the discussion, if we could just weed out the extraneous noise and have people look at the data carefully and draw conclusions...

[chrisw]

The purpose of this post is to remind the CCC community of the standards for evidence in medical outcomes research.

What does that have to do with the current discussion?

Bob, when a person's reputation and livelihood are stake---when accusations of scientific misconduct and business misconduct are made---the similarity to medicine is strong indeed.

Many of the people involved in present CCC discussions are highly intelligent and immensely experienced. But soberingly, the medical literature suggests that neither intelligence nor experience provides reliable protection against pseudoscientific reasoing. If anything, high intelligence makes both physicians and patients *more* prone---not less---to self-delusion.

In medicine, all students learn that nothing is more valuable than well-designed protocols, strong evidence, solid documentation, and careful peer review. And nothing is more harmful than ad hoc protocols, retrospective assessments, scanty or delayed documentation, and evaluation by public opinion.

Absolutely correct. And, considering the danger to the accusers of having their wallets surgically removed, one would have thought they'ld like to apply higher standards themselves, too.

This is the height of absurdity. Also there is no danger that Dr. Hyatt et al can be sued.

What colour is the sky in your world?

dark, dark brown.

Careful, chicken licken, all that "bullshit" will fall on your head