Alternative specification of defaults?

[tyarkoni]

Per discussion on twitter, blogs, etc., there have been concerns that the default specification of the alternative hypothesis isn't appropriate for many domains of psychology. @richarddmorey has pointed out that it was intended to cater primarily to typical effect sizes in cognitive psychology. But the package is now being used increasingly widely outside of cognitive psychology, often by people who aren't well versed in Bayesian hypothesis testing, and end up sticking with the defaults because they're not comfortable making other selections. The result is that it's easy for naive users to end up making claims that are at odds with common sense expectations.

For example, in personality psychology, effects on the order of r = .1 to .2 are very common (equivalent to d's of maybe .2 - .4). A naive user with n = 80 is unlikely to ever reject the null in favor of the alternative under such circumstances when using the default alternative, and may not realize that they're testing what is (for the domain) probably an inappropriate model.

It would be good to have some way of easily adjusting the default to meet different researchers' needs in a way that balances the need for simplicity and ease of use with the reality of changing expectations across domains. Here are some suggestions:

• Make the arguments for the alternative hypothesis mandatory, but prominently list suggested defaults in the documentation. These could then be tailored to different domains. This seems like the nuclear option, but would at least force users to own their decisions, rather than pleading ignorance later on once a paper is actually published.
• Add named values of 'small' and 'verysmall' for rscale, equivalent to (say) 0.3 and 0.1. This would be the easiest change, but also probably not terribly effective since it provides no indication of what 'small' means across different domains.
• Add an extra 'domain' argument that effectively changes the defaults and/or meaning of the named rscales. The default could be set to 'cognitive', so that users would get the current behavior by default. But other values (e.g., 'personality', 'social', 'epidemiology', etc.) would result in modified defaults. This seems like a nice balance between preserving the current user experience and still making it clear to the average user that they may want to use something different. Alternatively, one could use labels based on approach rather than domain ('experimental', 'correlational', 'epidemiological', etc.).

I'd be curious to hear opinions from @richarddmorey, @rouderj, and users of the package at varying levels of expertise.

[richarddmorey]

Another option I've considered is a "wizard" (ugh) interface that allows interactive, graphical, setting of the prior. A shiny applet would be possible (see eg the shiny app linked at the end of this post: http://bayesfactor.blogspot.co.uk/2014/02/bayes-factor-t-tests-part-2-two-sample.html)

[tyarkoni]

Oh, that would be even better--though obviously much more work. :) I'd be happy to submit a PR for any of the above suggestions, but not sure I could take on writing a full-blown wizard at the moment. A cheaper approach along the same lines would be to skip the GUI element and just have a text-based wizard that asks two or three questions and then makes a recommendation. The nice thing about either approach is that is that it could be completely unobtrusive, since if a user manually specified the alternative, the wizard would never pop up--or, users who knew what they were doing could just disable it once via a package-level setting.

[richarddmorey]

Another advantage of the wizard approach is that it could respect a more modular design. I have another package (https://github.com/richarddmorey/BayesFactorExtras) which is for extra stuff besides the main computation of the Bayes factors, and that would be a natural home for it. Given the other features of that package (really nice printing of the Bayes factors and MCMC objects) it will, I think, become a must-have package, so that wouldn't be relegating it to being ignored just because it is separate.

[rouderj]

Can I reply directly to this thread or do I have to log on somewhere?

Jeff Rouder
Middlebush Professor
Dept. of Psychological Sciences
University of Missouri

email: [email protected]
twitter: @JeffRouder
web: pcl.missouri.edu

On Jul 30, 2015, at 10:42 AM, Tal Yarkoni [email protected] wrote:

Oh, that would be even better--though obviously much more work. :) I'd be happy to submit a PR for any of the above suggestions, but not sure I could take on writing a full-blown wizard at the moment. A cheaper approach along the same lines would be to skip the GUI element and just have a text-based wizard that asks two or three questions and then makes a recommendation. The nice thing about either approach is that is that it could be completely unobtrusive, since if a user manually specified the alternative, the wizard would never pop up--or, users who knew what they were doing could just disable it once via a package-level setting.

—
Reply to this email directly or view it on GitHub.

[richarddmorey]

We can see that, but remove your signature before you reply :)

[rouderj]

I think by-and-large that it is a good idea to encourage ppl to think about their choices. I suspect over time that we will have nonlocal prior options as well. So, whatever gets done, it should be built with expandability. Also, are the scales part of the BF structure? I am sure whatever we do it will be used more strategically than judiciously, but that is life in psychology.

[richarddmorey]

Yes, scales are part of the BF structure (there is a whole part of the object specifying the prior structure). The priors are defined here, in this function: https://github.com/richarddmorey/BayesFactor/blob/master/pkg/BayesFactor/R/common.R#L175

Note that it is trivial to add more labels (as in 2). It's just a switch statement.

[Joe-Hilgard]

The chief concern is that end users may not stop to consider whether the prior applied is appropriate. To me, this is chiefly an educational, rather than a software, concern.

Perhaps a brief, catchy disclaimer would help to educate -- something along the lines of Simmons et al.'s 21-word disclaimer for transparent research:

"We explain our choice of priors under each hypothesis, detailing how their scale and location are appropriate to the research question and study design. The reader is cautioned that obtained Bayes Factors will differ proportionately to changes in the priors, and very different priors may yield different results. Bayes factors are answers to questions. Asking the right question requires choosing appropriate priors."

Reviewers and editors could ask "Oh, would you add the boilerplate Bayes disclaimer?" as a matter of course on all manuscripts containing Bayesian model comparisons.

[rouderj]

Not "proportionately," more like "modestly."

Sent via the Samsung GALAXY S®4, an AT&T 4G LTE smartphone

-------- Original message --------
From: Joe Hilgard [email protected]
Date:07/30/2015 11:54 AM (GMT-06:00)
To: richarddmorey/BayesFactor [email protected]
Cc: "Rouder, Jeffrey N." [email protected]
Subject: Re: [BayesFactor] Alternative specification of defaults? (#63)

The chief concern is that end users may not stop to consider whether the prior applied is appropriate. To me, this is chiefly an educational, rather than a software, concern.

Perhaps a brief, catchy disclaimer would help to educate -- something along the lines of Simmons et al.'s 21-word disclaimer for transparent research:

"We explain our choice of priors under each hypothesis, detailing how their scale and location are appropriate to the research question and study design. The reader is cautioned that obtained Bayes Factors will differ proportionately to changes in the priors, and very different priors may yield different results. Bayes factors are answers to questions. Asking the right question requires choosing appropriate priors."

Reviewers and editors could ask "Oh, would you add the boilerplate Bayes disclaimer?" as a matter of course on all manuscripts containing Bayesian model comparisons.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/63#issuecomment-126400669.

[Joe-Hilgard]

Well, again, Tal's concern is chiefly about people using ~Cauchy(.707) when they should be using ~Cauchy+(.1). I think that would be a pretty hefty difference.

[tyarkoni]

@Joe-Hilgard, I think it's optimistic to think this is an education issue--or at least, that can be easily solved with a few sentences of explanation. It's probably safe to assume that most users will read little if any of the docs, and will do no more than scan the argument list before trying to run a t-test. So I think a wizard of some kind and/or arguments with options that call out to be set appropriately (e.g., 'experimental' vs. 'correlational', etc.) is preferable to relying on docs and/or warning messages--though those would still be good to have as well. I guess a boilerplate warning message that shows up every time you run a test might work, but that seems pretty obnoxious.

[richarddmorey]

I just pushed a commit to the branch flexible-prior that allows specification of prior scales on a per-effect basis (rather than the per-effect-type basis that was previously added for convenience). This will go a ways to allowing more flexibility.