The School of Data Science is excited to announce the appointment of Jeffrey D. Blume as the Quantitative Foundation Associate Dean of Academic and Faculty Affairs, effective July 25. Learn more: t.ly/kqpv
Jeffrey Blume
Jeffrey D. Blume is an American data scientist and biostatistician. Since 2026 he has been the dean of the University of Virginia School of Data Science. More on Wikipedia
-
Jeffrey Blume reposted
UVA School of Data Science
@uvadatascience
-
Jeffrey Blume
@StatEvidence
Thanks for the highlight!
Richard Stevens
@ebhcmedstats
and this by Fletcher Mercaldo & @StatEvidence academic.oup.com/biostatistics/article/21/2/236/5092384
-
Jeffrey Blume reposted
Vanderbilt University Data Science Institute
@VUDataScience
Our DS Teams class, taught by @cassyld, partnered with industry client @TheGeneralAuto for a final project to create a predictive modeling approach for their needs. Our first place prize goes to Ali, Yasi, Lakshay, and Drew! Runners up: Joey, Umang, Wenying, and Baiyu. Congrats!
-
Jeffrey Blume reposted
Simon Vandekar
@VandyAtVandy
... is asymptotically unbiased under model misspecification.
2/2 -
Jeffrey Blume reposted
Simon Vandekar
@VandyAtVandy
Cohen's "d" won't cut it? Try our robust effect size index, it has the same formula regardless of model (e.g. LM, GLM, GLMER), can accommodate covariates, and ...
open access link rdcu.be/b3jLt
with @StatEvidence @dragontanrao
1/2 -
Jeffrey Blume
@StatEvidence
Exciting Opportunity!
Vanderbilt University Data Science Institute
@VUDataScience
What makes the Vanderbilt M.S. in Data Science program unique? Check out our new video! As a reminder, the deadline to apply is this Saturday, February 15th.
This media couldn't be displayed. See it on the original post. -
Jeffrey Blume reposted
Dr. Melinda Aldrich, MPH, PhD
@MelindaAldrich
Happy to announce the release of our work evaluating lung ca screening guidelines in African Americans. Diversity matters! @SandlerRhodes Evaluation of USPSTF Lung Cancer Screening Guidelines Among African American Adult Smokers jamanetwork.com/journals/jamaoncology/fullarticle/2737091
-
Jeffrey Blume reposted
Vanderbilt University Data Science Institute
@VUDataScience
WE'RE HIRING! Looking for Data Scientists of all levels (chief, senior, entry). Come join us in launching our new trans-institutional Data Science Institute at @VanderbiltU. vanderuniv.taleo.net/careersection/.vu_cs/jobdetail.ftl?j…
-
Jeffrey Blume reposted
Abraham E. Gracia R.
@Abraham_RMI
If you want to understand why not use P-values to compare the basic characteristics of the groups to be compared in a clinical trial, then you have to read this MAGNIFICENT twetorial. 👇🏼👇🏼👇🏼
Andrew Althouse
@ADAlthousePhD
TOPIC: P-Values In Table 1 of RCT's. Time to revisit this poll.
-
Jeffrey Blume
@StatEvidence
Very excited about this!
-
Jeffrey Blume
@StatEvidence
Deep learning versus linear regression with higher order terms. Seems like we are seeing more and more instances where these two approaches are often functionally equivalent for prediction.
David Baranger
@David_Baranger
Linear regressions with 2nd/3rd order polynomials were equal to or better than neural-nets (aka 'deep learning') in every data set they looked at.
This post is unavailable. See it on X. -
Jeffrey Blume reposted
Boback Ziaeian 🤦🏻♂️
@boback
Fixed it for you. Nice work! Look forward to more statistical tweetorials @raj_mehta @StatEvidence x.com/i/moments/1009133356943007744
-
Jeffrey Blume
@StatEvidence
22/ The lesson: Likelihood error rates, even when expanded to data-dependent hypotheses, are readily controllable. Simulation code here: github.com/StatEvidence/website/blob/master/RS LR sims.R The burden now on critics (like @learnfromerror) to catch up with the leading edge.
-
Jeffrey Blume
@StatEvidence
21/ Thanks for reading! Here is a paper that shows how to take advantage of this idea for planning and monitoring clinical trials. arxiv.org/abs/1711.01527 Its got the theoretical results but is much easer to read. Questions welcome.
-
Jeffrey Blume
@StatEvidence
20/ The likelihood approach tries to leverage good science. There is no point in rejecting mu=0 when the data support atls that are practically zero anyway. This simple, sensible idea eliminates a very large number of Type I errors when repeated looking at data sequentially.
-
Jeffrey Blume
@StatEvidence
19/ These intervals bounce around zero, and every so often they barely exclude zero, triggering our rule maxLR>8 & we stop. The super massive important point is that these data are NOT evidence against mu=0, but rather the opposite. All relevant alts (under likelihood) are null.
-
Jeffrey Blume
@StatEvidence
18/ To be concrete, here is what happens: By 650 observations, the 1/8 likelihood support interval (the width of the likelihood, think of it as a CI) is +/- 0.08, by 1130 it is down to 0.06 and by 2056 it is down to 0.04. But since the null is true, the MLE is very near 0 itself.
-
Jeffrey Blume
@StatEvidence
17/ The picture illustrates what is happening. Plotted are likelihood functions that trigger stopping at sample sizes 650, 1130, 2056. These were the 25th, median, 75th stopping time percentiles. The likelihoods are nearly a spike at zero. These data favor the null.
-
Jeffrey Blume
@StatEvidence
16/ Then frequentist police bust in: “Tisk Tisk. Look at that error rate! 42%. You’ve been bad. Time out in the thinking chair.” But in the thinking chair it becomes clear. The problem is not error control; its that small meaningless deviations are being over-interpreted.
-
Jeffrey Blume
@StatEvidence
15/ Ex #3: Now expand further to any alt mu> 0.05*SD. If we see that any mu>0.05*SD yields a LR>8 at any point, we stop. Then, P( max LR>8 for some n=1,2,3…)=0.492. This is way inflated over the fixed alt=0.05*SD case of P( LR>8 for some n=1,2,3…)=0.12.
-
Jeffrey Blume
@StatEvidence
14/ Ex #2: Expand the set of eligible alternatives to any mu> 0.5*SD . If we see that any mu>0.5*SD yields a LR>8 at any point, we stop. Then,
P( max LR>8 for some n=1,2,3…)=0.16. Reasonable, but inflated over the fixed alt=0.5*SD case of P( LR>8 for some n=1,2,3…)=0.093. -
Jeffrey Blume
@StatEvidence
13/ Stop and think. We’ve expanded the alternative space from one alt hyp (mu=1SD) to a very very large set of alts (any mu>1SD) & the inflation in P(Err) - with infinite looks - is tiny at only 0.018. Why? Because most of the Type I errors occur at alts close to the null.
-
Jeffrey Blume
@StatEvidence
12/ To start, we’ll use any mu> 1SD as our alternative set. If we see that any mu>1SD yields a LR>8 at any point, we stop. Sims show that P( max LR>8 for some n=1,2…)=0.088. This is only a small increase over the 2 Hyp. case where P( LR>8 for some n=1,2…)=0.07 when alt=1SD.
-
Jeffrey Blume
@StatEvidence
11/ There is a bound on P( maxLR>k for some n=1,2,…) but its super ugly. Too ugly for twitter. If anyone wants to know what this is, you are welcome to read my PhD thesis github.com/StatEvidence/website/blob/master/blume_dissert… with apologies in advance. I’m gonna avoid PTSD and just simulate some examples.
-
Jeffrey Blume
@StatEvidence
10/ For a data-dependent alternative the stopping condition changes from LR>k to maxLR>k, where the max is taken over the set of eligible alternatives. So we want to know P( maxLR>k for some n=1,2,…) . This heavily depends on the set of eligible alternatives, e.g. all mu>1SD
-
Jeffrey Blume
@StatEvidence
9/ Let’s consider a set of alternative hypotheses, and if any alt in the set is better supported over the null we will stop. This is a very flexible approach; referred to as using a data-dependent alternative. Now P(Err) will go up, but - surprise! - it is still very manageable.
-
Jeffrey Blume
@StatEvidence
8/ A common criticism of likelihood methods is that they depend only on two simple hyps and this is why p(Err) behaves. This is not a fair criticism; all the nice properties of hypothesis tests come in the simple vs simple case too. But it gets raised, so let’s address it.
-
Jeffrey Blume
@StatEvidence
7/ There are many interesting things going from a foundational & frequentist point of view. The take home message is that the likelihood error rate does get inflated with looks at the data, but still remains controllable. More here github.com/StatEvidence/website/blob/master/Blume_tepee.pdf But wait! There is more
-
Jeffrey Blume
@StatEvidence
6/ So this is different. The Likelihood Type I Error converges to zero as the sample size grows (like Type II error). So P(LR>k for some n=1,2,…) gets inflated slower because individual contributions at each sample size are zeroing out. This is why the bound holds. See graph.
-
Jeffrey Blume
@StatEvidence
5/ Analytical formula are available for these quantities See github.com/StatEvidence/website/blob/master/Blume2002-Tut… . What is interesting about this situation is that
P(LR>k) < P(LR>k for some n=1,2,…) <1/k
As the number of looks grow, so does the P(err), but even so it remains bounded by 1/k. -
Jeffrey Blume
@StatEvidence
4/ How often do we see a large LR that supports the alt. hyp. when the null hyp. is true? For a fixed sample size study with no interim looks, we write P(LR>k). For a sequential study (looking at the data after each observation) we write P(LR>k for some n=1,2,…)
-
Jeffrey Blume
@StatEvidence
3/ When the LR is large enough, we have evidence supporting mu=1 SD over mu=0. This condition is written as LR>k where k is some constant, to be general. Usually we take k=8 or k=20, and there is literature to support this is reasonable. (but off topic for this tweettorial).
-
Jeffrey Blume
@StatEvidence
2/ Likelihood ratios measures the strength evidence for one simple hypothesis over another. Let the LR be the likelihood ratio for an alternative mean, say mu=1 SD, to the null mean, say mu=0. (here I’ll scale everything by SD so I don’t have to specify what the SD is).
-
Jeffrey Blume
@StatEvidence
1/ I’ll keep the discussion as non-technical as possible. and I’ll take it slow. First we need data! We’ll use a simple case: sampling from the standard normal distribution so observations are x1,x2,x3,… from N(0,1). Pretend we can sample from N(0,1) for as long as we want.
-
Jeffrey Blume
@StatEvidence
0/ Tweettorial: Let’s take a look at what happens when likelihood ratios are used to repeatedly examine accumulating data. I get many questions about this. Likelihoodists claim they can look as often as they want and still control the P(Type I Error). Let’s see how they do it.
-
Jeffrey Blume
@StatEvidence
Warning: A tweettorial is coming on likelihood methods and repeated looks at data. Its 22 tweets long. Apologies if not of immediate interest.
-
Jeffrey Blume
@StatEvidence
It is never a cop-out to pay attention to the scale of the data and its scientific relevance. It is a cop-out to regurgitate age-old criticisms without understanding how they pertain to the relevant arguments of today.
♕Deborah Mayo♕
@learnfromerror
Blume repeats Savage's cop-out: change the problem (which is caused by allowing a data-dependent alt,or an optional stopping rule) into one where the alternative is predesignated. The likelihoodist has at most a pre-data error control that's then flouted by data dependent alt.
This post is unavailable. See it on X. -
Jeffrey Blume
@StatEvidence
At least read the tweet you are quoting. I reporting the case with a data-dependent stopping rule (a flexible one) and even in that case the error probability was well controlled. You really need to be careful here. This example is exactly what you were claiming could not happen
♕Deborah Mayo♕
@learnfromerror
Blume repeats Savage's cop-out: change the problem (which is caused by allowing a data-dependent alt,or an optional stopping rule) into one where the alternative is predesignated. The likelihoodist has at most a pre-data error control that's then flouted by data dependent alt.
This post is unavailable. See it on X. -
Jeffrey Blume
@StatEvidence
Fundamentals lost here. Balance needed for proper scientific inference. If no balance, then adjust statistically. Randomization is tool that tends to produce approximate balance. It is not a scientific principle itself. It does not need to be implemented perfectly & often is not.
Abraham E. Gracia R.
@Abraham_RMI
This is a scandal. And if this happened in @NEJM , what can we expect from other journals? CC @davidcnorrismd @statsepi @StatEvidence @f2harrell @medevidenceblog
This post is unavailable. See it on X. -
Jeffrey Blume
@StatEvidence
I’m not convinced of this. It all depends on what is done after you ‘see’ inbalances. If one tinkers too much with the model then perhaps. But adding a few unbalanced covariates to a model is going to be fine. Randomize and ‘close your eyes’ is never good.
Frank Harrell
@f2harrell
But note that in a randomized trial, unless there is an accusation of cheating, even the act of examining covariate balance leads to nothing but bad statistical practice.
-
Jeffrey Blume
@StatEvidence
The only way to really protect against this is to use a regression model and include key covariates, allow for non-linearity and interactions. This is partly why @f2harrell advocates for modeling. Table 1 can look balanced and the groups might still not be comparable.
Jeffrey Blume
@StatEvidence
Nice blog, but focus on univariate balance is misplaced. Two groups of 100 could be balanced on age and gender but grp 1 has 50 young women and 50 old men while grp 2 has 50 old women and 50 young men. These groups are not ‘balanced’. www.ncbi.nlm.nih.gov/m/pubmed/15559341/
Felix Thoemmes
@FelixThoemmes
New blog post on the always controversial topic of balance in covariates in randomized trials.
www.felixthoemmes.com/blog/balance-checks/ -
Jeffrey Blume
@StatEvidence
Nice blog, but focus on univariate balance is misplaced. Two groups of 100 could be balanced on age and gender but grp 1 has 50 young women and 50 old men while grp 2 has 50 old women and 50 young men. These groups are not ‘balanced’. www.ncbi.nlm.nih.gov/m/pubmed/15559341/
Felix Thoemmes
@FelixThoemmes
New blog post on the always controversial topic of balance in covariates in randomized trials.
www.felixthoemmes.com/blog/balance-checks/ -
Jeffrey Blume
@StatEvidence
Looks like a good list for stimulating discussion in biostat grad programs.
Felix Thoemmes
@FelixThoemmes
Just compiled reading list for next semester's journal club. Topic: "Myths"
-
Jeffrey Blume
@StatEvidence
Here is a short summary of prediction modeling using regression. The downside is that it is limited to regression only and does not discuss any of the popular alternatives.
Cecile Janssens
@cecilejanssens
New version of our introduction on Prediction Research is online. It's a 'short' easy-read about the main topics in prediction research, the course manual of our class at @EmoryRollins www.cecilejanssens.org/wp-content/uploads/2018/05/Predict…
-
Jeffrey Blume
@StatEvidence
Oh yea. Written terribly is an understatement. But it’s not at all uncommon to build prediction models without using p-values. And in many cases this is very reasonable. Just not for the reasons stated here.
♕Deborah Mayo♕
@learnfromerror
Not only are p-values terribly explained, it says they shouldn't be used because they are altered by multiple testing. I thought one of the biggest problems was ignoring multiple testing & pretending they don't alter the assessment. (I'm speaking to data scientists next week.)
This post is unavailable. See it on X. -
Jeffrey Blume
@StatEvidence
Sloppy, common & costly scientific mistake. A NS pv means the results are inconclusive, not in support of the null hypothesis. Second-generation p-values solve this problem. A SGPV of 1 means the data support the null.
Frank Harrell
@f2harrell
Yet again, @NEJM turns "no significant difference" into a conclusion. Is it too painful to just state the truth about a "negative" frequentist statistical analysis (we need more data)?
This post is unavailable. See it on X. -
Jeffrey Blume
@StatEvidence
Interesting stuff.
Ferenc Huszár
@fhuszar
Judea Pearl claims all we do in ML is curve fitting. I wrote this post to explain that claim and introduce the basics of causal inference to ML folks.
Machine Learning beyond Curve Fitting: An Intro to Causal Inference and do-Calculus
www.inference.vc/p/f01e8d56-5117-482e-8a87-82dec692f14d/ -
Jeffrey Blume
@StatEvidence
There is nothing stopping them from post-hoc summarizing the evidence using a likelihood or Bayesian approach. It may shed some light on what is actually going on in the data.
Andrew Althouse
@ADAlthousePhD
There is something very disturbing about a trial with HR=0.76, 95% CI 0.55-1.04 for the primary endpoint being stopped early for futility. IMO this falls partly on the frequentist statistical paradigm and trials designed to a certain “power” level with guessed effect size.
-
Jeffrey Blume
@StatEvidence
This is a major problem and it’s one reason we set about developing our second-generation p-value, which resolves this very problem. journals.plos.org/plosone/article?id=10.1371/journal.pone…
Andrew Althouse
@ADAlthousePhD
"Nevertheless, at least one important conclusion can be drawn — the routine use of ECMO in patients with severe ARDS is not superior to the use of ECMO as a rescue maneuver in patients whose condition has deteriorated further"
-
Jeffrey Blume
@StatEvidence
Excellent Tweetorial! Controversial statistics & ECMO go way back. Question: It sounds like the pvs are adjusted for multiple looks. True? If so, part of the problem could be the adjustment itself. The futility outcome might not loom so large with out those adjustments.
Andrew Althouse
@ADAlthousePhD
This article calls for a TWEETORIAL (h/t to the clever @boback and @ihtanboga for catching this and posting about it)
This post is unavailable. See it on X. -
Jeffrey Blume reposted
Cristina
@biologeek
Thanks a lot! (For the code but also the #openaccess / #openminded attitude even more)
-
Jeffrey Blume
@StatEvidence
Amazing. The Log-rank test can give answers that appear unintuitive because of its implicit weighting scheme and the null. Second-generation p-value are called for here. Example attached from journals.plos.org/plosone/article?id=10.1371/journal.pone…
Robert W. Yeh
@rwyeh
The disconnect between the text and the graph below is almost enough to make you lose all hope for frequentist interpretations. That sound you here is @f2harrell's brain exploding.
h/t: @ADAlthousePhD @boback -
Jeffrey Blume
@StatEvidence
good analogy
Raj Mehta
@raj_mehta
I prefer swimming/lifeguard analogy myself.
If your a beginner swimmer, it's fine to start learning on your own in shallow waters.
But dont go into the deep end w/o a lifeguard. Even if you make it, a mistake could be disastrous. No need to chance it. -
Jeffrey Blume
@StatEvidence
Perhaps, except exercise routines are not published & recommend as a new way to improve other’s health. Truth be told, people don’t always follow best exercising practices either and they hurt themselves, are inefficient or ineffective. A personal trainer would help many.
David Robinson
@drob
Saying "no one should analyze data without consulting a statistician, they could misinterpret the results" is like saying "no one should exercise without a personal trainer, they could injure themselves."
-
Jeffrey Blume
@StatEvidence
Completely agree. False models can have higher predictive accuracy than true models. This is partly why @f2harrell stresses the science when building models. Another reference is arxiv.org/pdf/1101.0891.pdf
Oliver Maclaren
@omaclaren
This is still an underappreciated point - prediction and truth aren’t the same thing. We should be more explicit about which we care about most and when.
This post is unavailable. See it on X. -
Jeffrey Blume
@StatEvidence
And, of course, it’s important to remember that low Type I error is a big driver of low FDR. So the control is need experimentally, it’s just not what you want to report once the study is done.
David Colquhoun on Bluesky 🇺🇦 💙
@david_colquhoun
type 1 error control is not what's needed. It's the false positive risk that you need to control, IMO
arxiv.org/abs/1802.04888 -
Jeffrey Blume
@StatEvidence
Agree the false discovery rate is important to assess. But you also have to know what the ‘result’ is. Can’t just report the FDR. Solution seems to be reporting SGPV and the FDR that it is ‘correct’.
David Colquhoun on Bluesky 🇺🇦 💙
@david_colquhoun
type 1 error control is not what's needed. It's the false positive risk that you need to control, IMO
arxiv.org/abs/1802.04888 -
Jeffrey Blume reposted
Ken Rothman
@ken_rothman
2nd Generation P-values. Interesting thread by @val_welty
This post is unavailable. See it on X. -
Jeffrey Blume
@StatEvidence
Proud to report our paper on Second-generation p-values reached 5,000 views today (32% downloads). Published on 3/22. Teaser slide attached. @LucyStats @f2harrell journals.plos.org/plosone/article?id=10.1371/journal.pone…
-
Jeffrey Blume
@StatEvidence
Great slides!
Karl Broman
@kwbroman
Psyching myself up to talk #dataviz to MD-PhD students. I’ve given this talk like 30 times, but still ~nervous~. www.biostat.wisc.edu/~kbroman/presentations/graphs2017.pdf
-
Jeffrey Blume
@StatEvidence
Yes it's nice. A fair amount written about it too. Connection with profile likelihood, can be viewed as a constrained profile likelihood
Oliver Maclaren
@omaclaren
& if not iid/exchangeable then no issue either - just have effective sample size of one. So I guess I find empirical likelihood quite prom.
-
Jeffrey Blume
@StatEvidence
Yes and no. As long as the hypothesis has meaning under model failure you get two different approximations to the 'true' likelihood
Oliver Maclaren
@omaclaren
Which I suppose means (for eg empirical likelihood) that the models are different for two different samples, since parameters=weights on obs
-
Jeffrey Blume
@StatEvidence
Related in the limit. Robust adj like and EL will (should?) end up at same place in large samples.
Oliver Maclaren
@omaclaren
I'll take a look. How does it compare to eg empirical likelihood? Logical limit is to take empirical distribution as max likelihood no?
-
Jeffrey Blume
@StatEvidence
Or could be a non-comparative assessment. But seems like a basic function of statistics in a rudimentary situation is missing.
Jeffrey Blume
@StatEvidence
Agree. But it's a shame there is no consensus on when a set of data supports one simple hyp over another and by how much. Basic need I think
-
Jeffrey Blume
@StatEvidence
The language of data and uncertainty is statistics. If we want competency in this realm, then we need to teach the language.
♕Deborah Mayo♕
@learnfromerror
Yes many proclaim they're among the first to out bad stat, & while they're at it, they favor methods at odds w/key tools to avoid cheating!
Post not available here, view on X
Jeffrey Blume
@StatEvidence
Agree. But it's a shame there is no consensus on when a set of data supports one simple hyp over another and by how much. Basic need I think
-
Jeffrey Blume
@StatEvidence
No one worries about the frequency properties of boxplots or adjusting them for multiple comparisons or looks. They are treated as a summary
♕Deborah Mayo♕
@learnfromerror
Unfortunately, that's EXACTLY what's at issue in the debates, & some of us refuse to reduce the lang. to comparative likelihoods or beliefs.
-
Jeffrey Blume
@StatEvidence
One reason I like the likelihood approach is that it is very visual. I often refer to the likelihood as the boxplot of the evidence.
♕Deborah Mayo♕
@learnfromerror
Unfortunately, that's EXACTLY what's at issue in the debates, & some of us refuse to reduce the lang. to comparative likelihoods or beliefs.
-
Jeffrey Blume
@StatEvidence
My understanding of what investigators are asking for tends to lead me away from yes/no answers toward descriptive approaches.
♕Deborah Mayo♕
@learnfromerror
Unfortunately, that's EXACTLY what's at issue in the debates, & some of us refuse to reduce the lang. to comparative likelihoods or beliefs.
-
Jeffrey Blume
@StatEvidence
And to me, I don’t see the hypothesis testing approaches meeting the demands of investigators. At least, my experience is that they do not.
♕Deborah Mayo♕
@learnfromerror
Unfortunately, that's EXACTLY what's at issue in the debates, & some of us refuse to reduce the lang. to comparative likelihoods or beliefs.
-
Jeffrey Blume
@StatEvidence
Fine to skip likelihoods or beliefs, but the product of an alternate framework must be applicable to how statistics is used in practice.
♕Deborah Mayo♕
@learnfromerror
Unfortunately, that's EXACTLY what's at issue in the debates, & some of us refuse to reduce the lang. to comparative likelihoods or beliefs.
-
Jeffrey Blume
@StatEvidence
I agree language is an important part of the debate; especially since what is said and what is implied is often different in application.
♕Deborah Mayo♕
@learnfromerror
Unfortunately, that's EXACTLY what's at issue in the debates, & some of us refuse to reduce the lang. to comparative likelihoods or beliefs.
-
Jeffrey Blume
@StatEvidence
Statistical education has not been a priority for too long. Much of the sloppiness is not purposeful, but the result of poor training.
♕Deborah Mayo♕
@learnfromerror
Yes many proclaim they're among the first to out bad stat, & while they're at it, they favor methods at odds w/key tools to avoid cheating!
-
Jeffrey Blume reposted
Oliver Maclaren
@omaclaren
Side question - do you see sample size as part of data or part of model (ie parameter)?
-
Jeffrey Blume
@StatEvidence
Strong Vandy presence at #wsds2017 . Great to see.
Gabriela de Queiroz
@gdequeiroz
-
Jeffrey Blume
@StatEvidence
4/4 Fisher clearly understood that the measure of the strength of evidence is different from the propensity for evidence to be misleading.
-
Jeffrey Blume
@StatEvidence
3/4 …should not be confused with the force, or cogency, of such evidence." Fisher (1959)
-
Jeffrey Blume
@StatEvidence
2/4 Fisher: “In fact, as a matter of principle, the infrequency with which, in particular circumstances, decisive evidence is obtained, …
-
Jeffrey Blume
@StatEvidence
1/4 Given the recent discussion, I thought this quote from Fisher (1959) is on point. Fisher is not…um…concise. So more than 140 chars..
-
Jeffrey Blume
@StatEvidence
i/i But if statistics can’t separate these things then they can’t be properly assess, or even combined, later on for science.
♕Deborah Mayo♕
@learnfromerror
not an advance, the separation is what they've always been pushing, and I say it's a fatal mistake (for sci evidence)
-
Jeffrey Blume
@StatEvidence
h/i Agree that proper scientific inference uses both strength of evidence in data and the false discovery rate. Just not error probs.
♕Deborah Mayo♕
@learnfromerror
not an advance, the separation is what they've always been pushing, and I say it's a fatal mistake (for sci evidence)
-
Jeffrey Blume
@StatEvidence
g/i When error rates are high, it is possible to still observe strong evidence in the correct direction. It just does not happen often.
♕Deborah Mayo♕
@learnfromerror
not an advance, the separation is what they've always been pushing, and I say it's a fatal mistake (for sci evidence)
-
Jeffrey Blume
@StatEvidence
f/i And after seeing those 3 numbers match, the new odds (given 3 #s) measure the inferential uncertainty. New odds are analogous to FDRs.
♕Deborah Mayo♕
@learnfromerror
not an advance, the separation is what they've always been pushing, and I say it's a fatal mistake (for sci evidence)
-
Jeffrey Blume
@StatEvidence
e/i Lottery is good example. Odds of winning are low, but you have strong evidence for winning when 3 numbers are revealed and they match
♕Deborah Mayo♕
@learnfromerror
not an advance, the separation is what they've always been pushing, and I say it's a fatal mistake (for sci evidence)
-
Jeffrey Blume
@StatEvidence
d/i So in that case you can’t (by definition) have high error rates, get lucky and see strong evidence in right direction. But in life u can
♕Deborah Mayo♕
@learnfromerror
not an advance, the separation is what they've always been pushing, and I say it's a fatal mistake (for sci evidence)
-
Jeffrey Blume
@StatEvidence
c/i Failure to distinguish them is a conceptual error. If not, then evidential metric and error prob must move together in same direction.
♕Deborah Mayo♕
@learnfromerror
not an advance, the separation is what they've always been pushing, and I say it's a fatal mistake (for sci evidence)
-
Jeffrey Blume
@StatEvidence
b/i Its the tail area probability that represents error props and the strength of observed evidence and is misinterpreted as the FDR.
♕Deborah Mayo♕
@learnfromerror
not an advance, the separation is what they've always been pushing, and I say it's a fatal mistake (for sci evidence)
-
Jeffrey Blume
@StatEvidence
a/i Essential breakthrough IMHO. The fundamental problem of stat. inference is that it uses one quantity or # to represent three concepts
♕Deborah Mayo♕
@learnfromerror
not an advance, the separation is what they've always been pushing, and I say it's a fatal mistake (for sci evidence)
-
Jeffrey Blume
@StatEvidence
5/5 The main control we’ve shown is due to error rates going to 0 quickly so the accumulated inflation is bounded and controlled.
♕Deborah Mayo♕
@learnfromerror
I thought he pushed it backward frankly, what do you mean?
-
Jeffrey Blume
@StatEvidence
4/5 BTW likelihood is not immune to error rate inflation for many looks for comparisons. It has those things and controlls them well.
♕Deborah Mayo♕
@learnfromerror
I thought he pushed it backward frankly, what do you mean?
-
Jeffrey Blume
@StatEvidence
3/5 And because of this, it is much cleaner in how it deals with data and inference. No conflation of error rates and strength of evidence.
♕Deborah Mayo♕
@learnfromerror
I thought he pushed it backward frankly, what do you mean?
-
Jeffrey Blume
@StatEvidence
2/5 I don’t think this an “error” framework as you describe it, because it doesn’t use error probs to measure the evidence. LR does it.
♕Deborah Mayo♕
@learnfromerror
I thought he pushed it backward frankly, what do you mean?
-
Jeffrey Blume
@StatEvidence
1/5 He/we showed that using LR to measure the strength of evidence was reliable, efficient, and seldom misleading. More so than HT frankly.
♕Deborah Mayo♕
@learnfromerror
I thought he pushed it backward frankly, what do you mean?
-
Jeffrey Blume
@StatEvidence
Apologies @learnfromerror for coming twitter storm. Category 3. Key conceptual diffs needed space to explain. Appreciate chance to discuss.
-
Jeffrey Blume
@StatEvidence
Regardless, the real advance is what I call the evidential framework. Its the disentangling of the evidence measure and error probs.
♕Deborah Mayo♕
@learnfromerror
I thought you were talking likelihood, now it sounds like you're an error statistician assessing error probs
Jeffrey Blume
@StatEvidence
This is the path forward for anyone who wants to measure evidence with any tool. They need: evidence measure, error props, False disc rates
Jeffrey Blume
@StatEvidence
Regardless, the real advance is what I call the evidential framework. Its the disentangling of the evidence measure and error probs.
♕Deborah Mayo♕
@learnfromerror
I thought you were talking likelihood, now it sounds like you're an error statistician assessing error probs
-
Jeffrey Blume
@StatEvidence
A straw man likelihood ignores freq. props. There was early thinking about Freq. Props. But Royall and students ;) really pushed it forward.
♕Deborah Mayo♕
@learnfromerror
I thought you were talking likelihood, now it sounds like you're an error statistician assessing error probs
-
Jeffrey Blume
@StatEvidence
Point is freq. props have always been an essential part of likelihood methods. Pratt in ’62 discussion on LP points out the universal bound.
♕Deborah Mayo♕
@learnfromerror
Likelihoods ALWAYS work that way, you're trying to redefine the approach appealing to error probs- conflict w/ what likelihoodists argue
-
Jeffrey Blume
@StatEvidence
The solution that minimizes the sum of Type I and II error rates is to use the LL, as Savage pointed out long ago. Lets errors go 2 0 too.
♕Deborah Mayo♕
@learnfromerror
Likelihoods ALWAYS work that way, you're trying to redefine the approach appealing to error probs- conflict w/ what likelihoodists argue
-
Jeffrey Blume
@StatEvidence
Criteria for robustness and handling nuisance parameters was based on freq properties. Behavior of LL in sequential trials was my area. etc.
♕Deborah Mayo♕
@learnfromerror
Likelihoods ALWAYS work that way, you're trying to redefine the approach appealing to error probs- conflict w/ what likelihoodists argue
-
Jeffrey Blume
@StatEvidence
Just to be clear: Nearly all of Royall’s later students worked on evaluating frequency properties of LL. Its the modern view of LL.
♕Deborah Mayo♕
@learnfromerror
Likelihoods ALWAYS work that way, you're trying to redefine the approach appealing to error probs- conflict w/ what likelihoodists argue
-
Jeffrey Blume
@StatEvidence
Most statisticians believe in assessing Freq. properties (even most Bayesians). But those properties don’t describe evid in observed data.
♕Deborah Mayo♕
@learnfromerror
Likelihoods ALWAYS work that way, you're trying to redefine the approach appealing to error probs- conflict w/ what likelihoodists argue
-
Jeffrey Blume
@StatEvidence
Modern likelihoodists (Royall, myself) are all about controlling freq properties. We just don’t use those properties to measure the evidence
♕Deborah Mayo♕
@learnfromerror
Likelihoods ALWAYS work that way, you're trying to redefine the approach appealing to error probs- conflict w/ what likelihoodists argue
-
Jeffrey Blume
@StatEvidence
No, no conflict at all. The point of Royall’s book was that LL was better frequentist paradigm than HT or ST. LL minimizes total error rate+
♕Deborah Mayo♕
@learnfromerror
Likelihoods ALWAYS work that way, you're trying to redefine the approach appealing to error probs- conflict w/ what likelihoodists argue
-
Jeffrey Blume
@StatEvidence
Fixing the Type I error rate actually breaks the nice natural frequency properties of LRs. I alluded to this before. Natural errors go 2 0.
♕Deborah Mayo♕
@learnfromerror
Likelihoods ALWAYS work that way, you're trying to redefine the approach appealing to error probs- conflict w/ what likelihoodists argue
-
Jeffrey Blume
@StatEvidence
All data dependent hypotheses are composite hypothesis, by definition. They change with data. That is how they are understood in statistics.
♕Deborah Mayo♕
@learnfromerror
No, it's a particular data dependent alt, if you want to change 60 yrs of how it's been understood, then you're just changing the meaning
Jeffrey Blume
@StatEvidence
Those types of hypotheses don’t make any predictions. You use data to set the Hyp and then test the Hyp against same data. Very EB-like.
Jeffrey Blume
@StatEvidence
All data dependent hypotheses are composite hypothesis, by definition. They change with data. That is how they are understood in statistics.
♕Deborah Mayo♕
@learnfromerror
No, it's a particular data dependent alt, if you want to change 60 yrs of how it's been understood, then you're just changing the meaning
-
Jeffrey Blume
@StatEvidence
And vice-versa I suppose. Fine, then lets compare their frequency properties. I’d still take the LL approach; better freq. properties.
♕Deborah Mayo♕
@learnfromerror
hadn't seen this. Yes I concur the two approaches don't reconcile,that's why ours shouldn't be judged by yours
-
Jeffrey Blume
@StatEvidence
Although, with this description, no one would really consider the Jolly Green Giant or a Giraffe, as a suspect…
♕Deborah Mayo♕
@learnfromerror
No, inference (in our account) is not comparative, & not to pts. Consider ordinary life: evid killer ≥ 6ft-for an ex off the top of my head.
-
Jeffrey Blume
@StatEvidence
I hear what you are saying; just might be the two approach don’t reconcile.
♕Deborah Mayo♕
@learnfromerror
No, inference (in our account) is not comparative, & not to pts. Consider ordinary life: evid killer ≥ 6ft-for an ex off the top of my head.
-
Jeffrey Blume
@StatEvidence
From probabilistic view, this is problematic. If probabilities can’t be computed for a Hyp, then how to reason with uncertainty is not clear
♕Deborah Mayo♕
@learnfromerror
No, inference (in our account) is not comparative, & not to pts. Consider ordinary life: evid killer ≥ 6ft-for an ex off the top of my head.
-
Jeffrey Blume
@StatEvidence
Sure they do. LL is compelling. At least there is an axiom to argue about. What evidential axiom exists for p-values?
♕Deborah Mayo♕
@learnfromerror
Maybe, but LRs don't give strength of evidence–whatever that is (no one has defined it satisfactorily)-being comparative
-
Jeffrey Blume
@StatEvidence
Not an easy task with only 140 chars. So I googled and found: www.dummies.com/programming/big-data/data-science/how-to-…
Not a bad description; Title does not apply for sure!This post is unavailable. See it on X. -
Jeffrey Blume
@StatEvidence
I think its got potential, but would need to be compared to marginalized likelihoods under hierarchical models to see how it shakes out.
Oliver Maclaren
@omaclaren
What do you think of Aitkin's posterior distribution of LRs as a Bayesian alternative to marginalising to get BFs?
-
Jeffrey Blume
@StatEvidence
A student of mine was working on the equivalent of likelihood inference in posterior space. Good idea, but got sidetracked with EB ideas
Oliver Maclaren
@omaclaren
What do you think of Aitkin's posterior distribution of LRs as a Bayesian alternative to marginalising to get BFs?
-
Jeffrey Blume
@StatEvidence
It was applying likelihood ideas to the posterior distribution, correct? I liked the idea. Flows very naturally in a Empirical Bayes setting
Oliver Maclaren
@omaclaren
What do you think of Aitkin's posterior distribution of LRs as a Bayesian alternative to marginalising to get BFs?
-
Jeffrey Blume
@StatEvidence
Agree, of course. I think folks are used to having a one number summary however, and this is part of the resistance to LL.
Oliver Maclaren
@omaclaren
You can of course just present all point comparisons in the form of a likelihood _function_ (can also do similar with power _function_)
-
Jeffrey Blume
@StatEvidence
Even if true for “inference”. It is not correct when describing the strength of evidence. Perhaps this is disagreement.
♕Deborah Mayo♕
@learnfromerror
You overlook the diff in the inference & its reasoning. If you reject a pt, you may infer mu > mu' for the partic mu rep as generating data
-
Jeffrey Blume
@StatEvidence
@learnfromerror Still curious as to how to define and compute error rates if they are not a conditional probability.
Jeffrey Blume
@StatEvidence
Writings on this claim the rate is something different, but it is computed as a conditional prob. Is there an example where they disagree?
Jeffrey Blume
@StatEvidence
@learnfromerror Still curious as to how to define and compute error rates if they are not a conditional probability.
-
Jeffrey Blume
@StatEvidence
Fine to say at least one point in alternative is better supported over null. Not fine to say alternative is better supported over null.
♕Deborah Mayo♕
@learnfromerror
You overlook the diff in the inference & its reasoning. If you reject a pt, you may infer mu > mu' for the partic mu rep as generating data
Jeffrey Blume
@StatEvidence
Very different things. The first is exactly what is done; the latter is the typical interpretation. First can be true while second is not.
Jeffrey Blume
@StatEvidence
Fine to say at least one point in alternative is better supported over null. Not fine to say alternative is better supported over null.
♕Deborah Mayo♕
@learnfromerror
You overlook the diff in the inference & its reasoning. If you reject a pt, you may infer mu > mu' for the partic mu rep as generating data
-
Jeffrey Blume
@StatEvidence
Sounds like over interpreting to avoid the implication. Everyone must compute a probability, and a simple Hyp is required to do that.
♕Deborah Mayo♕
@learnfromerror
Both the nature and the justification of the inference are very different. "Everyone gets there eventually"? Depends where "there" is.
Jeffrey Blume
@StatEvidence
I agree inference after is separate. Choosing best case scenario to reject, then inferring that to applies to all Alternatives is a choice
Jeffrey Blume
@StatEvidence
Sounds like over interpreting to avoid the implication. Everyone must compute a probability, and a simple Hyp is required to do that.
♕Deborah Mayo♕
@learnfromerror
Both the nature and the justification of the inference are very different. "Everyone gets there eventually"? Depends where "there" is.
-
Jeffrey Blume
@StatEvidence
In order to compute a probability, a simple hypothesis is needed. So everyone gets there eventually. Likelihood just gives all possibilities
♕Deborah Mayo♕
@learnfromerror
I find it strange that "there's evidence of some discrepancy" would be hard to understand; it's just the logic of an existential claim.
Jeffrey Blume
@StatEvidence
I think this is what @omaclaren was trying to say earlier. The draw inference you need P(data|H). IF H composite, that prob does not exist
Jeffrey Blume
@StatEvidence
In order to compute a probability, a simple hypothesis is needed. So everyone gets there eventually. Likelihood just gives all possibilities
♕Deborah Mayo♕
@learnfromerror
I find it strange that "there's evidence of some discrepancy" would be hard to understand; it's just the logic of an existential claim.
Jeffrey Blume
@StatEvidence
…unless you either average all possibilities (Bayesian) or pick one to represent the set (Frequentist). There are pros and cons to both.
Jeffrey Blume
@StatEvidence
I think this is what @omaclaren was trying to say earlier. The draw inference you need P(data|H). IF H composite, that prob does not exist
Jeffrey Blume
@StatEvidence
In order to compute a probability, a simple hypothesis is needed. So everyone gets there eventually. Likelihood just gives all possibilities
♕Deborah Mayo♕
@learnfromerror
I find it strange that "there's evidence of some discrepancy" would be hard to understand; it's just the logic of an existential claim.
-
Jeffrey Blume
@StatEvidence
Frequentists use P(data|observed discrepancy) to represent P(data|some discrepancy). Bayesian average over “some discrp).
♕Deborah Mayo♕
@learnfromerror
I find it strange that "there's evidence of some discrepancy" would be hard to understand; it's just the logic of an existential claim.
-
Jeffrey Blume
@StatEvidence
It is an overstatement. Without a prior “some discrepancy” is not a viable hypothesis. You can’t compute P(x|Some discrepancy)”.
♕Deborah Mayo♕
@learnfromerror
I find it strange that "there's evidence of some discrepancy" would be hard to understand; it's just the logic of an existential claim.
-
Jeffrey Blume
@StatEvidence
You mean the FDR? As I said, there are many different version, but all equivalent in large samples (need at least one rejection).
♕Deborah Mayo♕
@learnfromerror
you're confusing a different notion which uses priors
-
Jeffrey Blume
@StatEvidence
Too literal. When saying “evidence mean >0”, we don’t mean means of 1 Billion +. We mean “reasonable means >0”. Its an important difference
♕Deborah Mayo♕
@learnfromerror
No we mean ordinary English: there's evidence mean ≥ 0.
-
Jeffrey Blume
@StatEvidence
Two new vignettes on error rates, false discovery rates, and the likelihood principle. See www.statisticalevidence.com/vignettes Comments encouraged.
-
Jeffrey Blume
@StatEvidence
The likelihood approach is just being more specific: It tells you which pos discrepancies are better supported. The majority are not.
♕Deborah Mayo♕
@learnfromerror
Isn't that strange? for us, to infer H': mu > c just means there's evidence of a pos discrep from c, not that all in H' are better supported
-
Jeffrey Blume
@StatEvidence
Right, but its also cherry picking. What is used to make this inference is the best supported pos discrep. Only best case scenario used.
♕Deborah Mayo♕
@learnfromerror
Isn't that strange? for us, to infer H': mu > c just means there's evidence of a pos discrep from c, not that all in H' are better supported
-
Jeffrey Blume
@StatEvidence
I found Royall respond to Curd quite convincing. The likelihood ratios in question are indeed assumed / specified.
♕Deborah Mayo♕
@learnfromerror
confusing ingredients w/ the inference account. Likelihoodists deny infer to compound hyps,eg.,Royall. See Curd crit errorstatistics.files.wordpress.com/2014/08/royall_2004_l…
-
Jeffrey Blume
@StatEvidence
I think this is exactly why it is such a poor tool. When we reject the a mean <0, do we really mean any positive mean is better supported?
♕Deborah Mayo♕
@learnfromerror
Don't confuse (1) an inference to H rejects not-H w/ (2) H is favored over J w/ data x. The exhaustion in N-P tests is over parameter space,
Jeffrey Blume
@StatEvidence
Alt Hyps near sample mean will be better supported, but very extreme means are not. Dichotomizing parameter space problematic for inference
Jeffrey Blume
@StatEvidence
I think this is exactly why it is such a poor tool. When we reject the a mean <0, do we really mean any positive mean is better supported?
♕Deborah Mayo♕
@learnfromerror
Don't confuse (1) an inference to H rejects not-H w/ (2) H is favored over J w/ data x. The exhaustion in N-P tests is over parameter space,
-
Jeffrey Blume
@StatEvidence
There is real difference. Investigators aren’t asking us to think for them. They are asking for a description of the models the data support
♕Deborah Mayo♕
@learnfromerror
it's the opposite:logics of induction lead to idea there's an ev relation betw H's & x's; warrant is a wd (I use) to pt to the need for more
-
Jeffrey Blume
@StatEvidence
It is! Often overlooked point. Fisher’s ST defines extremeness in tails. HT defines extremeness in LR space. They don’t always agree.
Oliver Maclaren
@omaclaren
Luckily in NP theory the test stat _is_ the likelihood ratio (should be anyway). So they agree: test stat is what given data indicates
-
Jeffrey Blume
@StatEvidence
In my experience this works well. There is a while theory of robust likelihoods. Blume (2007) does this for glms. www.ncbi.nlm.nih.gov/pubmed/17211856
Oliver Maclaren
@omaclaren
And EDA a la Tukey and robust stat a la Hampel, Huber also requires data derivatives eg influence functions etc
-
Jeffrey Blume
@StatEvidence
Catching up here, but I don’t see the issue. If there is more data, you combine the likelihoods. Many ways to do it, Random effect being one
Oliver Maclaren
@omaclaren
My prob with LL/LP is not that isn't OK for 'what the given data say' but that 'what similar but slightly diff. data say' is also relev.
-
Jeffrey Blume
@StatEvidence
My preference to think of this as the positive predictive value, or posterior probability. They are all equivalent in large samples.
♕Deborah Mayo♕
@learnfromerror
FDR was defined by Benjamini and Hochberg (1995) and does not use priors.
-
Jeffrey Blume
@StatEvidence
Many definitions of FDR. Its unfortunate. Most frequently used one is the pFDR by Storey, which is equivalent to FDR from BH ’95.
♕Deborah Mayo♕
@learnfromerror
FDR was defined by Benjamini and Hochberg (1995) and does not use priors.
-
Jeffrey Blume
@StatEvidence
The key is in noticing the Empirical Bayes framework from which the FDR or pFDR is estimated. Its hidden, but there.
♕Deborah Mayo♕
@learnfromerror
FDR was defined by Benjamini and Hochberg (1995) and does not use priors.
-
Jeffrey Blume
@StatEvidence
False. They are there. Its the prevalence of ‘true’ null hypotheses in the 2x2 table. Efron (2010 book) makes the dependence explicit.
♕Deborah Mayo♕
@learnfromerror
FDR was defined by Benjamini and Hochberg (1995) and does not use priors.
-
Jeffrey Blume
@StatEvidence
But then again, this assumes we all agree on how to compute a conditional probability. Not sure we all agree.
Jeffrey Blume
@StatEvidence
It's not the typical likelihood. It's the one that conditions on the p-value rather than the data. Not my first choice, but ok by LL.
Jeffrey Blume
@StatEvidence
I have. I'm ok with severe testing because it's a direct application of the law of likelihood. That p-value function is just a likelihood.
♕Deborah Mayo♕
@learnfromerror
the type I & II error probabilities aren't conditional probabilities; if you want to learn what I think, please read my published work
-
Jeffrey Blume
@StatEvidence
It's not the typical likelihood. It's the one that conditions on the p-value rather than the data. Not my first choice, but ok by LL.
Jeffrey Blume
@StatEvidence
I have. I'm ok with severe testing because it's a direct application of the law of likelihood. That p-value function is just a likelihood.
♕Deborah Mayo♕
@learnfromerror
the type I & II error probabilities aren't conditional probabilities; if you want to learn what I think, please read my published work
-
Jeffrey Blume
@StatEvidence
I have. I'm ok with severe testing because it's a direct application of the law of likelihood. That p-value function is just a likelihood.
♕Deborah Mayo♕
@learnfromerror
the type I & II error probabilities aren't conditional probabilities; if you want to learn what I think, please read my published work
-
Jeffrey Blume
@StatEvidence
If not, then they can't be equal to conditional probabilities all the time. How about a quick example where they are not equal?
♕Deborah Mayo♕
@learnfromerror
the type I & II error probabilities aren't conditional probabilities; if you want to learn what I think, please read my published work
-
Jeffrey Blume
@StatEvidence
Big claim. Either everyone has been computing these wrong for 80 years or everyone simply does not understand what they are computing.
♕Deborah Mayo♕
@learnfromerror
the type I & II error probabilities aren't conditional probabilities; if you want to learn what I think, please read my published work
-
Jeffrey Blume
@StatEvidence
Regardless, the propensity to collect misleading data has to be different from the propensity for observed data to be misleading.
♕Deborah Mayo♕
@learnfromerror
the type I & II error probabilities aren't conditional probabilities; if you want to learn what I think, please read my published work
-
Jeffrey Blume
@StatEvidence
error rate essential before data are collected. FDR essential after data are collected. How often obs. data are misleading is given by FDR
Jeffrey Blume
@StatEvidence
Do you agree @learnfromerror: P(reject|null) is not the same as P(Null|reject). First is 'error rate' second is false discovery rate.
-
Jeffrey Blume
@StatEvidence
Actually in 20 years of biomedical and statistical research, I've never even heard the term 'null result'. That's for mentioning.
♕Deborah Mayo♕
@learnfromerror
no, have you never heard of null results? Sometimes it's to infer no real effect, other times to set upper bounds. Look up Michelson Morley.
Post not available here, view on X
Jeffrey Blume
@StatEvidence
That's the point. They tend to give more misleading results but that does not mean the given result is more likely to be misleading.
Post not available here, view on X
Jeffrey Blume
@StatEvidence
Do you agree @learnfromerror: P(reject|null) is not the same as P(Null|reject). First is 'error rate' second is false discovery rate.
-
Jeffrey Blume
@StatEvidence
Totally lost track of this thread but love to see data and sims on twitter!
Daniël Lakens
@lakens
BTW, this is a plot of the relation between p-values from TOST and NHST from a two-sided one-sample test. If my code is right (I think it is). Seems not to be a simple p > 0.25 (but I have to think more about it, check code, etc).
-
Jeffrey Blume
@StatEvidence
It's all depend on how the results are summarized. The FDR is critical. That is the proper assessment of inferential reliability. Not alpha
Jeffrey Blume
@StatEvidence
That's the point. They tend to give more misleading results but that does not mean the given result is more likely to be misleading.
-
Jeffrey Blume
@StatEvidence
I proposed a measure that can explicitly measure when the data support the null. Or Alt. Or neither. So how do you draw that inference?
♕Deborah Mayo♕
@learnfromerror
That's a key flaw behind the approach you described (at SIS) & many others that buy the lie we learn nothing from neg results-we learn lots!
-
Jeffrey Blume
@StatEvidence
Hypothesis tests unable to distinguish null results from inconclusive. Two outcomes: alt/inconclusive. Need three alt/inconclusive/null.
♕Deborah Mayo♕
@learnfromerror
That's a key flaw behind the approach you described (at SIS) & many others that buy the lie we learn nothing from neg results-we learn lots!
-
Jeffrey Blume
@StatEvidence
It's essential to be able to measure when the data support a null hypothesis. Which is exactly why hypothesis testing has failed science.
♕Deborah Mayo♕
@learnfromerror
That's a key flaw behind the approach you described (at SIS) & many others that buy the lie we learn nothing from neg results-we learn lots!
-
Jeffrey Blume
@StatEvidence
That's not even close to what I said. @learnformerror must have mistaken me for someone else! I said just the opposite.
♕Deborah Mayo♕
@learnfromerror
That's a key flaw behind the approach you described (at SIS) & many others that buy the lie we learn nothing from neg results-we learn lots!
-
Jeffrey Blume
@StatEvidence
For sure we can affirm a 'null' effect, just not with a hypothesis test.
♕Deborah Mayo♕
@learnfromerror
The assumption we'll find out if we're right or not belies the presumption (F in my judgment) that we never affirm a null effect.
Jeffrey Blume
@StatEvidence
Proposal at #ssi2017 to reduce stat. salary by the Type 1 error rate when results are false. Fine if salary is raised by Type 2 when right
-
Jeffrey Blume
@StatEvidence
Final thought #ssi2017: reporting & understanding statistical evidence is hard. No single number summary does the job well. Must use brain.
-
Jeffrey Blume
@StatEvidence
Ah! For efficiency (and because it was there as the default).
This post is unavailable. See it on X. -
Jeffrey Blume
@StatEvidence
It's critical. And don't forget the false discovery rate. You need all three for a coherent framework of Evidence.
Oliver Maclaren
@omaclaren
I like eg Royall's, @StatEvidence etc emphasis on distinction betweeen evidence measure and probability of misleading evidence
-
Jeffrey Blume
@StatEvidence
Love the 3min limit for asking questions and quick comments at #ssi2017 #asasymposium2017
-
Jeffrey Blume
@StatEvidence
Great Vanderbilt @vandy_biostat presence at the #ssi2017 #asasymposium2017 happy our curriculum prepared our students well for discussions.
-
Jeffrey Blume
@StatEvidence
Many thanks to @LucyStats for the voluminous tweeting at #ssi2017 #asasymposium2017 very helpful to all.
-
Jeffrey Blume
@StatEvidence
Proposal at #ssi2017 to reduce stat. salary by the Type 1 error rate when results are false. Fine if salary is raised by Type 2 when right
-
Jeffrey Blume
@StatEvidence
Sitting next to @f2harrell. Tweeting back and forth. Welcome to communication in the 21st century. Glad my kids aren't watching...
-
Jeffrey Blume
@StatEvidence
Right on target. Welcome to the new world of "efficiency".
This post is unavailable. See it on X. -
Jeffrey Blume
@StatEvidence
Many folks at #ssi2017 are very open to trying new things and improving practice. I hope this sentiment grows in the field.
-
Jeffrey Blume
@StatEvidence
It is not enough to know how misleading a result is. Need to know the result too. This is the problem. Statistics uses 1 number for 2 things
Jeffrey Blume
@StatEvidence
Nope. You do want to quote the FP rate, but the p-value (supposedly) measures of the strength of evidence in the data. Both #'s are needed
David Colquhoun on Bluesky 🇺🇦 💙
@david_colquhoun
.@LucyStats @StatEvidence Surely "what you wish the P value was" is the false positive risk
-
Jeffrey Blume
@StatEvidence
Nope. You do want to quote the FP rate, but the p-value (supposedly) measures of the strength of evidence in the data. Both #'s are needed
David Colquhoun on Bluesky 🇺🇦 💙
@david_colquhoun
.@LucyStats @StatEvidence Surely "what you wish the P value was" is the false positive risk
-
Jeffrey Blume
@StatEvidence
Looking forward to @f2harrell talk at #ssi2017 this morning!
Frank Harrell
@f2harrell
As in my #ssi2017 talk this morning, Bayes posteriors for combinations of efffects are probably the easiest thing to do.
-
Jeffrey Blume
@StatEvidence
They key is to understand how composite hyp get simplified because that requires additional assumptions. Also, using mle is composite.
Jeffrey Blume
@StatEvidence
Everything relies on pt hyps. Composites are reduced to simple pts by averaging (Bayesian), maximizing(frequentist).
Oliver Maclaren
@omaclaren
Island problem also suffers from issue of point vs composite and point vs point mix-ups imo. 'Not guilty' is comp. hence no likelihood ratio
-
Jeffrey Blume
@StatEvidence
Everything relies on pt hyps. Composites are reduced to simple pts by averaging (Bayesian), maximizing(frequentist).
Oliver Maclaren
@omaclaren
Island problem also suffers from issue of point vs composite and point vs point mix-ups imo. 'Not guilty' is comp. hence no likelihood ratio
-
Jeffrey Blume
@StatEvidence
Nope. It means that your similar to 87.5% of the popular tests.
Nick Strayer
@NicholasStrayer
I just wrote a test that relies on the central limit theorem. Does this mean I'm banned from both programming and #statistics?
-
Jeffrey Blume
@StatEvidence
But you do need a hypothesis to evaluate. That's what statistical evidence is about: how well the data support various hypotheses.
Frank Harrell
@f2harrell
You don't need a hypothesis do to good science, and Bayes can provide simple evidence for a quantity being in any interval you choose.
-
Jeffrey Blume
@StatEvidence
That's because the computation is done under the marginal model, which is essentially a point hypothesis.
Frank Harrell
@f2harrell
You don't need a hypothesis do to good science, and Bayes can provide simple evidence for a quantity being in any interval you choose.
-
Jeffrey Blume
@StatEvidence
Why worry? All probability computations assume a point hypothesis. Bayes too. It averages over a set of pt. hyps to get another pt. hyp.
Frank Harrell
@f2harrell
But I worry about any method that implies that point hypotheses are what science is typically about
-
Jeffrey Blume
@StatEvidence
Yup. Not a bad idea. Bayes factors are likelihood ratios under a more complex model. And likelihood ratios are optimal evidence measures.
-
Jeffrey Blume
@StatEvidence
The 'frequentist factor' is a likelihood ratio, but conditional on p-value rather than data. Likelihood ideas everywhere. #ssi2017
-
Jeffrey Blume
@StatEvidence
Lot's of talk about 'odds' at #ssi2017 but few people really understand odds. Proof? People still buy lottery tickets....
-
Jeffrey Blume
@StatEvidence
Something interesting on p-values and data dredging: www.oncotherapynetwork.com/qa/p-values-and-data-dredging-…
-
Jeffrey Blume
@StatEvidence
Because they act like a little red badge on research abstract. It's become an indicator of worthiness; an attention getting device.
-
Jeffrey Blume
@StatEvidence
Looking forward to introducing our Second-generation p-value at the ASA SSI conference arxiv.org/abs/1709.09333 #ssi2017 #asasymposium2017











