Hi I wrote some other response in this thread where I called you a techbro, so sorry about that I probably count as one too I wasn't trying to insult you too much.
Anyway when you say "Furthermore, it's not only about whether 0.05 is the sole criteria, but also about whether it's a useful criteria for us to highlight at all, depending on whether the anchoring effect of it is damaging relative to alternatives." I love this analogy of the null hypothesis vs. alternative hypotheses in frequentist statistics to the 'anchoring effect' cognitive bias that you try to work to your advantage in marketing and sales and negotiation or management. https://en.wikipedia.org/wiki/Anchoring_(cognitive_bias)
If you don't want to be tied to p-values and you only care about downstream decisions rather than quantifying beliefs, you can use some ideas in decision theory https://en.wikipedia.org/wiki/Decision_theory
For example, maybe you are deciding between two alternative ways of doing something. You don't know which one is better, and you are confronted with not only the decision of which one to use, but also with the decision of whether to experiment (with A/B testing for example) to be more sure of which one is right, versus whether to exploit the one that you currently think is better. This is the multi-arm bandit problem and it doesn't necessarily use p-values so your intuition is right! https://en.wikipedia.org/wiki/Multi-armed_bandit
Maybe that's not your situation. Maybe your situation is that you have an existing business process and you want to know whether to switch to one that might be better. Someone might say it's a p-value problem, but again I agree with your intuition that it really isn't best to think about it that way, especially when there is a cost to switching. Instead, it's a more complicated decision that depends on what is the switching cost, how much better you think the new process would be (including uncertainty of it), and what kind of business horizon you care about. There might even be a multi-armed bandit effect again even in this situation, where you also have to weigh the costs of reducing your uncertainty of the switching improvement or even of reducing the uncertainty of the switching cost itself.
Anyway, these problems do involve concepts from probability and statistics but it's for sure true that the decisions don't always reduce to P < 0.05 at the end! Good luck best wishes living your best techbro life!
I've been called worse things, and at this stage in life it is hard to be offended by people who don't know me well enough to give an insightful insult. But I hadn't responded to the earlier comment because it was more generally antagonistic and seemed to reflect a (possibly intentional?) misinterpreted and negative reading of the blog. "Don't feed the trolls", as the saying goes.
I started writing that particular blog with different experimental methods in mind, but wrote so much as a prerequisite that I wanted to stop and make that first part a standalone post. My last paragraph was supposed to make it clear that this was a launching off point, rather than a summation of everything I know about decision theory or experimentation.
Thanks for writing more substance into this comment. These opinions are sensible, I agree with much that you wrote here. On one: I do use MABs and see them as a feasible method for many organizations, and at some point I'd like to write about some challenges with those too.
Wishing you the best in your techbro or ftxbro or <whatever>bro life too!
Anyway when you say "Furthermore, it's not only about whether 0.05 is the sole criteria, but also about whether it's a useful criteria for us to highlight at all, depending on whether the anchoring effect of it is damaging relative to alternatives." I love this analogy of the null hypothesis vs. alternative hypotheses in frequentist statistics to the 'anchoring effect' cognitive bias that you try to work to your advantage in marketing and sales and negotiation or management. https://en.wikipedia.org/wiki/Anchoring_(cognitive_bias)
If you don't want to be tied to p-values and you only care about downstream decisions rather than quantifying beliefs, you can use some ideas in decision theory https://en.wikipedia.org/wiki/Decision_theory
For example, maybe you are deciding between two alternative ways of doing something. You don't know which one is better, and you are confronted with not only the decision of which one to use, but also with the decision of whether to experiment (with A/B testing for example) to be more sure of which one is right, versus whether to exploit the one that you currently think is better. This is the multi-arm bandit problem and it doesn't necessarily use p-values so your intuition is right! https://en.wikipedia.org/wiki/Multi-armed_bandit
Maybe that's not your situation. Maybe your situation is that you have an existing business process and you want to know whether to switch to one that might be better. Someone might say it's a p-value problem, but again I agree with your intuition that it really isn't best to think about it that way, especially when there is a cost to switching. Instead, it's a more complicated decision that depends on what is the switching cost, how much better you think the new process would be (including uncertainty of it), and what kind of business horizon you care about. There might even be a multi-armed bandit effect again even in this situation, where you also have to weigh the costs of reducing your uncertainty of the switching improvement or even of reducing the uncertainty of the switching cost itself.
Anyway, these problems do involve concepts from probability and statistics but it's for sure true that the decisions don't always reduce to P < 0.05 at the end! Good luck best wishes living your best techbro life!