Hello, everybody.
It is not first thread about our's post. To tell the truth, I'm a little bit tired of answering same questions on each thread. So it will be my single post :)
I advice you to visit https://www.reddit.com/r/RaidShadowLegends/comments/rfhklt/clan_boss_drop_changes/
You can find raw data link inside post and process it like you want. Also our conclusions were verified by independent user https://www.reddit.com/r/RaidShadowLegends/comments/rfhklt/comment/hoq0wgn/?utm_source=share&utm_medium=web2x&context=3
Also you can find some opinion about this situation on hell haides channel: https://youtu.be/FrwKtfavc_M?t=439
Breef:
1) We crowdsource data, so it can be compromised, but we don't see any evidence for this theory
2) This post is about our second report. First report did not find anything strange on drop distribution. Moreover until 17th nov our data was in broadly agreement with datamined probs
3) We did not find any anomalies with our users, fill rate or something, that can be related to the bot itself and affect collected data. We just see that a lot of people with average drop (until 17th nov) started to fill drop with huge amount of epic books after 17th nov.
4) Splitting users makes sense in case of A/B test for example
5) Filtering assumes removing users with incorrect drop (5 sacred per day) or with low fill rate in case we try to work with month data. Actually it is common practice for crowdsourced data.
6) We also apply stat tests for our data (raw and filtered). We did not publish it because nobody in russian comunity read anything with math inside:)
Feel free to use our data (you can find it at reddit post) for your own research and good luck.
// Vadim, bot developer
I'm not arguing with your raw data, that is what it is. But putting the work on determining our own P-values for your conclusions, conclusions you drew from data you edited under your own parameters, without sharing those parameters, is not common practice at all.
And since your data made it over here to my side of the forums as "absolute proof" that something has changed, to qoute your own report, I think asking for an explanation of your process and asking for actual stat tests is fairly reasonable. And would be common practice if I was sent this mess here at the University.
Now I have to go analyze your data for you, to see if there is any actual causation, rather than just blind predetermined correlation. Even if there ends up being some, this still has every problem you've seen above.
Sneaky fyi, for all you reading this, when someone DOESN'T share their p-values and statistical testing in scientific pursuits, it is almost ALWAYS becuase the testing shows no provable correlation or causation.