A/B tests are good and incredibly valuable. People (like Linus or Jake) just don't know how to use them. And, more broadly, whether the test is good or not depends on the metric you are measuring.Its worse than that, its actively indicative of a hostile activity. When you release a feature because the users want the feature, you just release it and off it goes. When you feel the need to A/B test it, then your saying "They don't want/need it, I'm looking for the version that gives me [company] the best outcome". Whether that's the least user annoyance or rejection, the most artificial engagement, or whatever other behavior your actually seeking, A/B Testing is about the response, not the change. And that should tell you all you need to know about the priorities.
You can easily construct a test for actually needed and desired features, instead of engagement bait. YouTube's thumbnail A/B tests measure Watch Time, for example. But this is obviously the wrong metric to look at if you're optimizing purely for high CTR and clickbait.
And this also means that you can't directly go off of A/B tests that measure Watch Time if you are trying to increase your reach and viewership beyond your core demographic, since (when the test is run early on in the video's life cycle) the thumbnail that "wins" in the A/B test will be the one your subscribers prefer, but their preferences won't necessarily reflect what's actually clickable. Seeing how other YouTubers use the tests, made me realize how utterly incompetent 99% of them are.
This is also literally what the "Creator Liaison" at YouTube said when they launched A/B testing back in 2023:
