I agree with this so much. My earliest childhood memory is from when i was 7, which is probably a bit later than usual. I just don't remember any of the stressful events or accidents that occurred prior to that, all I know is what I've been told about by my parents.
And although it doesn't bother me personally, i suppose that if i was younger and more impressionable, and a bunch of politically motivated, manipulative people would gather around me to convince me that something real bad happened to me when I was like 3 or 4, perhaps they could mess with me well enough to make me believe it and "remember" it.
Which is just another reason why i don't trust therapists or any other people that try to explain to you what "really" happened to you and what does it mean - I have a sneaking suspicion that there people who get off on on being able to manipulate someone, reframe their personality, and take over their thinking process. Not unlike demons, sucking up your soul.