My memories go back quite far, I even have a few memories age 2 or even 1. My father's job had us moving every few years, often to places where there is no mistaking where and therefore when something happened. When we finally settled in a place for more than the usual time is when I have serious trouble remembering whether something happened when I was 10, 13, or 16 because everything blurs together because there's no reference for anything and the place we lived in was boring as fuck. There's no mistaking when part of the memory involves bundling up for -60 degree weather, a beach within walking distance of your school, or a foreign country.
It makes me wonder if people actually remember a lot further back than they think, but given most people rarely move very far often or have any other event to give them a frame of reference, things tend to blur for them.