24 Sussex has always kind of been a dump. The government forced a purchase of it from the original owner in the 1940s. It was nearly a century old at that point. It's only been the official residence since the mid 1950s. Prior to that PMs just had their own house. Like the Laurier House which is worth a tour.
The building itself is just a money pit and in all honesty, should be torn down. There are almost no photos of it, because it's hideous. It also serves as only a PMs residence. Despite its size it is unsuitable to hosting official functions or dignitaries. Compared to 10 Downing St in the UK which is the PMs office and private apartments. Doing double duty as an actual office building where most of the PMs staff is. The White House has the President's home, West Wing which houses the oval office. The East Wing, soon to be ballroom with the Presidential bunker underneath. Compare that to 24 Sussex which is just an old crumbling mansion.
In all honesty there should be something that serves as residence and official offices for the PM and staff. With a space for hosting dignitaries. Right now we use the Mariott.