There are plenty of highly mutagenic viruses that are absolutely lethal. Ebola for one, which was a huge bitch to eventually make a vaccine for. HIV is the obvious one as well, constantly eluding the combined medical resources of the world for 4 decades now.
Again, the very fact that viruses, which have literally existed for billions of years, are still perfectly capable of killing you proves that this “downward slope” of lethality may exist on very, very short timescales, but it is not an axiom of virology. Otherwise, why do viruses still kill us after so long?