This is when the DM (should) have a BBEG show up with a massive army, burn a few of the PC's home towns down, and threaten to bring upon a world ending apocalypse.
I get you need to cater to your players desires too, but the DM spent perhaps dozens of hours working on a campaign, you can at least grab a plot thread every now and again (you want to start a bar? Fine, I'll accommodate, but you better be ready to let the Druid Plain Shift you to the next plot point when it starts calling). This goes double for a show like Critical Role, yeah the downtime stuff is fun, I'll admit, but it looses a lot of its charm when it's not broken up by action, the occasional fight, or a looming and consistent threat in the background.
This is why I keep saying campaign 1 is my favorite, from pretty much the beginning it had a goal, an end game, a mad lich god threatening to tear the world asunder. Without a clear end goal, at least in a D&D game, all people are going to get is a bunch of dicking around.