I went through Enterprise pretty quickly, so there isn't really a whole lot I remember. Or, rather, a whole lot that I thought was worth remembering!

The show, much like Voyager before it, had huge potential. And, again, like Voyager, capitalized on this potential pretty rarely of the course of it's run. The good episodes were excellent (and often just fleshing out of things alluded to in other series... as you say, niche appeal), such as the Borg episode and the Klingon virus episode.

However, the show was just directionless. After lackluster Seasons 1 &2, they decided to do an entire arc in The Delphic Expanse with the Xindi and the Temporal Cold War (which is a stupid as fuck thing to reoccur throughout the series... even worse than Q). It was more like Voyager lite. Then the fourth season came around, and they went, again, in another direction; much like you, I thought the Season 4 was the start of what they should have been doing all along... which is to say, short story arcs expanding the canon, not just rehashing things throughout the series. And then, BOOM, before it could gain traction the series was canceled.

I still think that Season 4 was the best season, and would have ramped into something decent. But Enterprise is the worst. Most is harder to watch than even the Original or Animated Series!