I love both movies, but I think in this matchup I will go with Wrath of Khan.
The Fellowship of the Ring is a great film with a wonderful score, great visuals and a deep storyline. However, out of the three trilogy movies I feel it is the slowest of the 3. The Two Towers is my favourite of the trilogy and had the battle of Helms deep as its main event, while Return of the King had Minas Tirith. While Moria was a big part of the movie, it wasn't as involved as the other two battles. While the Fellowship is a bit slower, it was necessary to be like this so that it could properly setup the trilogy. However, in the matchup against Wrath of Khan, the slow pacing of Fellowship of the Ring is beaten out by the tight pacing of Wrath of Khan.
The Wrath of Khan also has a wonderful score and also has decent visuals. The storyline is tighter and more focused than not only Fellowship of the Ring, but also its prequel Star Trek movie: The Motion Picture. Unlike the future iteration of the movie, Wrath of Khan has a well casted villain who also has a believable storyline and motives. Captain Kirk is punished for his lax treatment of Khan previously, which ended with Khan marooned in a world of his own, marooned but not dead. The cunning and cruelty of Khan is shown throughout the film, and it takes Kirk and his crew a lot of effort to fight back Khan and his schemes.
While I enjoy rewatching both movies, I think the much longer length and slower story of Fellowship of the Ring lose out to Wrath of Khan to me. I can just put on the movie and enjoy the story, which is a decent length. Not too long and not too short. Fellowship of the Ring on the other hand (especially the extended edition) is a bit more effort to watch, with a good amount of time needed to watch it within one day or even two nights. Out of the LOTR Trilogy I think I would put The Two Towers and Return of the King before Fellowship of the Ring, but I still think they are all fantastic movies.