I'm not sure there's an algorithm in the world that would be able to recommend videos that aren't similar to the ones you currently watch but that you would still like. I think the best route is to just reccomend videos that are similar and rely on external sources (friend/community recommendations, Reddit, etc) for lateral movement across genres.
I can recommend OReilly's book Collective Intelligence. It shows several ways to get useful recommendation feeds from existing databases, irrespective of the similarity of content.
