Well, if it's not the server speed in general, then it's something specific to your site that's making Relevanssi this slow. It's not the amount and size of your posts, though.
Fixxi.net, you can only find out by testing, but it is quite likely you'll run into some kind of performance issues, yes.
Relevanssi is designed for thousands of posts, not tens of thousands of posts.