I turned FileVault on because i travel with my mini. I keep my files on a remote server and SSH. FileVault lets me not worry about saving passwords in browsers etc. Protecting my data is much more important than the hardware or even the OS install - both easily replaced.
I have had no problem with performance. In fact, it reclaims space (on logout) much faster than my MacBook. It tool a while to encrypt when I first turned it on, but has been working just fine since.
I don't keep a lot of data on my mini, but do sync gmail via IMAP in Mail. If you store large files in your home directory, you might see slowness.
Provided you can remember your password, I think its the way to go.