My friend went to Sri Lanka on honeymoon and they absolutely loved it, she booked through Trailfinders and had her own driver for 10 days.
It's somewhere I'd love to visit, lots to see and do and stunning beaches if you want to chill out. And elephant... I love elephants!
I wouldn't worry about violence there, do a bit of research and you'll see the areas to avoid aren't places you'd visit anyway. Hey, I went to Egypt with everyone's 'warnings' ringing in my ears and had the most incredible holiday!