That's funny IADad, that's the first thing I thought of as well I'm like "Wow, I take a medication for my allergies called Allegra, but its a pretty name too" lol. Random!
But anyway, I think my opinion is that (as closed-minded and traditional as this may sound) children should have their father's last name. And I imagine he wont dissappear altogether, hopefully. And like someone else said, that way your kids will both have the same last name. At least it will advertise that whether he is around or not, your kids did in fact both come from the same father (cause people at their schools etc will not neccessarily know your last name). So that's what I would do there.
However, I think that's the ONLY part of her name he should have anything to do with. If he doesn't want to stick around and be a father, then he doesn't get to decide what her name will be. I think as for the rest of it, you should name her whatever you want. If you don't like Esme, change it. If he doesn't like you changing it - boo hoo, he should have thought about that when he decided the single life would be more rewarding. You're the one that's going to have to do all of the hard work to raise this girl, there is nothing at all wrong with naming her what you want.
I also wanted to say that I LOVE the name Felix, very very cute!!!