I think she does need some firmer boundaries particularly since she is lashing out, but i dont know given the way things have always been done if just laying down the law is the way to go as it is a very sharp contrast to the extremely lazzai faire approach that you have taken.
I would not allow the spiteful comments just as I would not tolerate them from a friend. So I would definitely penalize such behavior on principal. But rather than bottle things up I would address each incident. If she says something a little on the borderline Id firmly remind her to watch her mouth, if she crosses the line I would penalize it.
One technique that I have come up (after some reading
with in parenting my niece is to negotiate the rules together and then when she breaks them she knows what the penalty is. Obviously the adult is the one in the drivers seat of the initial negotiation but it helps if the kid feels they have a seat at the table.