There's definitely two sides of the coin here
If it were me, I'd steer clear of the situation. While the intentions are good and you likey don't have any inappropriate feelings, it still raises eyebrows.
I know, as a father, that if my ex's boyfriend or husband was giving my 4 y/o daughter...