Irrespective of his appalling behaviour towards you, you have a duty to protect your daughter from this. Being around someone who is instigating arguments and upsetting a child to the point that they feel they have to intervene, is not a healthy environment for a toddler and can have a seriously detrimental effect on her.
You do seem to realise that you do not deserve to be treated in this way, and neither does your daughter need to be around someone so toxic. Think of her and I’m sure you’ll find the strength to get this negative person out of BOTH of your lives.