Hi
No they never do leave you alone. The only thing you can do is try to change yourself for the better.
Know who you are and who they are so when the name calling starts it just roles off your back. Learn how to be calm when you interact with him so you can look like and be the bigger person.
Throughout my time spent with narcs the poke you to make you look irrational they manipulate our behaviour if you don’t let this happen the arguments pointless for them.
It takes time but setting boundaries is good. Would you tolerate being spoken to like that by anyone else? Im sure you wouldn’t so why is he allowed. Decide what you will do when these boundaries are broken will you walk away? will you just ignore them?
Even after cutting contact with the father of my children for a considerable amount of time now and the children no longer seeing him he still finds ways to get to me. I ignore it now but it hasn’t always been easy.