Hello Ilikechicken,
This sounds like a really difficult situation. It sounds like your ex is very much acting on his own need to manipulate and control you rather than what is in the best interests of the children and this must be incredibly distressing and frustrating for you.
If it would be helpful, Rights of Women have a family law advice line, in case you need some information around your rights and options: https://rightsofwomen.org.uk/get-advice/
Take care and keep posting,
Lisa