Yes Icandothis, that is definitely very threatening. He has no right to ‘pay you a visit’.
‘No’, means no, and you can call the police and explain what he’s said after you telling him you don’t want him back, and they will likely advise you that if he turns up you should call 999 and they will respond to remove him.
You shouldn’t have to live your life feeling wrong for saying no, or face intimidation because he doesn’t agree with your decision.
He doesn’t have to agree, but he does have to accept and its not your job to help him to accept it, but perhaps the police can.
warmest wishes
ts