Quoting speaktruth2powr:" Not really....maybe he feels he would be a better parent. He has every legal right to take her to court for custody."
I'm pretty sure it was the threat that pissed her off. He isn't looking for joint custody, he is threatening to take her baby away from her.
OP, he probably won't be able to if he tries. But you do need an established visitation set up by a judge. Because he can take the child, and refuse to return him/her and then you would have to go through a judge to get your child back which would take a long time.