Quoting Jude the Super Kinkster:" <blockquote><b>Quoting -BLT-:</b>" So let them be mad. What's the big deal? I would ... [snip!] ... friends you want. Don't get caught up in the petty grudges."</blockquote> The big deal is that they already hate me."
The party is about the LO so I think you should invite the LOs relatives. grandparents aunts and uncles. I don't know how big your family' are but saying you each have 6 brothers and sisters who are married that would be 24 aunts and uncles and 4 grandparents. then add your few friends. Send them invites and if the LOs aunts husbands brothers cousin dose not get an invite then tell them to bad.
If its just too stressful because of them you can say "F" it and just have you and your LO but I don't think you should let his family control what you do for your LO.
If you do want them to come but just can't afford it then think about the pot luck idea. That is a good way to have people help out.
What dose your SO want to do