Just because you were invited, doesn't mean you have to go.
Your contact with family is now controlable. Will you be able to cope when you can't control it? There is nothing wrong with protecting your children and your family; it's actually very wise.
Hey Janny, if you can get in your library, get How we Love by Milan and Kay Yerkovich. Your dad sounds a bit like a controller from what you said. You will understand a lot of things better about your families dynamics once you get through the book.
That book is not in the library system here. What subjects does it talk about?
How we as children attach to our families and how the broken style of relating we learn in our families affects how we love our spouse, our children and others close to us. Most libraries will order books for you from other libraries or even get it if you ask.