The way I look at it is, we have chosen these dates as points in time to reflect on an event. We do it collectively as brothers and sisters with a common faith.
We do know when Christ was crucified. It was at the time of Passover. That's a simple one.
It's too bad that you are dragged to these celebrations. I am one that hates being forced to do anything that I don't want to do as well. Being dragged to a religious event, with your beliefs, would seem a bit much.
Given clues found in the Bible (Herod's death right after His birth, for one) can give us a rough 6-month ballpark figure.
https://en.wikipedia.org/wiki/Herod_the_Great
Says that according to the writings of Josephus, there are 3 possible years that Herod could have died: 5BC, 4BC, and 1BC.
4BC seems likely because an Eclipse was on March 13th, 4BC, about 29 days before Passover (Josephus wrote that Herod's death was between an eclipse and Passover).
So, if Herod died somewhere around late March to early April of 4BC, this means that Jesus was born I would estimate anytime between October of 5BC to February of 4BC. December, of course, falls right in there.
The reason I say that, is because the Bible says that Herod, once learning about the Messiah, ordered that all male children younger than 2 years old in Bethlehem be killed, and Mary, Joseph, and Jesus went to Egypt to stay there until Herod was dead. They would have needed some time to make the journey from Bethlehem to Egypt, so I'll give at least 1 month for that.
It fits the timeline. Of course, there's no hard proof that it was December 25th, but I think Christ knows that we don't know the exact date, and we just chose December 25th because it's "close enough" (and of course, Early Christians were trying to overpower/cover up pagan holidays around the same time).