If you looked into it and still don't see the pagan origins there's not much more I can say. Since the 4th century Christmas has been going on, and it's earliest celebrations were just as they celebrated Saturnalia, because the RCC wanted pagans to come in. Whether you look at this website down below or look into your own sources, look into unbiased sources. Literally the Catholic database even describes what I'm saying. Everything from the Sabbath being changed to Sunday, to Easter, the reasons they decided to adopt the pagan traditions are all found online and in their databases.
I am taking in to account the calendar shift, in fact because of the calendar shift the dates of Saturnalia went from the 17th through the 23rd of December prior to the shift, and after it became 19th - the 25th. The 25th was the climax of the festival.
You say you don't observe any of the pagan traditions? The gift giving itself is one of the main aspects of saturnalia. The Christmas tree is another aspect of paganism used in saturnalia as well as in other pagan festivities.
Jeremiah 10:2-5
2Thus saith the LORD,
Learn not the way of the heathen, and be not dismayed at the signs of heaven; for the heathen are dismayed at them.
3For the customs of the people are vain: for one cutteth a tree out of the forest, the work of the hands of the workman, with the axe.
4They deck it with silver and with gold; they fasten it with nails and with hammers, that it move not.
5They
are upright as the palm tree, but speak not: they must needs be borne, because they cannot go. Be not afraid of them; for they cannot do evil, neither also
is it in them to do good.
Saturnalia: The Real Roots of Christmas