I don't know about atheists, but it drives many Christians crazy.
Like those who take the Bible seriously and try to follow the Lord and not Church myths.
At the time of the founding of the United States of America, most Christians, and especially, most Christian leaders, with the probable exception of the Anglicans/Episcopalians who wanted their own denomination to become the established religion,
did not want America established as a Christian nation.
For one thing, established religion would lead to established denomination (which probably would have been the Episcopalians.)
But more to the theological point, they thought it was an insult to God to suggest that God is not powerful enough on his own, not to require established religion. They considered the truth of God to be it's own evidence, and any establishment of America as a Christian nation would be another way of saying, God's truth isn't powerful enough or well evidenced enough for a free people to become convinced of it.
Thus, if someone at the time of the nation's foundation had asked the most devoted and religious Presbyterian or Baptist or most any other denomination of Christian, "Do you think America should be established as a Christian nation," the vast majority of them would have said, "No," and would have expounded on how the very idea of such was an insult to God, that God is too weak to convince the people on His own, without the establishment of America as a Christian nation.
The Founding Fathers, furthermore, strongly felt that the people had the right to choose, by their own enlightened thinking, prayer, reading and general education, the religious path of their choice.
Charlie