I've never had any issues with Christmas.
It doesn't matter when the Christ child was born. The fact is that I celebrate His birth on December 25th. And since none of us living today, or anytime in the past 1,500-plus years were present on that most blessed of days, it is impossible to be certain when our Lord and Saviour was born.
Pagans celebrated Yule on or around December 25th. Pagans celebrated a lot of things all throughout the year. I'm certain if one looked hard enough he or she could match a pagan festival to every single Christian holiday on the map.
So, that renders the pagan origins debate moot to me.
I and my family use the weeks leading up to December 25th and the day itself to focus more than usual on the blessings of the Lord and the ministry of His life. We take time to thank God for all of our blessings large or small, and center Christ firmly at the root of our celebrations.
Now if the Holy Spirit convicts you to not celebrate Christmas, than that's between yourself and the Holy Spirit. As for me and my house, we will continue to use the holiday to honor Christ in all His magnificence and (to borrow a line from Dickens), I say God bless it.
