Rather than worry about whether you have committed this heinous sin, I suggest you ask yourself a couple other questions.
1. Has God the Father revealed to your heart that Jesus is his Son and is the righteousness of God manifest in the flesh? The true light that there is no darkness in?
2. Did you receive that as truth?
3. Do your works (the things you do everyday) reflect that belief in honesty by a hard pressed striving to get to know the living God and Jesus Christ whom he has sent? Desiring above all things to be molded into the image of Christ?
If you answered, Yes, Yes and Yes then I wouldn't really worry about what happened with the church thing as one who answers yes to all three has an advocate with the Father in Heaven, Jesus Christ the righteous, who bought all mankind with his shed blood upon the tree.
May God the Father bless you and settle your heart on this issue that you might be able to serve him without distraction.