Do you have people in your church who are mature examples for you, and you trust these people? These, I would think, are ones much more old than you, and they have helped you, already, to get more real with God and learn how to relate in a close relationship. You can share with these people about your situation, while praying for how God has you understand and how He has you growing in submitting to Him all the time.
Learn to do everything with God; then, when you come to something which is an issue, already you have experience in simply doing what God has you do, and so . . . you just do what He has you doing with Him in His peace > Colossians 3:15
If you don't even know and trust people, I would not worry about what they think. But share with ones you know are for real and mature, and be prayerfully ready to evaluate what they say. And take your time

Your mature Jesus people can be much better for you, than someone who is your speed with your interests
About this guy . . . I consider it can be good to share with various Christians who are good for you. And do not only try to get affectionate, if he is not really helping you to grow in Jesus, know God's word better, and get more into loving any and all people. If your feelings are more isolating you with him, this can be an ego thing and self-seeking > and Jesus says >
"if you love those who love you, what reward have you?" (in Matthew 5:46)
If, then, I have someone whom I deeply appreciate and the person is so good for me, I need to love others as myself, by welcoming and encouraging the person to share with others so they also can benefit from the ones who are so special to me.
And even if you do not get much more personal and close with each other . . . his children need various mature Christians to share with them so these children can grow to know how a Jesus person is and how we relate in love. And ones can share with him about how to bring up his children. And you can feed on their example of how they relate with him.
I am afraid that people won’t accept our relationship because he has kids.
First, do you have people in your church who you know are really Christian people and they have helped you with their example? In case you are afraid that really Christian people will have a problem with you, then why do you not trust ones who are God's examples and whom God trusts > Hebrews 13:17 < to take care of His children?
Even so, I don't think the genuinely caring Christians would have a problem with it; but there can be people in a church who do not know how to love, and it is wise not to let yourself be steered and bullied by people who do not know how to love > if ones do not know how to love, their problem is not you, in any case!
But you do need to evaluate how you really are, as a person. It is possible to trick yourself into worrying about him having children, when the real issues are elsewhere . . . including if and how you are growing in Jesus, and if he is; or are you two just looking for some feelings and pleasures?
Boredom and loneliness can get us into some very harmful things. We need to be able to be quiet with God in His love, and submit to Him; and do not give in to dominating and tormenting emotional things and dictatorial drives which will attack and abuse us, in order to keep us from becoming sensitive and quiet and submissive with God >
"rather let it be the hidden person of the heart, with the incorruptible beauty of a gentle and quiet spirit, which is very precious in the sight of God." (1 peter 3:4)
If he has children but not their mother . . . why? In case he got into a relationship which could not work because of how he and she were, then it is still possible that he might use similar ways to get into something like that, with you. And what sorts of relationships have your ways been getting you into? Get wise to any ways which have helped you go the wrong way, and make sure you are not just getting into yet one more item . . . in case this applies to you. I would be more concerned about this, than about what people who do not even really know you think.
Are you being charmed, or are you deeply helping one another get more real in Jesus love? And are you sharing with each other the Jesus people who have been helping you so much? Has he introduced to you the people who have been helping him grow in Christ and learn how to relate and bring up his children? And have you been sharing your role models with him, so he can benefit from them, too?

Please correct me about any way I have not understood you right. I don't know you enough to know what's really happening with you; so I have offered these things, for however much they might apply
