It's not a matter of "having a problem with" him in a personal sense. It's a matter of having a moral (and legal, which is not trivial) responsibility to protect any children in the church.
I can personally believe that someone who has been accused of this is innocent, but I have absolutely no discretion to treat that person as innocent, but must exercise a duty of care to ensure the safety of any children in our congregation or activities.
The way things are done in Australia, that will generally mean such a person will be supervised at all times when on church property/at church events, and their presence will be minimised to only what is absolutely necessary (eg. they might come to a communion service but not social events).