I think part of secret lies in what His brethren said, that "no man works in secret if he seeks to be known openly." Throughout His ministry, Jesus is constantly telling the people that He heals to "tell no one about this", or else doesn't explicitly say to let everyone know. So, Jesus tells them that He will not go, only to show up later without telling them, specifically so that He can go "not publicly but in private".
Why does He do this? I think it's debatable and has a few possible explanations, but one seems to be that, if He had told those He healed to tell everyone, the people He told would then tell them that "Jesus of Nazareth told me to tell you these things", and His actions would be understood as a plea for attention rather than a godly act. He knew that people would spread the word anyways, and because of the humility He practiced in not commanding His deeds be told, His actions would be considered genuine by others.
Another part might lay in His own words, "my time has not yet come". "Time" seems to be synonymous with "Hour", which references His crucifixion. He didn't seek to go "publicly" because He knew it would result in a premature death for Him. Instead, Jesus went "privately", which I assume means that He went without the pomp and circumstance, say, of Palm Sunday. He came into town just like anyone else. Word certainly spread about Jesus, but perhaps hardly anyone could really recognize Him by sight alone, and this allowed Him to slip in and out of the "spotlight" as needed.