Isn't the mystery God that He gave up His Son for all men?
In part, yes, but as Paul states in Colossians 1:25-27, another aspect is the eternal indwelling of God. "The riches of the glory of the mystery is Christ in us.
This is something "Eye had not seen, ear had not heard, nor had it entered into the hearts of men."
I just gave several passages you can look at to get an idea of the Mystery of Christ if you want to take a look.
One of the things that is important about this doctrine is that we see a distinct change in salvation between the Old Testament and the New. It is the same salvation, to be sure, but we (those in relationship with God through the New Covenant) have an added step in the progression of Eternal Redemption, and that is our being baptized into Christ.
The Old Testament was not.
Some will ask, "Does this mean they were not saved?" The answer is no, they were saved from the eternal perspective of God, but they were not born again during their lifetimes as we are. This can be likened to the "step" we have not yet been granted and that is glorification. That will not take place in the Body of Christ until the Rapture.
But we are no less saved because we have not been glorified, just as the Old Testament Saint was no less saved because they were not born again during their lifetime. They died not having received eternal remission of sins, but received it when Christ died on the Cross (Hebrews 10:1, Hebrews 10:14, Hebrews 9:12-15, Romans 3:24-25).
The Old Testament Saints received progressive revelation concerning salvation through a Messiah, so we understand that Isaiah had more knowledge about Him than Adam, Noah, and Abraham. The women at the well, a woman of Samaria, was awaiting Christ. But it would not be until after He died that a proper context for why Christ is the Savior was revealed unto men (by the Spirit sent down from Heaven (John 16:7-9; 1 Peter 1:10-12). Today, many Jews have rejected Jesus Christ and are still awaiting Prophecy to be fulfilled. We would not ascribe belief and faith unto them, but many do that very thing with the Old Testament Saint. The saying is popular, "They were saved by looking forward to the Cross," but the problem with that is that many of the Old Testament Saints had no idea about the Cross. They had no idea that the Messiah would die. Even the disciples expected a physical fulfillment, which is not unjustified, because there will be a physical fulfillment.
but they didn't understand the death of Christ in their stead. This is why Peter was willing to commit murder in the Garden of Gethsemane—to keep Jesus from the Cross.
That is why he rejected the Gospel and rebuked Christ, saying basically, "No, Lord, the only thing that keep me from eternal separation—I don't want that to happen!" (Matthew 16:20-23)
The Mystery of the Gospel of Jesus Christ is one of the first things I recommend to people who seek to understand their salvation. When you understand the Mystery of the Gospel of Christ you understand the magnitude of the Cross, and what Christ has done for us.
God bless.