Read your Bible. The answers are in there. I could approach the answer two ways. The goal, according to 1 Cor. 15-35-54 is the spiritual transformation of the physical body. This is, apparently, dependent on our life, death, and resurrection, and the latter is dependent on that of Christ. Within that process are the experience of grace, mercy, love, justice, etc. Creation is a process. We humans are in-process, so to speak. This makes sin a problem only for humans, not God. God doesn't like sin but sin isn't an obstacle to His plan of spiritual transformation of those who meet the criteria.
The second is a bit more problematic theologically, rationally because it has sin as something proactively planned, planned for, and expected to the point of being necessary. Unblessedly, that's the way most think of God, creation, and themselves. God's provision of Christ was contingent upon sin and that makes God's plan dependent on sin, and that makes God dependent on sin. The supposedly good, righteous, and sinless Law Maker has had to act lawlessly in creating sin. That's simply a contradiction. This model is argued, often unawares, whenever someone says the process of living, falling, and being redeemed teaches us forgiveness and love and that's why sin happened. It sounds reasonable and correct but it runs smack into the contradiction I just described.
the problem with the latter is thinking that the world must be dialectic in all regards. Joy can exist only in the context of sadness, love only in the context of apathy, acceptance only in the context of hatred. It's not true. Neither is it Biblical.
God made a good sinless world full of unrealized dialectic potential and the realization of certain probabilities (like sin) is an undesirable occurrence, but it's not an obstacle to God's plan. In the end, if you don't make it to the goal you will have no one but yourself to blame. You can shake your fist and decry God all you like but He laughs at such protests. You can expect compassion but compassion has been shown you even though you don't deserve it and aren't entitled to it. In the end the ultimate compassion will be shown those who acknowledged God. There's no compassion to be had for those who will not have it - how can anyone who denies it have it? Certainly a Buddhist can understand that.
And, btw, I've not read anything from you that leads me to read you as an "unorthodox thinker." Your complaints are centuries old, repeated ad nauseum by many a dissenter. They are clichéd, and ordinary. I'm going to leverage your Buddhism. It is normal for a person thusly confronted to do one of three things: minimize the critique, defend the self, or deny the reality. All three speak of attachment(s) that need to be discarded. The only correct enlightened response is to acknowledge the truth of the facts (it cannot be denied that you are not the first to make these complaints). So labeling yourself "unorthodox thinker" in this context is a complete misnomer. Enjoin the discrepancy and be free from the contradiction in more than just word.
Then understand that you clearly do not yet correctly understand the Christian paradigm. The question you asked, "Why would a compassionate and loving God create anyone in the first place, if he had the omniscient foreknowledge that even part of his creation would suffer hell?" is misguided and a symptom of the problem. A better question might be, "Why would knowing some would fail prevent God from creating at all?" So, when you realize that you've built your entire theology (or lack thereof) on a red herring I know (and you do to) the instantaneous somatic response you've just felt is the attachment-laden self rearing its ugly head in protest to protect itself from the truth.
You have an opportunity to be reconciled with God and enjoy all that such a relationship brings with it but you choose not to do so imaging there are legitimate intellectual obstacles to your doing so. You have an opportunity to be reconciled to God, to be relationally restored, to be transgressionally forgiven, to be spiritually regenerate and thereby die alive and you think the best response to that is, "Why would a compassionate and loving God create anyone in the first place, if he had the omniscient foreknowledge that even part of his creation would suffer hell?"
Please realize that response is neither Christian, Buddhist, or rational.
Repent. Change the way you think; change the way you live.
(apologies for the length of the post)
