Matt 12:36 But I say unto you, That every idle word that men shall speak, they shall give account thereof in the day of judgment.
Will you, yourself, give account for all of your idle words? If you've repented and asked The Lord for forgiveness for those idle words then must you still give account? Or has The Lord thrown those idle words so far behind Him that they never existed? If so, then the "all men" inclusive assumption in your Matthew verse is incorrect. Or misunderstood.
1 Corinthians 4:5, "Therefore judge nothing before the time, until the Lord come, who both will bring to light the hidden things of darkness, and will make manifest the counsels of the hearts: and then shall every man have praise of God."
The crux of the (mis-)understanding here, imo, is that Christians, on the whole, have an inordinate tendency to rob all of the blessings and promises given to the Jews for themselves and leave to the Jews what little that remains, the curses. We must keep in mind who is being spoken to and who is being spoken about. In short, in the Matthew verse above we see a Jew, speaking to Jews, about Jewish issues. Nothing in there about Christians or even, for that matter, those not in Christ. Obviously we have more than one judgment; so, 'which' judgment is for whom? Salvation is not based on works and, yet, there is to be a judgment of works. Still, why would the unsaved need to be judged if they are already judged wanting and are lost to begin with; indeed, the Scriptures tell us they already 'have their reward'. All things in the Bible are written FOR our understanding and edification; but not all things contained in the Bible are addressed specifically TO us (Christians).