I believe all sins are equal. They are all imperfections in the the sight of the Most Perfect God. So if I snap angrily at a person or murder someone, they are all sins, and sins, except for the forgiveness of Christ, distance me from the Father. Crabby people and murderers alike may be forgiven.
I interpret James different than you. James 2:10 tells us: "For whosoever shall keep the whole law, and yet offend in one point, he is guilty of all." That is certainly true, but no where in Scripture does it mention masturbation. There are vague references to sexual immorality, but I believe that refers to things such as inappropriate contentography, adultery, homosexuality and the other things that did and do occur that I am not going to mention. Since it is so wide spread, if God was offended by it, I believe He would have said something about it. He didn't.
And I don't want to mislead you, but I have not given it up. I have far more important concerns than that.