First of all, the Seventh Day Adventists seem to get it wrong simply by equating Sunday worship with celebration of the Sabbath. Nowhere does the Bible prescribe Sunday church services. It can perhaps be inferred from Acts 20:7, in which Paul gathered church at Troas together to share the Lord's Supper and to hear a teaching of the Word. Many times the Bible says that we should be in fellowship with other believers, but I would hardly call modern day worship services "fellowship" (this happens more in the context of smaller groups), so this doesn't seem to be what is intended in Scripture. Let me be clear in saying that I am not saying we shouldn't have Sunday worship. To worship as a church on Sunday mornings is a wonderful and godly thing, and I almost never neglect it. It is also a most ancient church tradition, dating back to New Testament times, and continued even in the second century (St. Justin Martyr wrote about it in his apologies). But it is not a Scriptural commandment.
But I think your question can be distilled down to: why has Sunday taken such importance in the church? It can be shown from Scripture that after Christ's resurrection, the first day of the week began to take prominence as the day when God inaugurated his new creation through his Son. Acts 20:7 is one example. Others are John 20:19 and 1 Corinthians 16:2. Revelation 1:10 specifically calls it the Lord's day. It appears that the early church had no problem worshiping on the first day of the week. So why should we? The Biblical commandment is to rest on the Sabbath, not perform corporate worship. So really, it seems to me that the question is moot.
Besides that, for many of us Sunday is hardly a day of rest. On Sundays I give several people rides to church, go fellowship for dinner, and sometimes I even cook. It's a rewarding and edifying day to be sure, but not always a day of rest. Likewise, Sundays are not my sole day of worship. My church's college ministry does corporate worship on Thursday nights, and I attend this as regularly as Sunday services (a bit more, since once or twice a semester I'm travelling on Sunday). It seems rather premature to equate the Sabbath with the day of corporate worship.