If you want to, by your own freedom of conscience, refrain from work on Saturday then you are free to do so. But you are not free to force other Christians to do the same. There is no command for Christians to keep Shabbat; since Christians are not under the Covenant God established with the Children of Jacob at Mt. Horeb.
There is no command as to what day Christians ought to gather for worship, neither the Sabbath or on the Lord's Day, but since the beginning Christians have gathered at least weekly on the first day of the week, calling this day the Lord's Day. It is an apostolic practice that has endured for the last two thousand years. Of course Christians don't only gather on the first day of the week, but have gathered regularly throughout the week in addition to the Lord's Day.
The gathering together is something we ought to do, since we come together for the express purpose of hearing God's word and celebrating the Sacraments, and these things are essential to our life as Christians. What day we gather isn't as important as that we gather. But, again, apostolic practice has always been that we meet at least once a week on the first day of the week; not by divine commandment, but by received practice since the days of the apostles themselves.
Is it wrong that you keep Shabbat? No. That's between you and your conscience.
Is it wrong that you don't worship with fellow Christians on Sunday? That's between you and the Lord. Are you intentionally avoiding gathering together? If that's the case then you should inspect yourself and address why that is the case.
-CryptoLutheran