If you look at Colossians 2:16 by itself, then it is ambiguous as to whether Paul was saying not to let anyone judge them for keeping God's holy day or for not keeping them, but if we look at the context of the views of the people judging them and well as keep in mind the broader theme that we must obey God rather than man, then it becomes clear:
Colossians 2:8 See to it that no one takes you captive by philosophy and empty deceit, according to human tradition, according to the elemental spirits of the world, and not according to Christ.
Paul would never have described those teaching people to obey the holy, righteous, and good Law of our God in accordance with Christ's example as taking people captive by philosophy and empty deceit according to human tradition and not according to Christ. He went into more details about what these elemental spirits of the world are later in the chapter:
Colossians 2:20-23 If with Christ you died to the elemental spirits of the world, why, as if you were still alive in the world, do you submit to regulations— 21 “Do not handle, Do not taste, Do not touch” 22 (referring to things that all perish as they are used)—according to human precepts and teachings? 23 These have indeed an appearance of wisdom in promoting self-made religion and asceticism and severity to the body, but they are of no value in stopping the indulgence of the flesh.
So the Colossians were keeping God's holy days in obedience to God's commands, they were being judged by those teaching human traditions and precepts, self-made religion, asceticism, and severity to the body, and Paul was writing to encourage them not to let any man keep them from obeying God.
Romans 14:5. It's a Romans 14 issue. An issue that only causes contention and debate, not one that helps to edify another.
Again, it is good to make a distinction between what is said about man's opinion and about the commands of God. The issue being discussed in the chapter is stated in the first verse, namely that it is in regard to how to handle disputes of human opinion, not in regard to whether followers of God should follow His commands. So nothing in the chapter in the chapter should be interpreted as being in favor of disobeying the God that we serve and nothing in the chapter even spoke about whether we should obey God's command to keep the Sabbath.
In Romans 14:5-6, it is speaking about those who were eating unto the Lord or refraining from eating, so it is speaking about fasting. As a matter of opinion, it had become a common practice to fast twice a week or to commemorate certain events, those who esteemed those days for fasting were judging those who did not, they were in turn being resented, and this is exactly the sort of judging each other over opinions that Paul was seeking to quell. We aren't to keep the Sabbath because man esteemed it, but because God esteemed it, blessed it, made it holy, and command His followers to keep it.