I see that very well. More than their clothing, though, is God's condemnation of their sin and rebellion toward Him which believers are to preach to the lost. Such divinely-mandated preaching repels a great many of the lost. It is for this reason, I think, that Jesus said,
Matthew 7:14
14 Because narrow is the gate and difficult is the way which leads to life, and there are few who find it.
John 3:19-20
19 And this is the condemnation, that the light has come into the world, and men loved darkness rather than light, because their deeds were evil.
20 For everyone practicing evil hates the light and does not come to the light, lest his deeds should be exposed.
Paul also wrote:
Romans 9:33
33 As it is written: "Behold, I lay in Zion a stumbling stone and rock of offense, And whoever believes on Him will not be put to shame."
Offending the lost is the frequent consequence of living for and preaching Christ. It is why the righteous suffer in this world. The lost find Jesus a "stumbling stone and rock of offense." We shouldn't needlessly add to the offense God's truth is to the lost, but we shouldn't be thinking, "At all costs do not offend!" either.
I think dressing well for a Sunday worship service communicates the value and importance of that service. If one wants to make the lost feel careless about the service, believers dressing carelessly for it certainly helps.