So I assume you believe he was talking to different people in chapter 3 than he was in chapter 4?
Chapter 4 as it continues ...
21Tell me, ye that desire to be under the law, do ye not hear the law?
22For it is written, that Abraham had two sons, the one by a bondmaid, the other by a freewoman.
23But he
who was of the bondwoman was born after the flesh; but he of the freewoman
was by promise.
24Which things are an allegory: for these are the two covenants; the one from the mount Sinai, which gendereth to bondage, which is Agar.
25For this Agar is mount Sinai in Arabia, and answereth to Jerusalem which now is, and is in bondage with her children.
26But Jerusalem which is above is free, which is the mother of us all.