Continue reading and pay attention to the context. You’re cherry picking that one verse out and ignoring the rest of the discussion. He specifically says that they were going back into bondage wanting to be under the law. He’s contrasting between being in bondage to the law of Sinai and being children of the free woman, not going back to practicing paganism. That’s specifically what the ENTIRE BOOK is about.
“Tell me, you who want to be under law, do you not listen to the law? For it is written that Abraham had two sons, one by the bondwoman and one by the free woman. But the son by the bondwoman was born according to the flesh, and the son by the free woman through the promise. This is allegorically speaking, for these women are two covenants: one proceeding from Mount Sinai bearing children who are to be slaves; she is Hagar. Now this Hagar is Mount Sinai in Arabia and corresponds to the present Jerusalem, for she is in slavery with her children. But the Jerusalem above is free; she is our mother. For it is written, “Rejoice, barren woman who does not bear; Break forth and shout, you who are not in labor; For more numerous are the children of the desolate Than of the one who has a husband.” And you brethren, like Isaac, are children of promise. But as at that time he who was born according to the flesh persecuted him who was born according to the Spirit, so it is now also. But what does the Scripture say? “Cast out the bondwoman and her son, For the son of the bondwoman shall not be an heir with the son of the free woman.” So then, brethren, we are not children of a bondwoman, but of the free woman.”
Galatians 4:21-31 NASB1995
Go back and read chapter 3 or ANY CHAPTER FOR THAT MATTER the entire book is contrasting between the law and faith not paganism. You’re cherry picking one verse out of the surrounding context.
“You foolish Galatians, who has bewitched you, before whose eyes Jesus Christ was publicly portrayed as crucified? This is the only thing I want to find out from you: did you receive the Spirit by the works of the Law, or by hearing with faith? Are you so foolish? Having begun by the Spirit, are you now being perfected by the flesh? Did you suffer so many things in vain—if indeed it was in vain? So then, does He who provides you with the Spirit and works miracles among you, do it by the works of the Law, or by hearing with faith? Even so Abraham believed God, and it was reckoned to him as righteousness. Therefore, be sure that it is those who are of faith who are sons of Abraham. The Scripture, foreseeing that God would justify the Gentiles by faith, preached the gospel beforehand to Abraham, saying, “All the nations will be blessed in you.” So then those who are of faith are blessed with Abraham, the believer. For as many as are of the works of the Law are under a curse; for it is written, “Cursed is everyone who does not abide by all things written in the book of the law, to perform them.” Now that no one is justified by the Law before God is evident; for, “The righteous man shall live by faith.” However, the Law is not of faith; on the contrary, “He who practices them shall live by them.” Christ redeemed us from the curse of the Law, having become a curse for us—for it is written, “Cursed is everyone who hangs on a tree”— in order that in Christ Jesus the blessing of Abraham might come to the Gentiles, so that we would receive the promise of the Spirit through faith. Brethren, I speak in terms of human relations: even though it is only a man’s covenant, yet when it has been ratified, no one sets it aside or adds conditions to it. Now the promises were spoken to Abraham and to his seed. He does not say, “And to seeds,” as referring to many, but rather to one, “And to your seed,” that is, Christ. What I am saying is this: the Law, which came four hundred and thirty years later, does not invalidate a covenant previously ratified by God, so as to nullify the promise. For if the inheritance is based on law, it is no longer based on a promise; but God has granted it to Abraham by means of a promise. Why the Law then? It was added because of transgressions, having been ordained through angels by the agency of a mediator, until the seed would come to whom the promise had been made. Now a mediator is not for one party only; whereas God is only one. Is the Law then contrary to the promises of God? May it never be! For if a law had been given which was able to impart life, then righteousness would indeed have been based on law. But the Scripture has shut up everyone under sin, so that the promise by faith in Jesus Christ might be given to those who believe. But before faith came, we were kept in custody under the law, being shut up to the faith which was later to be revealed. Therefore the Law has become our tutor to lead us to Christ, so that we may be justified by faith. But now that faith has come, we are no longer under a tutor. For you are all sons of God through faith in Christ Jesus. For all of you who were baptized into Christ have clothed yourselves with Christ. There is neither Jew nor Greek, there is neither slave nor free man, there is neither male nor female; for you are all one in Christ Jesus. And if you belong to Christ, then you are Abraham’s descendants, heirs according to promise.”
Galatians 3:1-29 NASB1995