The problem with that is it would rule out any defensive force at all or doing anything with government "property," including driving on government roads, using American money, going to public schools/universities, sending a letter, getting any license of any kind, etc. One of Lysander Spooner's points in
No Treason is that voting doesn't legitimatize the system, because it's self-defense. If someone throws you in a battle field, would you just lie down and do nothing? Or even worse, what if you saw someone else about to be killed?
Agorism works fine for hipsters and cult groups like Freedomain Radio, but it doesn't work well for the average person. That's why the ideals of Murray Rothbard and Ron Paul are picking up in popularity, while Stefan Molyneux and Samuel Konkin are not (nor will they ever).
Rothbard's Practical Politics - YouTube
Rothbard also makes a good point here about how the hardcore Randroids end up being nothing but statists, since they refuse to ally or have a coalition with anyone but their little cult. I fear many libertarians are in that same position.
Rothbard on Alan Greenspan - YouTube