I saw the movie because my brother liked it. My brother is 19 and a non-Christian (hasn't rejected it, just doesn't know Christ yet). Interestingly, he seemed to be a little more interested in Christianity afterwards as something that helps you determine and act on right vs. wrong.
The problem with vigilante justice in God's name is that we aren't God and we don't know what's going on to the extent he does. The law authorities aren't and don't either, but they have more collective information at their disposal. Additionally, the use of a system of weights and measures in meting out penalties (through the courts) is the best thing we can come up with on earth as imperfect humans to try to correct or suspend people from their wrongdoing. Again, we're not God and living people have yet to stand in his judgement. But we intervene on earth in cases where the general consensus of society is that these folks are impeding or endangering others.
DON'T get me wrong- even as a fairly mild-mannered person, I definitely entertain vigilante fantasies occasionally. The thing is, though, that I also know what it's like to be wrongly accused or chided (not by law enforcement, but by family or friends). If I saw something with my own eyes and knew it to be completely wrong- for example, if my grandfather was killed in front of me- I would be very tempted to kill that person. Despite that I know I should forgive. The rage would be too great and no other outcome would seem fitting.
Now as I recall, the stuff in Boondock Saints largely deals with scenarios where it's completely obvious (as in eyewitness evil) that the bad guys are bad guys. Hmmmm. I think that part of the appeal to this film is two Christians who choose to be waaay out there in the world, not at all closeted, acting constantly on their interpretation of their faith, and boldly standing up to evil. But that can be something we can achieve through means such as prison ministries, ministering in crime-ridden areas. Also prayer for the authorities (to improve them!) Active voting (for legislators and legislation that supports law enforcement) and community participation in community policing as well. Or going into policing ourselves! In fact, my grandfather was a cop by day and a lay preacher by weekend (in a church that was a blend of methodists, presbyterians and episcopalians).
Anyways, I think that the counter-measure to considering vigilante justice a la Boondock Saints is watching Mystic River. (SPOILER) In it, a local shady guy rather brutally murders the suspect in his daughter's murder (who happens to be a boyhood friend). The next day he finds out that the man did not murder his daughter and in fact (as the man had stated as he begged for his life) he had been murdering a pedophile the night of the girl's death.

Vigilante justice twice over and unfortunately resulting in tragic consequences for both men. At the same time, the movie does acknowledge your point (Caelum) that the authorities can be lackluster- a third boyhood friend is the investigating detective who is struggling to figure out who murdered the girl. The girl's father says when the detective tells him that the third (now dead) friend was truly beating up a pedophile and who the real murderer of his daughter was something along the lines of "well, I'm glad you caught him. I wish you'd done it a bit sooner". Sorry, that last sentence is a bit mixed up but hopefully understandable.