It happens in my town too. I'm in a small town of about 20,000 people, with occasional crimes but not a particularly high-crime area. My black friends and students report being stopped by police uncomfortably often. It's happened often enough that we're trying to take steps in the town, and in the university that's located in the town, to try to remedy this apparent bias.
I believe that the police in our town really are trying to do the right thing to keep the town safe. But unconscious prejudices can run deep, and can be hard to see in one's self.
One of the things that isn't happening--but should--in small situations like yours is the loss of "community policing."
A few years ago, my daughter was reporting for the local NPR station in Bloomington IL. They were in the process of hiring a new chief of police and she interviewed the candidates.
One of the candidates was a highly decorated veteran of big city, rought-area policing. He had been in town a few weeks actually walking the streets to get to get "ground truth" of what the job entailed.
Bloomington bordered by another town (Normal IL), and between the two I think the population is about a 100,000. There is no "ghetto" per se, but the west side is distinctly lower income. The population there is racially mixed--it's not predominantly black, but a majority of the black residents of Bloomington live on the west side. That's also where the recent influx of Latino residents come to reside.
Bloomington has in the last ten years been attracting the attention of gangs (it's a stopover on the Interstate between St Louis and Chicago), but they haven't taken serious hold there yet. Bloomington thus doesn't really have
bad areas, just areas where you need to be more careful.
The real policing task is light enough that I've seen them call in two cars for someone stopped on a bicycle.
So this police officer had done a personal study of the city and was discussing it with my daughter.
He pointed out that a fault of the police force was that they had--like most other forces--given up "community policing"--and unnecessarily so, given the small community. He pointed out that in any community there are stable residents--people with families, people who are there for life--and those people want to live in safety. He said there was no reason why Bloomington police should not know those people by name, no reason they shouldn't make a practice of stopping and talking to people, making personal connections.
He said there should be an understanding of the difference between "keeping the peace" and "enforcing the law," and let the stable residents understand that the police were just as interested in "keeping the peace" as they are, and want to be their partners in "keeping the peace."
He said that would mean, for instance, that when police officers respond to a trouble call from a stable resident--someone they've established a relationship with--they won't be looking around that person's house to find something to arrest the person who called them, but dealing with the problem they were called for (although that doesn't mean they wouldn't give an unofficial warning).
And if they get a tip of someone they're looking for at a certain address, they'd know that was a particular old lady's house and the guy was her grandson...maybe they don't need to do a no-knock with the SWAT team.
In his experience, that kind of policing would result in the effects they all really wanted. Residents would be much more likely to drop a tip like, "You know that burglary last week? Well, there are some knuckleheads from Chicago that have been hanging our at the Chicken Shack. You might want to check them out."