How do you know if you are compatible with someone or not?
Both people should feel comfortable being their true selves around each other, and accept the other person for who they truly are. How long this takes will depend on the individuals and their dynamic as a couple.
As far as specifics, I'd look to lifestyle choices, personalities that complement each other, life philosophies that mesh well enough, and maybe similar background (as much as the romantic side of me would rather overlook that sort of thing, I'm aware that it can be a factor).
That being said, I think less compatible couples can succeed if that's what both want and both work hard enough to make it happen. Also, compatibility can vary among the same people. After all, nobody is 100% compatible 100% of the time, but I'm answering your question based on what I think natural compatibility looks like.
I'm kinda confused how you could be INcompatible with someone and not know it?
This can be harder than it seems. Behaviors and expectations are shaped by personal experiences and things like family influence and societal pressure. One or both people may appear quite different than they are on a fundamental level. At least through each others' eyes, and it isn't necessarily intentional. It just works out that way because of how we're socialized. At the end of the day, they may in fact be compatible, but the opposite could also prove to be true. It isn't always obvious.