I have heard from a friend of mine who has a MA in Psychology that men and women perceive things differently. Women tend to take in a lot more and have a more expansive image of things. Ie: when meeting a person we will notice their hair, their outfit and the location in which we met them all in one image. Men, though, will be more focused. When meeting someone they tend to focus on one thing, so they'll picture the outfit or the hair or the location, but not really put it all together.
I also think there has to be other differences that are both inherent and social. Men and women are treated differently, so right there, we will usually develop differently socially and psychologically, social interactions and conditioning does actually change the brain physically. But there are also other life experiences that will / can influence the way we are. Women get periods once a month and have a very different fertility cycle than men, as, perhaps, one of the biggest differences. I can't say how that influences sex-differences, but I can't see how it can't... Most people I know who identify as trans-gender or as no-gender, tend to stop their periods (if a female). There must be a reason for that...
But there are also probably more similarities than we realise aswell. Both sexes exist in all of the personality types, for example. So an INFJ of whatever, will have certain characteristics, and behaviours regardless of whether or not they are male of female. But then again, me and my best friend are INFJs and both females, but we are still distinguishably different people... so there is also the individual level in all this...