This is because our culture enforces the concept that gender is strictly masculine and feminine. Often they are thrown into a third group that is truly undefined, simply considered an "other group" that some look down on. There is no real reason for people to view those who are not simply male or female as anything other than ordinary people. However, most who do it do so because they are afraid of the unknown and the unfamiliar, which I believe should be the case.