I think the answer is pretty simple and uncomplicated unless we choose to make it otherwise. The way the perfume industry and designers market fragrances contributes a lot to the way people distinguish different perfumes and decide what a masculine or feminine scent is. For example, most people take it for granted that Fahrenheit is a masculine scent. That's how Christian Dior wanted to market it; as a perfume for men, but it's a perfume I wear from time to time and I'm a woman.
People sometimes associate some scents with a particular gender due to what they are familiar with or what is mostly available. In addition, certain perfume ingredients can be linked to one gender or the other due to societal influence. As a result, it can be concluded that a particular perfume is inherently masculine or feminine.
A good example of societal influence is the strong woody type perfumes that are commonly worn by both men and women in the Middle East. In the West on the other hand, this type of perfume will most likely be classified into the masculine category. Another good example is floral perfumes which are commonly deemed to be women's perfumes. Yet, I personally know men who wear Light Blue for women by Dolce and Gabanna and Euphoria by Calvin Klein.
Therefore, it's not such an uncommon thing for men to wear women's perfumes and vice versa. I've even caught my brothers; all macho and masculine grown up men sneaking sprays of my perfumes on several occasions. On request, I've even made them blends of some light floral signature perfumes I had formulated for my own personal use.
So, who or what determines what gender a perfume should be classified as, and why bother with at all? Well, most perfume designers like to categorize their perfumes based on the fragrance notes in each blend. If a perfume is heavy on rich florals or candy notes, the finished product will most likely fall into the perfumes for women category. Another reason why perfumes are gender classified is for marketing and promotional purposes. Advertiser agents like to identify and target the most profitable group of customers for a particular product so as to ensure the success of that product when it's launched.
All the same, I find that the gender line often gets blurry, especially with modern, fresh fragrance types. The clean notes in this category of perfumes can make the process of distinguishing between what is feminine or masculine a bit difficult. Unlike the classic style of perfumes; with lots of deep and rich notes that can make it easier to link a perfume to a particular gender. In short, modern, aquatic type perfumes have been quite successful in making a lot of us fragrance hermaphrodites, which is not necessarily a bad thing.
In my opinion, perfume doesn't have a gender, if you like the fragrance and it smells good on you, go ahead and enjoy it. There's absolutely nothing wrong with using a fragrance targeted at the opposite gender, male or female!