Do We Feel Undervalued in the Dating Market?A Speed Dating Experiment Data AnalysisWhat influences people’s perception of choosing a dating partner? Is it this person’s appearance, personality or humor? During a series of experiments conducted by the Columbia Business School professors Ray Fisman and Sheena Iyengar from 2002 to 2004, over 500 participants were asked to have a four-minute first date with other participants of the opposite sex, rate their attractiveness, sincerity, intelligence, fun, ambition, and shared Interests, and answer the question whether they would go on another date with their partners again. The dataset was found on Kaggle and it contains questionnaire answers including demographics, dating habits, self-perception and ratings across key attributes, as well as dating decisions.What Am I Trying to Understand?Various data analyses have been performed with this dataset and insights range from gender differences in mate selection to racial preferences in dating. My goal is to perform an analysis that may have not been conducted before and I am particularly interested in how the key attributes affect dating decision as well as whether people have a clear awareness of their self value versus their perceived value or not.Data Preparation ProcessFilter out irrelevant dataI first read the Speed Dating Data Key.doc to understand each field and then filtered out the data that I didn’t need, such as the SAT scores, tuition information, income level and zip code.Normalize dataI normalized the data that were collected through different scale methods, such as the rating method on a scale of 1–10 versus the 100 point distribution method, ordinal scale versus interval scale (e.g. when 3 = one date/week, I converted it to 52 times/year).Process missing dataFor each entry, if the number of missing variables was significant, I chose to remove the entire entry. For example, if the partner’s ratings over all attributes for a participant were missing, I removed the entire entry in the attribute analysis but kept it for other valid analysis. If only one variable was missing, such as sincerity, I replaced it with the mean of other nine partners’ ratings on the same participant. Besides manual cleaning, I used the nearest neighbor method to automatically replace remaining missing values.Data Analysis FindingsI. General InformationThere were 552 participants in total, in which 275 were female and 270 were male (7 missing values). While the oldest and the only participant was 55 years old, 75% of the participants were 24 to 34 years old. The first majority of participants were European/Caucasian-American while the second majority were Asian.II. Attribute AnalysisWhat are the most important attributes in dating?Before the dating experiment started, the participants were asked to distribute 100 points to the attributes below according to how important they are in a potential date. Overall, the whole participant group considered attractiveness as the most important attribute in a potential date, followed by intelligence.To female, the top three attributes were intelligence, sincerity and attractiveness, while to male, they were attractiveness, intelligence and fun. Also, male tended to have more distinctive preference but female’s preference appeared to be more even.However, when the results are breakdown by race, I found that Asian was the only race group that ranked sincerity more important than intelligence.Moreover, when the results are breakdown by work industry, I found that while many groups ranked intelligence more important than attractiveness, Architecture was the only group that gave intelligence a much higher recognition.III. Perception AnalysisBefore the experiment, the participants were also asked to rate themselves over the five attributes on a scale of 1–10 (1=awful, 10=great), excluding shared interests. After each four-minute first date, they were asked to rate their partners over the six attributes.Do we feel that we are always underestimated?Comparing self-ratings and the ratings that the participants assumed they would get can potentially provoke many sociology questions. Do people have the perception or experience that they are being underestimated? Do they tend to exaggerate their strengths and hide their weaknesses? The results show that people generally think that their partners underestimate their attributes, except for intelligence. This would be an interesting point for further research, what makes people believe that their intelligence would be overestimated? Besides, ambition appears to be the only attribute that people think they are what they show.Do we actually overestimate ourselves?When comparing self-ratings and the actual ratings, it shows that the participants did generally overestimate themselves. The attribute that they overestimated the most was fun while the one that overestimated the least was intelligence. It’s intriguing that while people believe their intelligence would be overestimated, other people actually do know how smart they actually are!While we think that we are underestimated, are we actually underestimated?When comparing their assumed partner’s ratings and the actual ratings, I found that while the participants knew that they were underestimated, the ratings they perceived were even lower than they thought over every attribute. How sad!Is it true that the first one is always the best one?Since the participants met several partners during each experiment, I did a Pearson correlation analysis in order to understand whether the meeting order affect people’s ratings. First, I found a negative correlation between the order and how much they liked their partners. The negative correlation is also proven between the order and the four attributes, sincerity, intelligence, fun and ambition. For example, the later they meet their partners, the less sincere they think their partners are. It’s worth conducting more research to see over more experience whether people show less sincerity or simply feel less sincerity from their partners even if their partners have been equally sincere.I also found that there are positive correlation among the attributes. Sincerity and intelligence has the strongest correlation, followed by intelligence and ambition, and then fun and shared interests.IV. Match AnalysisHow many “yes” do we need to say to hear a “yes” back?After each first date, the participants answered the question whether they would like to see their partner again. In this experiment, they sent out 6 “yes” and got 2 matches on average. In other words, for every 3 times they tried, they would always get 1 “love response” back.Are we optimistic about love?After the experiment, the participants were also asked to predict how many matches they would get and the actual number of match was collected. On average, they predicted that they would have 3 matches during each experiment but they actually had 2.The number above was calculated by subtracting the match estimate from the number of match. As it shows, the prediction of the male participants tended to be more optimistic than the female.Regardless the other undefined race, Black/African American and Asian seemed to be the most optimistic while the Latino/Hispanic group had the most accurate prediction.Can love be practiced?On average, the participants went on a date twice a month. But does more dating experience help them get more matches and predict their match more accurately? I did a Pearson correlation analysis and found that the frequency of date has a slight positive correlation with the number of match but not the accuracy of match estimate.V. Dating Decision PredictionI was interested in building a dating decision model to predict people’s decision on whether they would see their partners again over the six attributes and race. So I used the logistic regression analysis to build this model with the 7,777 entries and split the training set and validation set by 80:20 using a randomized method. With missing values, I replaced them with the mean in the same class. For example, when one person met 10 partners and received 9 ratings over attractiveness, I took the mean of the attractiveness rating to replace the missing one. Since there were 15% more “no” decision than the “yes” decision, I also corrected the weight accordingly. In the end, I used the same method to build the model 5 times to exam the stability of the performance.The area under the curve (0.828) proves good performance of the classification when predicting the dating decision to be “yes” or “no”.Although we all know that there’s no equation for love, at least we have the equation to predict how likely you will go on a date with him/her:I found that in actual dating decision-making, attractiveness was still the most important attribute, followed by shared interests and then fun, which is unexpected since shared interests didn’t appear to be very important in the participants’ preference ranking. On the contrary, intelligence didn’t relate to the decision much in this case. Surprisingly, ambition and sincerity seemed to be two negative attributes when it comes to actual dating decision. Moreover, although interracial partners didn’t seem to affect the decision at all, same race had a slightly negative correlation with the decision.Do we really know what we want?At the beginning of my attribute analysis, it illustrates that the ranking of importance to the participants for a potential date were attractiveness, intelligence, sincerity/fun (they are equivalent), shared interests and ambition. However, when people make an actual dating decision, the ranking is different: attractiveness, shared interests, fun, intelligence, sincerity, ambition. It appeared that people undervalued shared interests and overvalued sincerity.ConclusionTo sum up, generally attractiveness is the primary thing that people consider while ambition is the least important when it comes to dating. We expect our partners to be smart and sincere, but once we get to know each other more, we would realize how important shared interests are in a potential relationship. However, don’t be fooled by our own perception because it can also be affected as we meet more people. Sometimes we might be over optimistic about love and overconfident about ourselves but the reality turns out to be disappointing. Should we give up? No! Because even if more dating experience doesn’t make us smarter when finding soulmates, we would still have a higher chance to run into them unexpectedly ❤.