What is a model?<\/b><\/h2>\n

Model represents a real world scenario with some Epsilon, where Epsilon represents the Error factor.<\/p>\n

Y = f(X) + epsilon<\/p>\n

What is an Imbalanced Target Variable?<\/b><\/h3>\n

Let us first go through few real time examples:<\/p>\n

\n
• Telecom Domain:<\/b><\/li>\n<\/ul>\n

  In Telecom, subscribers tend to move frequently from one mobile operator to another\u00a0for better service or offers. This phenomena known as Customer Churn, ranges from 5 to 10%. In order to model this, the entire customer database is coded into 1 \u2013 CHURN customers or 0 \u2013 ACTIVE customers.\u00a0Since the number of Active customers far outweigh the Churn customers and the distribution of such is also not uniform, the data set is called Imbalanced.<\/p>\n\n\n\n\n\n

  <\/td>\n	<\/td>\n	\n # of Observation<\/p>\n<\/td>\n	\n Target Variable<\/p>\n<\/td>\n<\/tr>\n
  \n Target Variable (Binary)<\/p>\n<\/td>\n	\n 1<\/p>\n<\/td>\n	\n 50,000<\/p>\n<\/td>\n	1 = CHURN customers<\/td>\n<\/tr>\n
  \n 0<\/p>\n<\/td>\n	\n 10,00,000<\/p>\n<\/td>\n	0 = ACTIVE customers<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n  <\/p>\n \n Healthcare<\/b> Domain:<\/b><\/li>\n<\/ul>\n A multi-specialty Hospital wanted to predict whether a patient is prone to Diabetes now or in the near future.\u00a0 Modern conveniences have resulted in a more sedentary lifestyle globally thus causing an explosion in the rate of Diabetes affliction. Recent studies have shown that close to 92.5% of all the patients were\u00a0Diabetic or prone to Diabetes, and only 7.5% of the total patients were found to be healthy.<\/p>\n\n\n\n\n\n <\/td>\n <\/td>\n \n # of Observation<\/p>\n<\/td>\n \n Target Variable<\/p>\n<\/td>\n<\/tr>\n \n Target Variable (Binary)<\/p>\n<\/td>\n \n 1<\/p>\n<\/td>\n \n 925<\/p>\n<\/td>\n 1 = Patients prone to Diabetes<\/td>\n<\/tr>\n \n 0<\/p>\n<\/td>\n \n 075<\/p>\n<\/td>\n 0 = Patients without Diabetes symptoms<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n  <\/p>\n What is a Rare Event?<\/b><\/h3>\n An event is said to be rare if the number of times it occurs is very minimum or low<\/p>\n In both the scenarios mentioned above \u2013 Telecom & and Healthcare, the management was interested in predicting (modelling) CHURN customers & PATIENTS without Diabetes symptoms.\u00a0 These two events are called RARE EVENTs, since its overall presence is relatively less when compared to the levels of the other TARGET VARIABLE (Y).<\/p>\n How will you statistically evaluate whether the Target Variable is imbalanced \/ skewed?<\/b><\/h3>\n Perform a Chi-Square Test using the below command (*here it is being evaluated using R-Open Source software)<\/p>\n Chi-Square Test conducted using R-Software<\/b><\/p>\n Patient.Count<\/p>\n Diabetes\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 925<\/p>\n Without Diabetes\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 75<\/p>\n Chi-squared test for given probabilities<\/p>\n Null Hypothesis : Data is uniformly distributed<\/p>\n Alternative Hypothesis: Data is not uniformly distributed<\/p>\n data:\u00a0 Clinical.Test[, 1]<\/p>\n X-squared = 722.5, df = 1, p-value < 0.00000000000000022<\/p>\n \u00a0<\/b><\/p>\n Chi-Square Test conducted using Minitab<\/b><\/p>\n Chi-Square Goodness-of-Fit Test for Observed Counts in Variable: Count<\/b><\/p>\n  <\/p>\n Using category names in Disease<\/h3>\n\n\n\n\n\n Category<\/th>\n Observed<\/th>\n Test Proportion<\/th>\n Expected<\/th>\n Contribution to Chi-Sq<\/th>\n<\/tr>\n Y<\/td>\n 925<\/td>\n 0.5<\/td>\n 500<\/td>\n 361.25<\/td>\n<\/tr>\n N<\/td>\n 75<\/td>\n 0.5<\/td>\n 500<\/td>\n 361.25<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n N\u00a0 DF\u00a0 Chi-Sq\u00a0 P-Value<\/p>\n 1000\u00a0\u00a0 1\u00a0\u00a0 722.5\u00a0\u00a0\u00a0 0.000<\/p>\n  <\/p>\n As the \u2018p-value\u2019 < 0.05 (*which is commonly chosen Alpha value) we can Reject Null Hypothesis and conclude that \u2018Data is not uniformly distributed\u2019<\/p>\n How to overcome this problem?<\/b><\/p>\n This problem can be overcome by two main methods:<\/p>\n \n Sampling methods<\/li>\n<\/ul>\n \u00fc\u00a0 Over Sampling techniques<\/p>\n \u00fc\u00a0 Under Sampling techniques<\/p>\n \n Algorithms<\/li>\n<\/ul>\n \u00fc\u00a0 Penalized Likelihood Algorithms<\/p>\n Disclaimer:<\/b><\/p>\n This blog provides a Macro Level explanation on Imbalanced Targets (Y).\u00a0 It is very important to employ sound countermeasures against imbalanced targets prior to any modeling activity.<\/b><\/p>\n Detailed blog on OVER SAMPLING will be published next.<\/b><\/p>\n","protected":false},"excerpt":{"rendered":" What is a model? Model represents a real world scenario with some Epsilon, where Epsilon represents the Error factor. Y = f(X) + epsilon What…<\/p>\n Read more<\/a><\/p>\n","protected":false},"author":31170,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[7],"tags":[],"_links":{"self":[{"href":"https:\/\/serendio.com\/wp-json\/wp\/v2\/posts\/202"}],"collection":[{"href":"https:\/\/serendio.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/serendio.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/serendio.com\/wp-json\/wp\/v2\/users\/31170"}],"replies":[{"embeddable":true,"href":"https:\/\/serendio.com\/wp-json\/wp\/v2\/comments?post=202"}],"version-history":[{"count":4,"href":"https:\/\/serendio.com\/wp-json\/wp\/v2\/posts\/202\/revisions"}],"predecessor-version":[{"id":424,"href":"https:\/\/serendio.com\/wp-json\/wp\/v2\/posts\/202\/revisions\/424"}],"wp:attachment":[{"href":"https:\/\/serendio.com\/wp-json\/wp\/v2\/media?parent=202"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/serendio.com\/wp-json\/wp\/v2\/categories?post=202"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/serendio.com\/wp-json\/wp\/v2\/tags?post=202"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}