Regression for Imbalanced Data

 

Introduction 

Imbalanced data are the situation where the less represented observations of the data are of the main interest. In some contexts, they are expressed as “outliers” which is rather more dangerous. As a consequence of the “outliers” expression, such observations are excluded or removed from the data. That eventually ends up with unrepresented analysis and misleading results.

In our current world with all the extreme events of woods fire, pandemic diseases, and economic crises, it is easy to spot different fields of application for imbalanced data such as meteorological/ecological catastrophes, bank frauds, high-risk insurance portfolios, electricity pilferage,…, etc.

Image by author

Addressing classification problems for imbalanced data are quite famous and available in lots of papers and articles. While regression problems are overlooked almost every time, although dealing with them is significantly different. Basically most of the classification problems originally come from continuous variables that have been transferred to categorical for classification analysis!. In the transformation process, patterns and dependencies are ignored due to the change in the data type.

Overview

This article is covering:

• General intuition and techniques for dealing with imbalanced data for regression
• Data preprocessing techniques
• Model processing using the UBR technique
• Evaluation metrics for imbalanced regression
• Application on UBR using imbalanced data
• Conclusion

| Techniques for Imbalanced Data in Regression

The existing machine learning models for regression are mainly constructed based on balanced or almost balanced data. That leads to misleading and very bad performance for such models when dealing with imbalanced data. To be able to deal with imbalanced data using these models, you have one of two options: first, is to increase the representation of the observations of interest vs. the other observations (or vice versa). Second, is to adapt the model itself by parameter tuning based on customized criteria. I am going to discuss these two main strategies to deal with imbalanced data, namely data preprocessing and model processing. There are other two strategies: post-processing and hybrid, but I am not addressing them in this article.

| Preprocessing Techniques

Preprocessing techniques are mainly focused on applying oversampling, undersampling, or mixture between them on the data prior to applying the traditional machine learning regression model.

“Preprocessing techniques force the model to learn about the rare observations of interest in the data.”

For classification, preprocessing has a way easier task due to the clear segregation between the classes, which is already defined from the beginning. But in case of the continuous random variable, in almost all references the what so-called “Relevance function” is used to deal with such a difficult mission. The relevance function takes a value between 0 and 1, whenever the relevance tends more to 1 that refers more to having highly imbalanced data.

Heads up! in the continuous case, the imbalanced data are more referred to as skewed data. The relevance function definition varies based on the data and the available information about its distribution as well, later I will give an example of one of the used definitions.

There are few preprocessing techniques for continuous variables such as the random undersampling, SMOTE for regression, Gaussian noise, SMOGN algorithm. Briefly speaking, SMOTE for regression is considered as an oversampling technique. Gaussian noise and SMOGN algorithms are a mixture of both under/oversampling techniques.

 Model Processing

Model processing is the other strategy for handling imbalanced data, which is rarely addressed although its importance and efficiency in many applications!. Utility-based regression (UBR) is one of the rare techniques that work efficiently with imbalanced or non-uniform weights overall data. This technique is originally used in the data mining field. The basic idea of the UBR is to add a cost penalty for the wrong estimate, which varies based on the relevance function. In case the relevance function is close to 1 (which refers to very rare observation, highly contributing to the imbalance or skewness of the data) then the cost penalty increases and may reach the maximum.

The relevance function has many definitions as mentioned before, for example, it can be defined inversely proportional to the probability density function (if the density is well defined, you need to go for different definitions). UBR for imbalanced regression works as follows: first, choose one of the traditional machine learning models to work with, such as random forest regression, gradient boost regression,..etc. Second, define the relevance and utility functions to evaluate the performance of the model. Third, tune the model parameters that give the best evaluation.

| Evaluation Metrics

Finally and before getting our hands dirty with data, I will discuss the evaluation techniques in case of using the UBR for imbalanced regression.

“Normally for regression, model performance is measured by MSE or RMSE. For imbalanced data, such measures are not valid.”

There are other measures that are already used for classifications (namely; recall, precision, and F1 score), which are adapted using the utility function to evaluate the performance for imbalanced. In contrast to the MSE and RMSE, we look for the maximum value (close to 1) for the recall, precision, and F1 score to reflect better performance.

| Application

In the following, I show an application for the UBR for imbalanced regression and model evaluation. I use regression data that is used in multiple references for imbalanced data. The plot below shows that the data have some extreme cases that would significantly affect the performance of the traditional machine learning models for regression.

Figure 1 shows a clear pattern of imbalanced data, you can click here to check the data and code. If we consider a threshold of value 4, the data contain 31 (15.7%) rare cases.

I have used XGBoost (eXtreme Gradient Boosting) machine learning technique for regression. I evaluate the performance of 360 regression models with 5-fold cross-validation. I select the best two models in terms of the lowest MSE and highest F1 score. I focus on the F1 score rather than the precision and recall separately as it is a weighted average of the two, but you can choose to focus on other criteria based on your research target.

The final model’s results are

Table 1: Evaluation metric results

In Table 1, it is clear that the MSE and RMSE are not changing significantly for the two best models and both are quite high as well. While the F1 score with the smallest MSE changes to almost double that of the model with the largest F1 score. Using the tuned parameters of these two models, I now run the XGBoost on the test data and check the final performance.

Parameters for the best two models

Using these parameters, the evaluation of the two models on the test data is

Evaluation metrics based on the first model

Evaluation metrics based on the second model

Again it is clear that MSE for both models doesn’t have a noticeable difference (30.12 vs. 32.04) while there is a dramatic change in the F1 score (0.000020 vs. 0.431667). These results confirm the using traditional evaluation metric is not useful in the imbalanced data case.

| Conclusion

Don’t ignore rare cases on your data!. In the short term, you can see impressive results but it is not relevant to what actually happens in reality.

Even after recognizing the rare patterns, traditional tools for balanced data are not helpful for imbalanced/skewed data situations. Finally trying different machine learning models could lead to better/worse results, so you need to find an adequate model based on your data.

 

What is Multicollinearity? Here’s Everything You Need to Know 

Introduction

Multicollinearity might be a handful to pronounce but it’s a topic you should be aware of in the machine learning field. I am familiar with it because of my statistics background but I’ve seen a lot of professionals unaware that multicollinearity exists.

This is especially prevalent in those machine learning folks who come from a non-mathematical background. And while yes, multicollinearity might not be the most crucial topic to grasp in your journey, it’s still important enough to learn. Especially if you’re sitting for data scientist interviews!

So in this article, we will understand what multicollinearity is, why it’s a problem, what causes multicollinearity, and then understand how to detect and fix multicollinearity.

Table of Contents

• What is Multicollinearity?
• The Problem with having Multicollinearity
• What causes Multicollinearity?
• Detecting Multicollinearity with VIF
• Fixing Multicollinearity

 

What is Multicollinearity?

Multicollinearity occurs when two or more independent variables are highly correlated with one another in a regression model.

This means that an independent variable can be predicted from another independent variable in a regression model. For example, height and weight, household income and water consumption, mileage and price of a car, study time and leisure time, etc.

Let me take a simple example from our everyday life to explain this. Colin loves watching television while munching on chips. The more television he watches, the more chips he eats and the happier he gets!

Now, if we could quantify happiness and measure Colin’s happiness while he’s busy doing his favorite activity, which do you think would have a greater impact on his happiness? Having chips or watching television? That’s difficult to determine because the moment we try to measure Colin’s happiness from eating chips, he starts watching television. And the moment we try to measure his happiness from watching television, he starts eating chips.

Eating chips and watching television are highly correlated in the case of Colin and we cannot individually determine the impact of the individual activities on his happiness. This is the multicollinearity problem!

So why should you worry about multicollinearity in the machine learning context? Let’s answer that question next.

 

The Problem with having Multicollinearity

Multicollinearity can be a problem in a regression model because we would not be able to distinguish between the individual effects of the independent variables on the dependent variable. For example, let’s assume that in the following linear equation:

Y = W0+W1*X1+W2*X2

Coefficient W1 is the increase in Y for a unit increase in X1 while keeping X2 constant. But since X1 and X2 are highly correlated, changes in X1 would also cause changes in X2 and we would not be able to see their individual effect on Y.

“ This makes the effects of X1 on Y difficult to distinguish from the effects of X2 on Y. ”

Multicollinearity may not affect the accuracy of the model as much. But we might lose reliability in determining the effects of individual features in your model – and that can be a problem when it comes to interpretability.

 

What causes Multicollinearity?

Multicollinearity could occur due to the following problems:

• Multicollinearity could exist because of the problems in the dataset at the time of creation. These problems could be because of poorly designed experiments, highly observational data, or the inability to manipulate the data:
  • For example, determining the electricity consumption of a household from the household income and the number of electrical appliances. Here, we know that the number of electrical appliances in a household will increase with household income. However, this cannot be removed from the dataset

• Multicollinearity could also occur when new variables are created which are dependent on other variables:
  • For example, creating a variable for BMI from the height and weight variables would include redundant information in the model

• Including identical variables in the dataset:
  • For example, including variables for temperature in Fahrenheit and temperature in Celsius

• Inaccurate use of dummy variables can also cause a multicollinearity problem. This is called the Dummy variable trap:
  • For example, in a dataset containing the status of marriage variable with two unique values: ‘married’, ’single’. Creating dummy variables for both of them would include redundant information. We can make do with only one variable containing 0/1 for ‘married’/’single’ status.

• Insufficient data in some cases can also cause multicollinearity problems

 

Detecting Multicollinearity using VIF

Let’s try detecting multicollinearity in a dataset to give you a flavor of what can go wrong.

I have created a dataset determining the salary of a person in a company based on the following features:

• Gender (0 – female, 1- male)
• Age
• Years of service (Years spent working in the company)
• Education level (0 – no formal education, 1 – under-graduation, 2 – post-graduation)

	df=pd.read_csv(r'C:/Users/Dell/Desktop/salary.csv')
	df.head()

Multicollinearity can be detected via various methods. In this article, we will focus on the most common one – VIF (Variable Inflation Factors).

” VIF determines the strength of the correlation between the independent variables. It is predicted by taking a variable and regressing it against every other variable. “

or

VIF score of an independent variable represents how well the variable is explained by other independent variables.

R^2 value is determined to find out how well an independent variable is described by the other independent variables. A high value of R^2 means that the variable is highly correlated with the other variables. This is captured by the VIF which is denoted below:

So, the closer the R^2 value to 1, the higher the value of VIF and the higher the multicollinearity with the particular independent variable.

	# Import library for VIF
	from statsmodels.stats.outliers_influence import variance_inflation_factor
	def calc_vif(X):
	# Calculating VIF
	vif = pd.DataFrame()
	vif["variables"] = X.columns
	vif["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
	return(vif)

• VIF starts at 1 and has no upper limit
• VIF = 1, no correlation between the independent variable and the other variables
• VIF exceeding 5 or 10 indicates high multicollinearity between this independent variable and the others

	X = df.iloc[:,:-1]
	calc_vif(X)

We can see here that the ‘Age’ and ‘Years of service’ have a high VIF value, meaning they can be predicted by other independent variables in the dataset.

Although correlation matrix and scatter plots can also be used to find multicollinearity, their findings only show the bivariate relationship between the independent variables. VIF is preferred as it can show the correlation of a variable with a group of other variables.

 

Fixing Multicollinearity

Dropping one of the correlated features will help in bringing down the multicollinearity between correlated features:

	X = df.drop(['Age','Salary'],axis=1)
	calc_vif(X)

The image on the left contains the original VIF value for variables and the one on the right is after dropping the ‘Age’ variable.

We were able to drop the variable ‘Age’ from the dataset because its information was being captured by the ‘Years of service’ variable. This has reduced the redundancy in our dataset.

Dropping variables should be an iterative process starting with the variable having the largest VIF value because its trend is highly captured by other variables. If you do this, you will notice that VIF values for other variables would have reduced too, although to a varying extent.

In our example, after dropping the ‘Age’ variable, VIF values for all the variables have decreased to a varying extent.

Next, combine the correlated variables into one and drop the others. This will reduce the multicollinearity:

	df2 = df.copy()
	df2['Age_at_joining'] = df.apply(lambda x: x['Age'] - x['Years of service'],axis=1)
	X = df2.drop(['Age','Years of service','Salary'],axis=1)
	calc_vif(X)

The image on the left contains the original VIF value for variables and the one on the right is after combining the ‘Age’ and ‘Years of service’ variable. Combining ‘Age’ and ‘Years of experience’  into a single variable ‘Age_at_joining’ allows us to capture the information in both the variables.

However, multicollinearity may not be a problem every time. The need to fix multicollinearity depends primarily on the below reasons:

1. When you care more about how much each individual feature rather than a group of features affects the target variable, then removing multicollinearity may be a good option
2. If multicollinearity is not present in the features you are interested in, then multicollinearity may not be a problem.