In this blog, you will learn

• What is Data Normalization?
• What is Feature Scaling?
• What are the different types of feature scaling?
• What is Min-Max Normalization?
• What is Z-Score Normalization?
• What is Data Standardization?
• What is Robust Scaling?

Feature Scaling

Feature scaling is an important step in the preprocessing of data for machine learning algorithms. It is a technique used to transform the range of independent variables or features of data to a common scale. Feature scaling is also known as data normalization or standardization.

The goal of feature scaling is to ensure that all the features of the data are on the same scale so that the model can learn from the data better. Without feature scaling, certain machine learning algorithms may not perform as expected. For example, the K-Nearest Neighbors algorithm relies on the distance between two points to determine the class of a data point. If the features are on different scales, then the distance between the two points will be dominated by the feature with the larger scale.

Why do we need to perform Feature Scaling?

Feature scaling can also help to reduce the time required to train a model. Since the features are on the same scale, the model can focus on the most important features and ignore the features that have a smaller impact on the outcome. This can help to reduce the amount of time required to train the model.

Feature scaling can also help to reduce the effect of outliers in the data. Outliers are data points that have values that are significantly different from the majority of the data points. If the features are on different scales, then the outliers will have a larger impact on the model. By scaling the features, the outliers will have less of an impact on the model.

Types of Feature Scaling

There are several different types of feature scaling that can be used. The most common type of feature scaling is min-max scaling, which is also known as normalization. Min-max scaling transforms the data so that all the values are between 0 and 1. This is done by subtracting the minimum value from each data point and then dividing by the range of the data.

Another type of feature scaling is standardization. Standardization transforms the data so that the mean of the data is 0 and the standard deviation is 1. This is done by subtracting the mean from each data point and then dividing by the standard deviation.

Finally, there is a type of feature scaling known as robust scaling. Robust scaling is similar to min-max scaling, but it is more robust to outliers. Instead of subtracting the minimum value from each data point, the median value is subtracted. This helps to reduce the effect of outliers on the data.

No matter which type of feature scaling is used, it is important to remember that the data should be scaled before the model is trained. If the data is not scaled before the model is trained, then the model may not perform as expected.

Now we will learn about each category of feature scaling in depth. For this blog, we are going to work on an Internet firewall dataset.

Importing the Libraries

import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report
from sklearn.metrics import f1_score
from sklearn.preprocessing import OrdinalEncoder
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import MinMaxScaler
from sklearn.preprocessing import MaxAbsScaler
from sklearn.preprocessing import RobustScaler

Importing dataset

Internet Firewall Data Data Set

Data Set Information:

There are 12 features in total. The action feature is used as a class. There are 4 classes in total. These are allowed, action, drop, and reset-both classes.

df=pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/00542/log2.csv")

df.head(5)

In order to work on the data, we need to convert the categorical column into a numerical column. We have multiple options to perform this operation.

1. One Hot Encoding:  It encodes the values into binary values. The string values are taken and sorted in alphabetical order and then converted into binary values. It can be achieved using the OneHotEncoder in Scikit-LearnLet’s take an example. Suppose we have three values: Apple, pear, and Banana. OnehotEncdoing will work in the following steps.

• The string value will be sorted in alphabetical order. Apple, Banana, and Pears.
• Binary values will be created for each string value. Apple will be given [1,0,0], Banana will be given a binary value of [0,1,0] and Pears will be given a value of [0,0,1].

Note The placement of 1 decides the value.

2. Ordinal Encoding: This method converts each unique categorical value into an integer value. Scikit-Learn’s OrdinalEncoder class provides us to convert the values.
In the previous example, if we use Ordinal Encoding, the values will be converted into 0.0,1.0,2.0. There is no specific ordering provided for that.

For this dataset, we will use an Ordinal Encoder and transform our target column ‘Action’ into a numerical one. Before using the OridnalEncoder, we need to convert the 1D array into a 2D array. For this, we will use np. reshape() method.

ordinalencoder=OrdinalEncoder()

df['Action']=ordinalencoder.fit_transform(np.reshape(df['Action'].values,(-1,1)))

df.head(5)

Once we have transformed our column, we can have a look at the values.

df['Action'].unique()

Now we have prepared our dataset, the next step in the process is to have the training set and target column separate and to achieve that we are going to use the pop() method.

In the next step, we will split the data into training and testing sets using the train_test_split() method.

y=df.pop("Action")

X_train, X_test, y_train, y_test = train_test_split(df, y, random_state=42,test_size=0.1)

Data Normalization

Data normalization can be used to reduce the complexity of the data by removing outliers, transforming the data into a more consistent format, and reducing the number of variables. It can also be used to reduce noise by removing irrelevant data points and reducing the amount of data. Normalization can also be used to make the data more consistent by transforming the data into a more uniform format.

Types of Data Normalization Methods

The most common type of data normalization is min-max normalization. This process involves transforming all data points to the same range, usually between 0 and 1. This is done by subtracting the minimum value from each data point and then dividing by the range (maximum value minus minimum value). This ensures that all data points are on the same scale, which can help to improve the accuracy of the machine learning algorithm.

Another type of data normalization is z-score normalization which is also called as Data Standardization. This process involves transforming data points to have a mean of 0 and a standard deviation of 1. This is done by subtracting the mean from each data point and then dividing it by the standard deviation. This ensures that the data points are on the same scale and that the machine learning algorithm is not biased toward any particular data point.

Why is Data Normalization necessary?

Data normalization is an important step in the preprocessing of data before it is used in a machine learning algorithm. It helps to reduce the effect of outliers and noise on the data, and it can help to improve the accuracy of the machine learning model. There are several different types of data normalization, and each one has its own advantages and disadvantages. It is important to choose the right type of data normalization for your data in order to ensure that the machine learning algorithm is not biased towards any particular data point.

Data normalization is also important for feature selection. Feature selection is the process of choosing the most relevant features for the machine learning algorithm. By normalizing the data, we can ensure that all features are treated equally and that the machine learning algorithm is not biased toward any particular feature. This can help to improve the accuracy of the machine learning model.

Min-Max Normalization

Each attribute is scaled to a range between [0,1]. This can be achieved using Scikit-Learn’s MinMaxScaler. The transformation is given by

X_scaled = (X - X_min) / (X_max - X_min)

Let us take an example to understand this properly. Let’s assume we have a data frame with attribute values [10,20,30]. Now we will apply the MinMaxScaler to this attribute

  X_min=10
  X_max=30

  10-> (10-10)/(30-10):0.0
  20-> (20-10)/(30-10):0.5
  30-> (30-10)/(30-10):1.0

As we can see the minimum value can be 0 and the maximum value can be 1.0. In our case, we are going to apply MinMaxScaler to our dataset to scale our features.

Algorithm: There are three steps involved in this process

• Initialize the Min-Max Scaler: The first step is to use the Min-Max Scaler class and initialize the Min-Max Scaler for the dataset.
• Fit the Min-Max Scaler with the fit() method: Once, we have initialized the Min-Max Scaler we need to find the minimum and maximum values for every attribute. This is achieved by using the fit method. Once the Min-Max Scaler is fitted it can be used for future purposes too.
• Transform the data with the fitted Min-Max Scaler: Fit method calculates the minimum and maximum of every column and the transform method applies it to every column of the data provided.

min_max_scaler=MinMaxScaler()

min_max_scaler.fit(X_train)

X_train_minmax=X_train.copy()

X_train_minmax[X_train.columns]=min_max_scaler.transform(X_train)

X_train_minmax.head(5)

We can see that our data has been scaled in a range of [0,1]. We could also check the maximum values for each attribute and see how these values have been scaled.

X_train.max()

Before the Normalization, each attribute has a different range of values.

X_train_minmax.max()

After Normalization, each attribute has been scaled down in the range of [0,1]. To reduce the steps to perform two operations :

1. Transform the data using a Min-Max Scaler
2. Initialize and train a Logistic Regression Model

We are going to use a Pipeline. A pipeline will allow us to implement multiple steps using a single pipeline.

Steps to implement a Pipeline:

1. Initialize the make_pipeline() method and mention the steps in the order you want them to be executed.
2. Use the fit() method to execute the steps.
3. Use the score() method to calculate your model’s performance.

pipe_minmax = make_pipeline(MinMaxScaler(), LogisticRegression(max_iter=2000))

pipe_minmax.fit(X_train, y_train)

pipe_minmax.score(X_test, y_test)

Let us try to combine these elements together, the complete example is listed below.

#Import Libraries
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import OrdinalEncoder
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.preprocessing import MinMaxScaler


#Data Processing
df=pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/00542/log2.csv")

ordinalencoder=OrdinalEncoder()
df['Action']=ordinalencoder.fit_transform(np.reshape(df['Action'].values,(-1,1)))

y=df.pop("Action")
X_train, X_test, y_train, y_test = train_test_split(df, y, random_state=42,test_size=0.1)


# Feature Scaling
pipe_minmax = make_pipeline(MinMaxScaler(), LogisticRegression(max_iter=2000))
pipe_minmax.fit(X_train, y_train)
pipe_minmax.score(X_test, y_test)

z=(x - u)/s. 

standard_scaler = StandardScaler()

standard_scaler.fit(X_train)

X_train_sscaler=X_train.copy()

X_train_sscaler[X_train.columns]=standard_scaler.transform(X_train)

X_train_sscaler.head(5)

Running the above code gives us a scaled dataset. Each attribute has been standardized to zero mean and unit standard variation.

print("Mean value before Standardizationn")
print(X_train.mean())

print("Mean value after Standardizationn")
print(round(X_train_sscaler.mean(axis=0),5))

We can see that the mean of the attributes before using the standard Scaler covers a range of values. Most of the estimators expect the mean to be centered at 0 and unit variance. If any of the features have variance other than that, that feature would be given more importance compared to the others. This can lead to biased results since more importance would be given to that specific feature.

Just like the mean, we can also check the standard deviation before and after applying the standardization.

print("Variance before Standardizationn")
print(X_train.std(axis=0))

print("Variance after Standardizationn")
print(round(X_train_sscaler.std(axis=0),4))

All the above steps can be reduced into two lines by using a Pipeline. A pipeline allows a combination of several steps which can be cross-validated together by tweaking a range of parameters.

In this example we are creating a pipeline that will have standardization as the first step and then modeling would be the next one.

pipe_standard_scaler = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))

pipe_standard_scaler.fit(X_train, y_train)

Once the pipeline has been created we can check how the model is performing on the test set by using the score() method which will calculate the F1 Score on the test set.

pipe_standard_scaler.score(X_test, y_test)

Running the above code gives us a score of 98%. This tells us that standardizing the dataset has helped our model to learn better. Let’s have a look at the complete code.

#Import Libraries
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import OrdinalEncoder
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler


#Data Processing
df=pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/00542/log2.csv")

ordinalencoder=OrdinalEncoder()
df['Action']=ordinalencoder.fit_transform(np.reshape(df['Action'].values,(-1,1)))

y=df.pop("Action")
X_train, X_test, y_train, y_test = train_test_split(df, y, random_state=42,test_size=0.1)


# Feature Scaling
pipe_standard_scaler = make_pipeline(StandardScaler(), LogisticRegression(max_iter=2000))
pipe_standard_scaler.fit(X_train, y_train)
pipe_standard_scaler.score(X_test, y_test)

MaxAbsScaler

It similarly scales the data but the scaled data lies in a range of [-1,1]. It achieves it by dividing each value by the maximum value of the respective attribute. We can use Scikit-learn’s MaxAbsScaler to perform this operation.

The transformation is given by

X_scaled = X/X_max

Let us take an example to understand this properly. Let’s assume we have a data frame with attribute values [10,20,30]. Now we will apply the MaxAbsScaler to this attribute

  X_max=30

  10-> 10/30= 0.33
  20-> 20/30= 0.66
  30-> 30/30= 1.0

Advantages:

1. Useful in sparse data i.e. when a lot of data points are zero.

Algorithm: There are three steps involved in this process

• Initialize the Max Absolute Scaler: The first step is to use the Max Absolute Scaler class and initialize the Max Absolute Scaler for the dataset.
• Fit the Max Absolute Scaler with the fit() method: Once, we have initialized the Standard Scaler we need to find the maximum value for every attribute. This is achieved by using the fit method. Once the Max Absolute Scaler is fitted it can be used for future purposes too.
• Transform the data with the fitted Max Absolute Scaler: Fit method calculates the maximum value of every column and the transform method applies it to every column of the data provided and returns the scaled data.

Now we will scale our data using MaxAbsScaler and see check its performance using F1 Score.

max_abs_scaler=MaxAbsScaler()

max_abs_scaler.fit(X_train)

X_train_max_abs=X_train.copy()

X_train_max_abs[X_train.columns]=max_abs_scaler.transform(X_train)

X_train_max_abs.head(5)

Running the example would result in scaling using MaxAbsScaler. Now we can check the performance of this transformation by training a Logistic Regression model and checking our performance using the F1 Score.

pipe_maxabs = make_pipeline(MaxAbsScaler(), LogisticRegression(max_iter=2000))

pipe_maxabs.fit(X_train, y_train)

pipe_maxabs.score(X_test, y_test)

Note: We can see that the performance of our model has decreased as compared to standardizing the data using a Standard Scaler. This may vary depending on the dataset you are using.

#Import Libraries
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import OrdinalEncoder
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import MaxAbsScaler


#Data Processing
df=pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/00542/log2.csv")

ordinalencoder=OrdinalEncoder()
df['Action']=ordinalencoder.fit_transform(np.reshape(df['Action'].values,(-1,1)))

y=df.pop("Action")
X_train, X_test, y_train, y_test = train_test_split(df, y, random_state=42,test_size=0.1)


# Feature Scaling
pipe_maxabs = make_pipeline(MaxAbsScaler(), LogisticRegression(max_iter=2000))
pipe_maxabs.fit(X_train, y_train)
pipe_maxabs.score(X_test, y_test)

Robust Scaling

Normalization is important for machine learning estimators. It is achieved by removing the mean and scaling the variance to a unit. But when outliers are present in the data, this could negatively influence the model. In these scenarios working with median and interquartile range give a better result.

RobustScaler removes the median of the attributes and scales the data between the 1st quartile (25th quantile) and the 3rd quartile(75th quantile).

Scikit-Learn provides us with RobustScaler to scale the features.

Disadvantages:

• It cannot be used on sparse data points.

Algorithm: There are three steps involved in this process

• Initialize the Robust Scaler: The first step is to use the Max Absolute Scaler class and initialize the Max Robust Scaler for the dataset.
• Fit the Robust Scaler with the fit() method: Once, we have initialized the Robust Scaler we need to find the median and quantiles for every attribute. This is achieved by using the fit method. Once the Robust Scaler is fitted it can be used for future purposes too.
• Transform the data with the fitted Robust Scaler: Fit method calculates the median and quantiles of every column and the transform method applies it to every column of the data provided and returns the scaled data.

In this example, we are going to scale our data using a RobustScaler and see whether it is impacting our model’s performance or not.

robust_scaler=RobustScaler()

robust_scaler.fit(X_train)

X_train_robust=X_train.copy()

X_train_robust[X_train.columns]=robust_scaler.transform(X_train)

X_train_robust.head(5)

Running the above code with result in scaling the features with RobustScaler and removing the median from the attributes. Let us check if the median has been removed from the data.

print("Before applying RobustScaler")
X_train.median(axis=0)

print("After applying RobustScaler")
X_train_robust.median(axis=0)

We can see that after applying the RobustScaler the median has been removed from the attributes. In the next step, we are going to create a pipeline that will perform RobustScaler standardization and model training in steps.

pipe_robust = make_pipeline(RobustScaler(), LogisticRegression(max_iter=2000))

pipe_robust.fit(X_train, y_train)

pipe_robust.score(X_test, y_test)

From running the above code we can see that our model has been achieving the highest score so far.

Note: This may or may not be applicable to different datasets. You need to check what kind of scaling is working best for you.

Exercise: Try to use different kinds of scalers on your dataset and comment on which one is giving the best score.

#Import Libraries
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.preprocessing import OrdinalEncoder
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import RobustScaler


# Data Processing
df=pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/00542/log2.csv")

ordinalencoder=OrdinalEncoder()
df['Action']=ordinalencoder.fit_transform(np.reshape(df['Action'].values,(-1,1)))

y=df.pop("Action")
X_train, X_test, y_train, y_test = train_test_split(df, y, random_state=42,test_size=0.1)


# Feature Scaling
pipe_robust = make_pipeline(RobustScaler(), LogisticRegression(max_iter=2000))
pipe_robust.fit(X_train, y_train)
pipe_robust.score(X_test, y_test)

Posts

• 6 Best Methods to Convert Categorical Data for Machine Learning