In this blog, we are going to learn about

• What is Underfitting?
• How to diagnose Underfitting
• How to address Underfitting?

Introduction

Underfitting is a common issue in machine learning, where a model fails to capture the underlying structure of the data. This results in poor performance on both the training and testing datasets. In this blog post, we will explore the concept of underfitting, its causes, and how to diagnose and address it using Scikit-learn in Python.

What is Underfitting?

Underfitting occurs when a machine learning model is too simple to accurately represent the underlying relationships in the data. In such cases, the model may have high bias and low variance, leading to poor performance on both the training and testing datasets.

Some common causes of underfitting include:

• Insufficient training data
• Too simple a model architecture
• Poor feature selection or engineering
• Inadequate model training (e.g., too few iterations or low learning rate)

Diagnosing Underfitting

To diagnose underfitting, you can look at the following indicators:

Poor training performance: If your model performs poorly on the training dataset, it may indicate that the model is unable to capture the underlying structure of the data.

• Poor testing performance: A model that underfits will typically perform poorly on both the training and testing datasets.
• Performance comparison with other models: If a more complex model significantly outperforms the current model, it could suggest that the current model is too simple and underfitting the data.
• Learning curves: By plotting the model’s performance on the training and testing datasets over time, you can identify whether the model is underfitting. If the model’s performance plateaus quickly and remains poor, it is likely underfitting.

Addressing Underfitting

To address underfitting, you can try the following approaches:

• Increase model complexity: Use a more complex model architecture to better capture the underlying relationships in the data. This can be achieved by adding more layers or neurons in a neural network, using a more complex algorithm, or increasing the depth of a decision tree.
• Feature engineering: Improve the quality of the input features by creating new features, combining existing features, or using domain knowledge to select more informative features.
• Increase training data: Gather more training data or use data augmentation techniques to artificially increase the size of the training dataset.
• Optimize training parameters: Adjust the model’s training parameters, such as increasing the number of training iterations or the learning rate, to help the model better learn the underlying structure of the data.

Scikit-learn Examples

In this section, we will provide examples of underfitting using Scikit-learn in Python. We will use the Breast Cancer dataset to demonstrate how to diagnose and address underfitting.

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.datasets import load_breast_cancer

# Load the Breast Cancer Dataset
data = load_breast_cancer()
X = data.data
y = data.target

# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train a simple linear regression model
lr = LinearRegression()
lr.fit(X_train, y_train)

# Calculate training and testing errors
train_error = mean_squared_error(y_train, lr.predict(X_train))
test_error = mean_squared_error(y_test, lr.predict(X_test))

print(f'Training error: {train_error:.2f}')
print(f'Testing error: {test_error:.2f}')

In this example, we train a simple linear regression model on the Boston Housing dataset. We then calculate the training and testing errors using mean squared error (MSE). If these errors are high, it may indicate that the model is underfitting.

To address underfitting, we can try using a more complex model, such as a decision tree or support vector machine (SVM). In this case, let’s use a decision tree regressor with Scikit-learn:

from sklearn.tree import DecisionTreeRegressor

# Train a decision tree regressor
tree = DecisionTreeRegressor(max_depth=4, random_state=42)
tree.fit(X_train, y_train)

# Calculate training and testing errors
train_error_tree = mean_squared_error(y_train, tree.predict(X_train))
test_error_tree = mean_squared_error(y_test, tree.predict(X_test))

print(f'Training error (Decision Tree): {train_error_tree:.2f}')
print(f'Testing error (Decision Tree): {test_error_tree:.2f}')

In this example, we train a decision tree regressor with a maximum depth of 4 on the same dataset. We then calculate the training and testing errors for the decision tree model. If the errors are lower than those for the linear regression model, it suggests that the decision tree model better captures the underlying structure of the data and addresses the underfitting issue.

Conclusion

Underfitting is a common problem in machine learning, where a model fails to capture the underlying structure of the data, leading to poor performance on both training and testing datasets. By understanding the causes and indicators of underfitting, as well as the various approaches to address it, you can improve the performance of your machine learning models. Scikit-learn offers a wide range of tools and algorithms to diagnose and tackle underfitting, making it an invaluable resource for data scientists and machine learning practitioners.

Must Read

• Mastering VIF in Machine Learning for Robust Model Performance
• Harness the Power of PCA in Machine Learning
• Uncovering the Hidden Dangers of Multicollinearity.
• Unlock the Power of Feature Selection.
• Top 8 Cross Validation methods!!!.
• Non Linear Transformations.
• Feature Scaling : Data Normalization vs Data Standardization.
• Best Methods to Convert Categorical Data for Machine Learning.

API’s

• sklearn.linear_model.LinearRegression
• sklearn.tree.DecisionTreeRegressor

Quiz Time

Test your understanding of Underfitting concepts and prepare well for interviews.