In this blog, we are going to learn

What are classification metrics?
Why are classification metrics important?
What are common classification metrics?
What are advanced Classification Metrics?

Introduction

Importance of classification in machine learning

Classification is one of the most fundamental tasks in machine learning. It involves categorizing data points into two or more predefined classes based on their features. Applications of classification range from spam email filtering and fraud detection to medical diagnosis and image recognition. It is a versatile and widely used technique, and as such, understanding how to evaluate the performance of classification models is crucial for data scientists and machine learning practitioners.

Overview of the Scikit-learn library

Scikit-learn is an open-source Python library that offers a comprehensive suite of tools for machine learning, including various algorithms for classification, regression, clustering, and dimensionality reduction. Additionally, it provides utilities for preprocessing data, selecting and evaluating models, and more. Due to its simplicity, extensive documentation, and active community, Scikit-learn has become one of the most popular libraries for machine learning in Python.

The objective of the blog post

The objective of this blog post is to provide a comprehensive guide to classification metrics in machine learning, with a focus on their implementation using the Scikit-learn library. We will cover the most common classification metrics, including accuracy, precision, recall, F1-score, AUC-ROC, and log-loss, and illustrate their usage with examples. Furthermore, we will discuss how to select appropriate metrics based on the problem at hand and how to evaluate and compare the performance of different classification models. By the end of this blog post, you will have a solid understanding of classification metrics and their practical applications in machine learning projects.

Understanding Classification Metrics

What are classification metrics?

Classification metrics are quantitative measures used to evaluate the performance of classification models. These metrics help data scientists and machine learning practitioners assess how well a model can categorize data points into their correct classes. By providing insights into the strengths and weaknesses of a model, classification metrics enable practitioners to identify areas for improvement, compare different models, and choose the best one for a given task.

Why are classification metrics important?

Classification metrics are essential for several reasons:

Model evaluation: Metrics provide an objective way to assess the performance of classification models and determine whether they meet the desired level of accuracy and robustness.
Model selection: Metrics allow practitioners to compare different classification models, facilitating the selection of the most appropriate one for a given task.
Model optimization: Metrics guide the process of improving a model by highlighting its weaknesses and directing efforts towards areas that require the most attention.
Decision-making: Classification metrics help stakeholders understand the implications of choosing a particular model, enabling them to make informed decisions in real-world applications.

Common classification metrics

Accuracy: The ratio of correctly classified data points to the total number of data points. It is a widely used metric but may be misleading in cases of imbalanced datasets.
Precision: The ratio of true positive predictions to the sum of true positive and false positive predictions. It is a measure of a model’s ability to correctly identify positive instances.
Recall: The ratio of true positive predictions to the sum of true positive and false negative predictions. It is a measure of a model’s ability to capture all positive instances in the dataset.
F1-Score: The harmonic mean of precision and recall. It is a balanced metric that takes both false positives and false negatives into account, making it suitable for imbalanced datasets.
Area Under the Receiver Operating Characteristic (ROC) Curve (AUC-ROC): A measure of a model’s ability to distinguish between positive and negative instances across different classification thresholds. It takes values between 0 and 1, with higher values indicating better performance.
Log-loss: A measure of the difference between the predicted probabilities and the true labels. It penalizes incorrect predictions more severely when the predicted probability is far from the true label, making it a suitable metric for probabilistic classification tasks.

When to use each metric

Choosing the right metric depends on the specific problem and the goals of the classification task. Some factors to consider include:

Dataset balance: For imbalanced datasets, metrics like F1-score, AUC-ROC, or log-loss are generally more informative than accuracy.
Cost of errors: If the cost of false positives and false negatives is different, consider using precision and recall, or other metrics that take both types of errors into account.
Interpretability: If stakeholders require easily interpretable metrics, consider using accuracy, precision, recall, or F1-score.
Probabilistic predictions: If the task requires estimating probabilities rather than binary predictions, consider using log-loss or AUC-ROC.

Ultimately, it is essential to understand the trade-offs between different metrics and select the most appropriate one based on the specific problem and objectives.

Preparing Data for Classification

Selecting a dataset

The first step in any machine learning project is selecting a suitable dataset. The choice of dataset will depend on the specific problem you are trying to solve and the type of classification task you want to perform. There are numerous publicly available datasets that can be used for different classification tasks, such as the Iris dataset for multi-class classification, the Titanic dataset for binary classification, or the MNIST dataset for image classification. Alternatively, you may have access to a proprietary dataset specific to your domain or business.

Data preprocessing

Data preprocessing is a crucial step in the machine learning pipeline. It involves cleaning, transforming, and organizing raw data to make it suitable for input into a classification model. Some common preprocessing tasks include:

Handling missing values: Impute missing values using techniques such as mean, median, or mode imputation, or remove instances with missing values altogether.
Encoding categorical features: Convert categorical variables into numerical format using techniques like one-hot encoding or label encoding.
Feature scaling: Normalize or standardize numerical features to ensure that they have the same scale, preventing any single feature from dominating the model.
Feature engineering: Create new features or transform existing ones to improve model performance or capture domain-specific information.
Feature selection: Select a subset of features that contribute the most to the model’s predictive power while minimizing complexity and overfitting.

Train-test split

To evaluate the performance of a classification model, it is essential to partition the dataset into separate training and testing sets. The training set is used to train the model, while the testing set is reserved for evaluating the model’s performance on unseen data. This process helps estimate the model’s ability to generalize to new data and provides a more realistic assessment of its performance.

A common practice is to split the dataset into 70-80% for training and 20-30% for testing. This can be easily done using Scikit-learn’s train_test_split function. It is also important to ensure that the train and test sets have a similar class distribution, especially in cases of imbalanced datasets. This can be achieved using stratified sampling, which is available as an option in the train_test_split function.

Dataset

Data Set Information:

Car Evaluation Database was derived from a simple hierarchical decision model originally developed for the demonstration of DEX, M. Bohanec, V. Rajkovic: Expert system for decision making. Sistemica 1(1), pp. 145-157, 1990.). The model evaluates cars according to the following concept structure:

CAR car acceptability . PRICE overall price . . buying buying price . . maint price of the maintenance . TECH technical characteristics . . COMFORT comfort . . . doors number of doors . . . persons capacity in terms of persons to carry . . . lug_boot the size of luggage boot . . safety estimated safety of the car

Input attributes are printed in lowercase. Besides the target concept (CAR), the model includes three intermediate concepts: PRICE, TECH, COMFORT. Every concept is in the original model related to its lower level descendants by a set of examples .

The Car Evaluation Database contains examples with the structural information removed, i.e., directly relates CAR to the six input attributes: buying, maint, doors, persons, lug_boot, safety.

Because of known underlying concept structure, this database may be particularly useful for testing constructive induction and structure discovery methods

Attributes:

buying: vhigh, high, med, low.
maint: vhigh, high, med, low.
doors: 2, 3, 4, 5more.
persons: 2, 4, more.
lug_boot: small, med, big.
safety: low, med, high.

import pandas as pd
from sklearn.preprocessing import OrdinalEncoder

columns=['buying','maint','doors','persons','lug_boot','safety','class']
df=pd.read_csv("https://archive.ics.uci.edu/ml/machine-learning-databases/car/car.data",names=columns)

ordinalencoder=OrdinalEncoder()
df[df.columns]=ordinalencoder.fit_transform(df)
df

classification metrics

Calculating Classification Metrics in Scikit-learn

Accuracy

Accuracy is the ratio of correctly classified data points to the total number of data points. It is a widely used metric but may be misleading in cases of imbalanced datasets.

Scikit-learn implementation: To compute accuracy in Scikit-learn, use the accuracy_score function from the sklearn.metrics module.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score


#Split the data into training and testing sets
y=df.pop('class')
X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.2, random_state=42)

#Train a logistic regression model
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)

#Make predictions
y_pred = model.predict(X_test)

# Accuracy
accuracy = accuracy_score(y_test, y_pred)
print(accuracy)

evaluation metrics for classification

Precision

Precision is the ratio of true positive predictions to the sum of true positive and false positive predictions. It is a measure of a model’s ability to correctly identify positive instances.

Scikit-learn implementation: To compute precision in Scikit-learn, use the precision_score function from the sklearn.metrics module.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import precision_score

#Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.2, random_state=42)

#Train a logistic regression model
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)

#Make predictions
y_pred = model.predict(X_test)

# Metric
precision = precision_score(y_test, y_pred,average='micro')
print(precision)

precision

Recall

Recall is the ratio of true positive predictions to the sum of true positive and false negative predictions. It is a measure of a model’s ability to capture all positive instances in the dataset.

Scikit-learn implementation: To compute recall in Scikit-learn, use the recall_score function from the sklearn.metrics module.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import recall_score

#Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.2, random_state=42)

#Train a logistic regression model
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)

#Make predictions
y_pred = model.predict(X_test)

# Metric
recall = recall_score(y_test, y_pred,average='micro')
print(recall)

recall classification

F1-Score

F1-Score is the harmonic mean of precision and recall. It is a balanced metric that takes both false positives and false negatives into account, making it suitable for imbalanced datasets.

Scikit-learn implementation: To compute the F1-score in Scikit-learn, use the f1_score function from the sklearn.metrics module.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score

#Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.1, random_state=42)

#Train a logistic regression model
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)

#Make predictions
y_pred = model.predict(X_test)

# Metric
f1 = f1_score(y_test, y_pred,average='micro')
print(f1)

f1 score

AUC-ROC

Area Under the Receiver Operating Characteristic (ROC) Curve (AUC-ROC) is a measure of a model’s ability to distinguish between positive and negative instances across different classification thresholds. It takes values between 0 and 1, with higher values indicating better performance.

Scikit-learn implementation : To compute AUC-ROC in Scikit-learn, use the roc_auc_score function from the sklearn.metrics module. Note that this function requires predicted probabilities instead of class labels.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

#Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.1, random_state=42)

#Train a logistic regression model
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)

#Make predictions
y_pred = model.predict_proba(X_test)

# Metric
auc_roc = roc_auc_score(y_test, y_pred,multi_class='ovr')
print(auc_roc)

auc roc score

Log-loss

Log-loss is a measure of the difference between the predicted probabilities and the true labels. It penalizes incorrect predictions more severely when the predicted probability is far from the true label, making it a suitable metric for probabilistic classification tasks.

Scikit-learn implementation: To compute log-loss in Scikit-learn, use the log_loss function from the sklearn.metrics module. This function also requires predicted probabilities instead of class labels.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import log_loss

#Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.1, random_state=42)

#Train a logistic regression model
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)

#Make predictions
y_pred = model.predict_proba(X_test)

# Metric
logloss = log_loss(y_test, y_pred)
print(logloss)

classification metrics log loss

Evaluating and Comparing Models

Cross-validation

Cross-validation is a technique used to assess the performance of a model on unseen data. It involves dividing the dataset into k equal-sized partitions, called folds, and training and evaluating the model k times, each time using a different fold for validation and the remaining k-1 folds for training. The average performance metric across the k folds is used as the final estimate of the model’s performance. Cross-validation helps to mitigate overfitting and provides a more reliable estimate of a model’s generalization ability.

Scikit-learn implementation: Scikit-learn provides the cross_val_score function from the sklearn.model_selection module to perform cross-validation. To use it, simply pass the classifier, the entire dataset (features and labels), the number of folds, and the scoring metric as arguments

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score

#Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.1, random_state=42)

#Train a logistic regression model
model = LogisticRegression(max_iter=1000)

#Cross-Validation
cv_scores = cross_val_score(model, df, y, cv=5, scoring='accuracy')
mean_cv_score = cv_scores.mean()
print(mean_cv_score)

cross validation

Grid search for hyperparameter tuning

Hyperparameters are the parameters of a machine learning model that are not learned from the data but must be set prior to training. Tuning hyperparameters involves searching for the optimal combination of hyperparameters that maximize the model’s performance. Grid search is a widely used method for hyperparameter tuning that involves exhaustively searching through a manually specified subset of the hyperparameter space.

Scikit-learn implementation: Scikit-learn provides the GridSearchCV class from the sklearn.model_selection module for performing grid search combined with cross-validation. To use it, first, create a dictionary of hyperparameters to search over, then instantiate the GridSearchCV object with the classifier, the hyperparameter grid, the number of cross-validation folds, and the scoring metric. Finally, call the fit method to perform the grid search.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
from sklearn.model_selection import GridSearchCV

#Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(df, y, test_size=0.1, random_state=42)

#Train a logistic regression model

svm_clf = SVC(kernel='linear')

param_grid = {'C': [0.1, 1, 10], 'kernel': ['linear', 'rbf']}
grid_search = GridSearchCV(svm_clf, param_grid, cv=5, scoring='accuracy')

grid_search.fit(df, y)
best_params = grid_search.best_params_
print(best_params)

hyperparameter tuning

Advanced Classification Metrics

Matthews Correlation Coefficient (MCC)

Matthews correlation coefficient (MCC) is a balanced measure of classification performance that takes all four values of the confusion matrix into account. It ranges from -1 to 1, with 1 indicating perfect classification, -1 indicating complete misclassification, and 0 indicating random classification.

Cohen’s Kappa

Cohen’s kappa is a measure of classification performance that accounts for the agreement that could be expected by chance. It ranges from -1 to 1, with 1 indicating perfect agreement, 0 indicating agreement by chance, and negative values indicating disagreement.

import numpy as np
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
from sklearn.metrics import matthews_corrcoef, cohen_kappa_score


# Load the Iris dataset
data = load_iris()
X, y = data.data, data.target

#Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

#Train a logistic regression model
model = LogisticRegression(max_iter=1000)
model.fit(X_train, y_train)

#Make predictions
y_pred = model.predict(X_test)

# Calculate advanced classification metrics
mcc = matthews_corrcoef(y_test, y_pred)
kappa = cohen_kappa_score(y_test, y_pred)
print(f"MCC: {mcc:.2f}, nKappa: {kappa:.2f}")

advanced classification metrics

Choosing the Right Classification Metric

Selecting the right classification metric depends on the problem domain, dataset characteristics, and the specific goals of the model. Here are some guidelines to help you choose the appropriate metric:

Use accuracy for balanced datasets and simple comparisons.
Use precision, recall, and F1 score for imbalanced datasets or when false positives and false negatives have different costs.
Use AUC-ROC and AUC-PR to assess the overall performance of a classifier across various threshold settings.
Use MCC and Cohen’s kappa to account for class imbalances and chance agreement, respectively.

Conclusion

Recap of the importance of classification metrics

In this blog post, we have explored the importance of classification metrics in machine learning. Classification metrics play a crucial role in evaluating and comparing the performance of different models on a given task. By understanding and selecting the right metrics for a specific problem, we can make more informed decisions about which models to use and how to fine-tune them for optimal performance. We have also demonstrated how to calculate various classification metrics using Scikit-learn and discussed how to prepare data, implement classification algorithms, evaluate models, and deploy and monitor them in production settings.

Encouraging further exploration and learning

Machine learning, and classification in particular, is a rapidly evolving field with many exciting developments and opportunities. We hope that this blog post has provided you with a solid foundation in understanding classification metrics and their implementation using Scikit-learn. However, this is just the beginning of your journey in machine learning. We encourage you to continue exploring different classification algorithms, experiment with more advanced ensemble methods and model explainability techniques, and dive deeper into other areas of machine learning, such as regression, clustering, and deep learning.

By staying curious and continuing to learn, you will be well-equipped to leverage the power of machine learning to solve real-world problems and create innovative solutions that can make a meaningful impact on the world.

Summary

We have learned about classification metrics.
We have learned about classification metrics important.
We have learned about common classification metrics.
We have learned about advanced Classification Metrics.

About Courses