Percentiles In Statistics : What Is It And How To Calculate?

In this article, we'll take a look at Hide

4) Why Are Percentiles Important in AI Engineering?

What are Percentiles in Statistics

Percentiles are a statistical concept used in machine learning to describe the distribution of a dataset. A percentile is a measure that indicates the value below which a given percentage of observations in a group of observations falls.

The 25th percentile (also known as the first quartile) is the value below which 25% of the observations in the dataset fall, while the 75th percentile (also known as the third quartile) is the value below which 75% of the observations in the dataset fall.

Percentiles can be used to summarize the distribution of a dataset and identify outliers. In machine learning, percentiles are often used in data preprocessing and exploratory data analysis to gain insights into the data. Python provides several libraries for calculating percentiles, including NumPy and Pandas.

Example using Numpy

Below is an example of how to calculate percentiles using NumPy −

import numpy as np

data = np.array([1, 2, 3, 4, 5])

p25 = np.percentile(data, 25)
p75 = np.percentile(data, 75)

print('25th percentile:', p25)
print('75th percentile:', p75)

import numpy as np

data = np.array([1, 2, 3, 4, 5])

p25 = np.percentile(data, 25)

p75 = np.percentile(data, 75)

print('25th percentile:', p25)

print('75th percentile:', p75)

In this example, we create a sample dataset using NumPy and then calculate the 25th and 75th percentiles using the np.percentile() function.

Example using Pandas

Below is an example of how to calculate percentiles using Pandas −

import pandas as pd

data = pd.Series([1, 2, 3, 4, 5])

p25 = data.quantile(0.25)
p75 = data.quantile(0.75)

print('25th percentile:', p25)
print('75th percentile:', p75)

import pandas as pd

data = pd.Series([1, 2, 3, 4, 5])

p25 = data.quantile(0.25)

p75 = data.quantile(0.75)

print('25th percentile:', p25)

print('75th percentile:', p75)

In this example, we create a Pandas series object and then calculate the 25th and 75th percentiles using the quantile() method of the series object.

Why Are Percentiles Important in AI Engineering?

In AI engineering, percentiles play an important role in helping us understand and manage data effectively. Since AI systems depend heavily on the quality of input data, knowing how the data is distributed is crucial. Percentiles allow us to:

Understand Data Distribution: Percentiles help show how values are spread out in a dataset. This is useful for spotting patterns and understanding where most of the data points lie.
Detect Outliers: Extreme values that fall far outside the normal range can negatively impact AI models. By checking values in the lower and upper percentiles (like the 1st or 99th), we can identify and handle outliers before training.
Set Thresholds and Benchmarks: In AI applications like fraud detection or risk scoring, percentiles help define cut-off points. For example, if the top 5% of users show risky behavior, the 95th percentile can serve as a warning threshold.
Feature Engineering: During preprocessing, engineers often use percentiles to normalize or categorize data. For instance, turning numerical data into percentile ranks helps compare values across different scales.
Bias and Fairness Checks: Percentiles can reveal hidden imbalances in data distributions between different groups, helping engineers design fairer and more balanced AI systems.

With this, you’ve reached the end of the percentiles in statistics. You’ve learned what a percentile is, why it’s useful in understanding data distribution, and how it plays an important role in machine learning and AI engineering.