Skip to content

feat(course): Improve Chapter 2.4 on Data Analysis #29

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
google-labs-jules wants to merge 2 commits into from
Closed

feat(course): Improve Chapter 2.4 on Data Analysis #29

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ea9add5
feat(course): Improve Chapter 2.4 on Data Analysis
google-labs-jules[bot] Aug 7, 2025
7764fdf
Merge branch 'main' into improve-chapter-2-4
fmind Aug 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
127 changes: 60 additions & 67 deletions docs/2. Prototyping/2.4. Analysis.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,100 +1,93 @@
---
description: Explore techniques for conducting comprehensive data analysis in notebooks, including visualizations, statistical tests, and exploratory data analysis (EDA) practices.
---
# 📈 2.4. Analysis

# 2.4. Analysis
After loading our dataset, the next crucial step is to perform Exploratory Data Analysis (EDA). EDA is the process of investigating our dataset to discover patterns, spot anomalies, test hypotheses, and check assumptions with the help of summary statistics and graphical representations. It's about getting to know our data before we start modeling.

## What is Exploratory Data Analysis (EDA)?
## 📊 Statistics

[Exploratory Data Analysis (EDA)](https://en.wikipedia.org/wiki/Exploratory_data_analysis) is a critical step in the data analysis process which involves investigating and summarizing the main characteristics of a dataset, often with visual methods. The goal of EDA is to obtain a deep understanding of the data’s underlying structures and variables, to detect outliers and anomalies, to uncover patterns, and to test assumptions with the help of statistical summaries and graphical representations.

EDA is a flexible, data-driven approach that allows for a more in-depth understanding of the data before making any assumptions. It serves as a foundation for formulating hypotheses, defining a more targeted analysis, and selecting appropriate models and algorithms for machine learning projects.

## How can you use pandas to analyze your data?

Dataframe libraries like [Pandas](https://pandas.pydata.org/) are an essential tool for EDA in Python, offering a wide array of functions to quickly slice, dice, and summarize your data. To begin analyzing your dataset with pandas, you can use the following methods:

- [`.info()`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.info.html): This method provides a concise summary of a DataFrame, giving you a quick overview of the data types, non-null values, and memory usage. It's a good starting point to understand the structure of your dataset.
- [`.describe(include='all')`](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.describe.html): Generates descriptive statistics that summarize the central tendency, dispersion, and shape of the dataset's distributions. By setting `include='all'`, you ensure that both numeric and object column types are included in the output, offering a more comprehensive view.

Here’s how you might use these methods in practice:
A great starting point for EDA is to compute descriptive statistics. This gives us a quantitative summary of our data.

```python
import pandas as pd
df = pd.read_csv('your_dataset.csv')
# Get a concise summary of the DataFrame
df.info()
# Get descriptive statistics for all columns
df.describe(include='all')

df = pd.read_csv("data/dataset.csv")
df.describe()
```

![Dataset statistics](../img/analysis/statistics.png)
<img src="../../img/analysis/statistics.png" width="100%">

The `describe()` function provides the following key statistics for each numerical column:

These functions allow you to quickly assess the quality and characteristics of your data, facilitating the identification of areas that may require further investigation or preprocessing.
- **count**: The number of non-null observations.
- **mean**: The average of the values.
- **std**: The standard deviation, which measures the amount of variation or dispersion of a set of values.
- **min**: The minimum value.
- **25%**: The first quartile (Q1), or the 25th percentile.
- **50%**: The median (Q2), or the 50th percentile.
- **75%**: The third quartile (Q3), or the 75th percentile.
- **max**: The maximum value.

## How can you visualize patterns in your dataset?
By examining these statistics, we can quickly grasp the scale of each feature and identify potential outliers. For instance, a large difference between the mean and the median might suggest a skewed distribution.

Visualizing patterns in your dataset is pivotal for EDA, as it helps in recognizing underlying structures, trends, and outliers that might not be apparent from the raw data alone. Python offers a wealth of libraries for data visualization, including:
## 🎨 Visualizations

- **[Plotly Express](https://plotly.com/python/plotly-express/)**: A high-level interface for interactive graphing.
- **[Matplotlib](https://matplotlib.org/)**: A widely used library for creating static, animated, and interactive visualizations.
- **[Seaborn](https://seaborn.pydata.org/)**: A library based on matplotlib that provides a high-level interface for drawing attractive statistical graphics.
While statistics give us a summary, visualizations help us see the story behind the numbers. A picture is worth a thousand words, especially in data analysis.

For instance, [Plotly Express's `scatter_matrix`](https://plotly.com/python/splom/) can be utilized to explore relationships between multiple variables:
### Scatter Matrix

A scatter matrix (or pair plot) is a fantastic tool for visualizing the relationships between multiple variables at once. It creates a grid of scatter plots for each pair of variables, and the diagonal of the grid shows the distribution of each individual variable (often as a histogram or a Kernel Density Estimate plot).

```python
import plotly.express as px
df = pd.read_csv('your_dataset.csv')
px.scatter_matrix(
df, dimensions=["feature1", "feature2", "feature3"], color="target_variable",
height=800, title="Scatter Matrix of Features"
)
```
from pandas.plotting import scatter_matrix

![Analysis scatter matrix](../img/analysis/scatter_matrix.png)
scatter_matrix(df, alpha=0.2, figsize=(6, 6), diagonal="kde")
```

This method enables the rapid exploration of pairwise relationships within a dataset, facilitating the identification of patterns, correlations, and potential hypotheses for deeper analysis.
<img src="../../img/analysis/scatter_matrix.png" width="100%">

## Is there a way to automate EDA?
From a scatter matrix, we can quickly identify:
- **Correlations**: If the points in a scatter plot form a line, it indicates a linear relationship between the variables.
- **Distributions**: The diagonal plots show us the shape of each variable's distribution (e.g., normal, skewed, bimodal).
- **Outliers**: Points that fall far from the main cluster of points can be potential outliers.

There are libraries designed to automate the EDA process, significantly reducing the time and effort required to understand a dataset. One such library is **[ydata-profiling](https://docs.profiling.ydata.ai/latest/)**, which generates comprehensive reports from a pandas DataFrame, providing insights into the distribution of each variable, correlations, missing values, and much more.
### Correlation Heatmap

Example with ydata-profiling:
To get a more quantitative view of the correlations between variables, we can compute the correlation matrix and visualize it as a heatmap.

```python
from ydata_profiling import ProfileReport
df = pd.read_csv('your_dataset.csv')
profile = ProfileReport(df, title='Pandas Profiling Report', minimal=True)
profile.to_widgets()
import seaborn as sns
import matplotlib.pyplot as plt

correlation_matrix = df.corr()
plt.figure(figsize=(8, 6))
sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', fmt=".2f")
plt.title("Correlation Matrix")
plt.show()
```
*Note: You may need to install seaborn (`pip install seaborn`) and matplotlib (`pip install matplotlib`) if you don't have them already.*

While automated EDA tools like ydata-profiling can offer a quick and broad overview of the dataset, they are not a complete substitute for manual EDA. Human intuition and expertise are crucial for asking the right questions, interpreting the results, and making informed decisions on how to proceed with the analysis. Therefore, automated EDA should be viewed as a complement to, rather than a replacement for, traditional exploratory data analysis methods.

## How can you handle missing values in datasets?
A heatmap provides a clear, color-coded view of the strength and direction of correlations:
- **Positive Correlation (Warm Colors)**: As one variable increases, the other tends to increase.
- **Negative Correlation (Cool Colors)**: As one variable increases, the other tends to decrease.
- **No Correlation (Neutral Colors)**: No clear relationship between the variables.

Handling missing values in datasets is crucial for maintaining data integrity. Here are common methods:
## 🗑️ Handling Missing Values

1. **Remove Data**: Delete rows with missing values, especially if the missing data is minimal.
2. **Impute Values**: Replace missing values with a statistical substitute like mean, median, or mode, or use predictive modeling.
3. **Indicator Variables**: Create new columns to indicate data is missing, which can be useful for some models.
Real-world datasets are often messy and may contain missing values. It's crucial to identify and handle them before modeling, as they can cause errors or lead to biased results.

[MissingNo](https://github.com/ResidentMario/missingno) is a tool for visualizing missing data in Python. To use it:

1. **Install MissingNo**: `pip install missingno`
2. **Import and Use**:
First, let's check for missing values:

```python
import missingno as msno
import pandas as pd

data = pd.read_csv('your_data.csv')
msno.matrix(data) # Visual matrix of missing data
msno.bar(data) # Bar chart of non-missing values
df.isnull().sum()
```
This will show the number of missing values for each column.

These visualizations help identify patterns and distributions of missing data, aiding in effective preprocessing decisions.
If there are missing values, common strategies include:
- **Deletion**: Removing rows or columns with missing values. This is suitable if the amount of missing data is small.
- **Imputation**: Filling in the missing values. For numerical data, this could be the mean, median, or mode of the column. For categorical data, it's often the mode.

## Analysis additional resources
## 🔑 Key Takeaways

- **[Example from the MLOps Python Package](https://github.com/fmind/mlops-python-package/blob/main/notebooks/prototype.ipynb)**
- [10 minutes to pandas](https://pandas.pydata.org/docs/user_guide/10min.html)
- **EDA is Fundamental**: Exploratory Data Analysis is a critical first step in any data science project.
- **Combine Statistics and Visuals**: Use both descriptive statistics (`.describe()`) and visualizations to get a comprehensive understanding of your data.
- **Visualize Relationships**: Scatter matrices and correlation heatmaps are powerful tools for uncovering relationships between variables.
- **Address Missing Data**: Always check for and handle missing values appropriately to ensure the quality of your analysis and models.