From 5dce805c09927628395d2d78c65a461dd854277f Mon Sep 17 00:00:00 2001 From: "google-labs-jules[bot]" <161369871+google-labs-jules[bot]@users.noreply.github.com> Date: Thu, 7 Aug 2025 19:16:54 +0000 Subject: [PATCH 1/2] feat(course): Enhance Section 2.2 on Configurations This commit significantly improves Section 2.2 of the MLOps Coding Course, focusing on configurations. The changes aim to make the content richer, more engaging, and easier for you to follow. The key improvements include: - Added emojis to all section titles for better visual engagement. - Restructured the content to flow from basic in-notebook configurations to more advanced, production-ready practices. - Added a new, comprehensive section on "Beyond Notebooks: External Configuration Files," which covers: - The benefits of using external configuration files (e.g., YAML). - A tutorial on loading and parsing YAML files with PyYAML. - An introduction to data validation for configurations using Pydantic, with clear code examples. - A dedicated subsection on managing multiple environments (development, production) with a base and override strategy. - An overview of advanced configuration management libraries like Dynaconf. - Best practices for securely managing secrets and API keys using environment variables and `.env` files. - Added a "Key Takeaways" section to summarize the most important concepts and best practices discussed in the document. These enhancements provide you with a clear, practical, and scalable approach to configuration management in your MLOps projects. --- docs/2. Prototyping/2.2. Configs.md | 290 +++++++++++++++++++++++++++- 1 file changed, 281 insertions(+), 9 deletions(-) diff --git a/docs/2. Prototyping/2.2. Configs.md b/docs/2. Prototyping/2.2. Configs.md index 28bc8ac9..40006895 100644 --- a/docs/2. Prototyping/2.2. Configs.md +++ b/docs/2. Prototyping/2.2. Configs.md @@ -2,9 +2,9 @@ description: Understand the importance of configurations in notebooks and how to structure them effectively to manage project settings and parameters for easy experimentation and reproducibility. --- -# 2.2. Configs +# βš™οΈ 2.2. Configs -## What are configs? +## πŸ€” What are configs? [Configurations](https://en.wikipedia.org/wiki/Computer_configuration), often abbreviated as "configs," serve as a cornerstone in programming. They encapsulate a set of parameters or settings designed to adapt the behavior of your code. By employing configs, you introduce a layer of flexibility and customization, enabling easy adjustments of critical variables without the need to tamper with the core logic of your codebase. This strategy not only enhances code usability but also its adaptability across various scenarios. @@ -33,7 +33,7 @@ PARAM_GRID = { } ``` -## Why should you create configs? +## πŸ‘ Why should you create configs? Incorporating configs into your projects is a reflection of best practices in software development. This approach ensures your code remains: @@ -44,7 +44,7 @@ Incorporating configs into your projects is a reflection of best practices in so Effectively, configurations act as a universal "remote control" for your code, offering an accessible interface for fine-tuning its behavior. -## Which configs can you provide out of the box? +## 🎁 Which configs can you provide out of the box? When it comes to data science projects, several common configurations are frequently utilized, including: @@ -63,7 +63,7 @@ TEST_SIZE = 0.2 RANDOM_STATE = 0 ``` -## How should you organize the configs in your notebook? +## πŸ—‚οΈ How should you organize the configs in your notebook? A logical and functional organization of your configurations can significantly enhance the readability and maintainability of your code. Grouping configs based on their purpose or domain of application is advisable: @@ -87,7 +87,7 @@ Details on defining and executing model pipelines ... Such categorization makes it easier for both users and developers to navigate and modify configurations as needed. -## What are options? +## βš™οΈ What are options? In the context of data science notebooks, options are akin to configurations but are specifically tied to the behavior and presentation of libraries such as [pandas](https://pandas.pydata.org/), [matplotlib](https://matplotlib.org/), and [scikit-learn](https://scikit-learn.org/stable/). These options offer a means to customize various aspects, including display settings and output formats, to suit individual needs or project requirements. @@ -103,7 +103,7 @@ pd.options.display.max_columns = None # Adjust sklearn output format sklearn.set_config(transform_output="pandas") ``` -## Why do you need to pass options? +## ❓ Why do you need to pass options? Library defaults may not always cater to your specific needs or the demands of your project. For instance: @@ -112,7 +112,7 @@ Library defaults may not always cater to your specific needs or the demands of y Adjusting these options helps tailor the working environment to better fit your workflow and analytical needs, ensuring that outputs are both informative and visually accessible. -## How should you configure library options? +## πŸ”§ How should you configure library options? To optimize your working environment, consider customizing the settings of key libraries according to your project's needs. Here are some guidelines: @@ -146,6 +146,278 @@ import sklearn sklearn.set_config(transform_output='pandas') ``` -## Configs additional resources +## πŸš€ Beyond Notebooks: External Configuration Files + +While defining configurations directly within a notebook is great for quick prototypes, a more robust and scalable approach is to use external configuration files. This practice, known as "separation of concerns," decouples your settings from your code, making your project cleaner, more maintainable, and easier to transition to production. + +### Why Use External Config Files? + +- **Centralized Management**: All your settings are in one place. +- **Environment-Specific Settings**: Easily manage different configurations for development, testing, and production environments. +- **Improved Readability**: Keeps your notebooks and scripts focused on logic, not setup. +- **Collaboration**: Team members can understand and modify configurations without digging through code. + +### πŸ“„ Using YAML for Configurations + +[YAML](https://yaml.org/) (YAML Ain't Markup Language) is a popular choice for configuration files because it's human-readable and easy to write. It uses indentation to denote structure, much like Python. + +Here’s how you can structure a `config.yml` file for a typical machine learning project: + +```yaml +# config.yml +project_name: 'mlops-course-project' + +data: + raw_path: 'data/raw/bike_sharing.csv' + processed_path: 'data/processed/bike_sharing_processed.csv' + target_column: 'cnt' + +model: + name: 'RandomForestRegressor' + params: + n_estimators: 200 + max_depth: 15 + min_samples_leaf: 4 + random_state: 42 + +experiment: + tracking_uri: 'mlruns' + name: 'bike-sharing-demand' +``` + +### 🐍 Loading Configurations in Python + +To use your `config.yml` file, you need to load it into your Python script or notebook. The `PyYAML` library is the standard tool for this. + +First, install it: +```bash +pip install pyyaml +``` + +Then, you can load the configuration like this: + +```python +import yaml + +def load_config(path='config.yml'): + with open(path, 'r') as f: + return yaml.safe_load(f) + +config = load_config() +print(f"Project Name: {config['project_name']}") +print(f"Model Name: {config['model']['name']}") +``` + +### βœ… Validating Configurations with Pydantic + +Hard-to-trace errors often arise from misconfigured settings (e.g., a typo in a key or a wrong data type). [Pydantic](https://docs.pydantic.dev/) helps prevent this by validating your configurations against a defined schema. + +First, install Pydantic: +```bash +pip install pydantic +``` + +Now, you can define Pydantic models that mirror your YAML structure and automatically parse and validate your config file. + +```python +import yaml +from pydantic import BaseModel, Field +from typing import Dict, Any + +# Define Pydantic models for structured configuration +class DataConfig(BaseModel): + raw_path: str + processed_path: str + target_column: str + +class ModelConfig(BaseModel): + name: str + params: Dict[str, Any] + +class ExperimentConfig(BaseModel): + tracking_uri: str + name: str + +class AppConfig(BaseModel): + project_name: str + data: DataConfig + model: ModelConfig + experiment: ExperimentConfig + +# Load and validate the configuration +with open("config.yml", "r") as f: + config_dict = yaml.safe_load(f) + +config = AppConfig(**config_dict) + +# Now you can access config with autocompletion and type-safety +print(f"Using model: {config.model.name} with {config.model.params['n_estimators']} estimators.") +``` +Using Pydantic not only catches errors early but also provides modern Python features like type hints and autocompletion in your editor, making your code more robust and developer-friendly. + +### 🌐 Managing Multiple Environments + +As your project grows, you'll likely need different configurations for different environments, such as `development`, `staging`, and `production`. For example, your production environment might use a different database or larger machine learning models than your development setup. + +A common pattern is to have a base configuration file and override specific settings with environment-specific files. + +**1. Create a `default.yml` for base settings:** + +```yaml +# config/default.yml +project_name: 'mlops-course-project' + +data: + raw_path: 'data/raw/bike_sharing.csv' + processed_path: 'data/processed/bike_sharing_processed.csv' + target_column: 'cnt' + +model: + name: 'RandomForestRegressor' + params: + n_estimators: 10 + max_depth: 5 + random_state: 42 +``` + +**2. Create an environment-specific file, e.g., `production.yml`:** + +You only need to specify the values that are different from the `default.yml`. + +```yaml +# config/production.yml +model: + params: + n_estimators: 300 + max_depth: 20 +``` + +**3. Load the configuration based on an environment variable:** + +You can use an environment variable (e.g., `APP_ENV`) to determine which configuration to load. + +```python +import os +import yaml + +def load_config(): + env = os.getenv('APP_ENV', 'development') # Default to 'development' + + with open('config/default.yml', 'r') as f: + config = yaml.safe_load(f) + + env_config_path = f'config/{env}.yml' + if os.path.exists(env_config_path): + with open(env_config_path, 'r') as f: + env_config = yaml.safe_load(f) + # Deep merge the environment-specific config into the base config + for key, value in env_config.items(): + if isinstance(value, dict) and key in config: + config[key].update(value) + else: + config[key] = value + return config + +# To load production settings, set the environment variable: +# export APP_ENV=production +config = load_config() +print(f"Running in '{os.getenv('APP_ENV', 'development')}' environment.") +print(f"Model estimators: {config['model']['params']['n_estimators']}") +``` + +### ⚑️ Advanced Configuration Management with Dynaconf + +For even more complex projects, manually managing configurations can become cumbersome. Libraries like [Dynaconf](https://www.dynaconf.com/) offer a powerful, streamlined solution. + +**Key features of Dynaconf:** + +- **Layered Configurations**: Automatically merges default, development, and production settings. +- **Multiple File Formats**: Supports `.toml`, `.yaml`, `.json`, `.ini`, and `.py` files. +- **Environment Variable Integration**: Seamlessly reads and casts environment variables. +- **Secrets Management**: Integrates with tools like HashiCorp Vault, AWS KMS, and GCP KMS. +- **Pydantic Validation**: Built-in support for validating configurations with Pydantic models. + +Here's a quick look at how Dynaconf simplifies things: + +**1. Install Dynaconf:** +```bash +pip install dynaconf +``` + +**2. Organize your config files:** +Create a `config` directory with `default.toml` and `production.toml` (or `.yaml`). Dynaconf will automatically find and layer them. + +**3. Load settings in your code:** +```python +from dynaconf import Dynaconf + +settings = Dynaconf( + envvar_prefix="DYNACONF", + settings_files=['config/default.toml', 'config/production.toml'], + environments=True, +) + +# Set the environment (e.g., export DYNACONF_ENV=production) +# Dynaconf automatically loads the correct settings. +print(f"Model Name: {settings.model.name}") +print(f"N_Estimators: {settings.model.params.n_estimators}") + +``` +Dynaconf handles the complexity of merging, validation, and secrets, allowing you to focus on your application's logic. + +### πŸ”’ Managing Secrets and Environment Variables + +It's a critical security practice to **never** store sensitive information like API keys, database passwords, or other credentials directly in your configuration files. Instead, you should use environment variables. + +You can load environment variables in Python using the `os` module. + +```python +import os + +# Best practice: Load secrets from environment variables +# For example, you might set this in your shell: +# export WANDB_API_KEY='your-secret-key' + +wandb_api_key = os.getenv('WANDB_API_KEY') + +if wandb_api_key: + print("W&B API Key loaded successfully.") +else: + print("W&B API Key not found. Please set the WANDB_API_KEY environment variable.") +``` + +Libraries like `python-dotenv` can also help manage environment variables in local development by loading them from a `.env` file. +```bash +pip install python-dotenv +``` + +Create a `.env` file (and add it to your `.gitignore`!): +``` +# .env +WANDB_API_KEY='your-secret-key-for-local-dev' +``` + +And load it in your script: +```python +from dotenv import load_dotenv +import os + +load_dotenv() # loads variables from .env file + +wandb_api_key = os.getenv('WANDB_API_KEY') +print(f"Loaded API key: {wandb_api_key}") +``` +This approach keeps your secrets safe and your configurations clean. + +## πŸ”‘ Key Takeaways + +- **Start Simple, Scale Up**: Begin with simple variable-based configurations in your notebooks for prototyping, but plan to move to external files as your project grows. +- **Separate Code from Configuration**: Use external files like `config.yml` to keep your settings decoupled from your application logic. This improves maintainability, readability, and makes your code easier to adapt for different environments. +- **Validate Your Configurations**: Use tools like Pydantic to validate your settings. This catches errors early, prevents bugs, and provides features like autocompletion and type safety. +- **Manage Environments Explicitly**: Use different configuration files (e.g., `default.yml`, `production.yml`) for different environments (development, staging, production) to manage settings cleanly and avoid mistakes. +- **Never Commit Secrets**: Always use environment variables for sensitive data like API keys and passwords. Never store them in your configuration files or commit them to version control. +- **Leverage Advanced Tools for Complex Projects**: For large projects, consider using libraries like Dynaconf to automate configuration management, including layering, secrets integration, and validation. + +## πŸ“š Configs additional resources - **[Configs example from the MLOps Python Package](https://github.com/fmind/mlops-python-package/blob/main/notebooks/prototype.ipynb)** From eb6d64a7024ec91050bf309034028ac301334539 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?M=C3=A9d=C3=A9ric=20Hurier=20=28Fmind=29?= Date: Thu, 7 Aug 2025 21:55:32 +0200 Subject: [PATCH 2/2] review --- docs/2. Prototyping/2.2. Configs.md | 42 +---------------------------- 1 file changed, 1 insertion(+), 41 deletions(-) diff --git a/docs/2. Prototyping/2.2. Configs.md b/docs/2. Prototyping/2.2. Configs.md index 40006895..e3cc23b6 100644 --- a/docs/2. Prototyping/2.2. Configs.md +++ b/docs/2. Prototyping/2.2. Configs.md @@ -325,46 +325,6 @@ print(f"Running in '{os.getenv('APP_ENV', 'development')}' environment.") print(f"Model estimators: {config['model']['params']['n_estimators']}") ``` -### ⚑️ Advanced Configuration Management with Dynaconf - -For even more complex projects, manually managing configurations can become cumbersome. Libraries like [Dynaconf](https://www.dynaconf.com/) offer a powerful, streamlined solution. - -**Key features of Dynaconf:** - -- **Layered Configurations**: Automatically merges default, development, and production settings. -- **Multiple File Formats**: Supports `.toml`, `.yaml`, `.json`, `.ini`, and `.py` files. -- **Environment Variable Integration**: Seamlessly reads and casts environment variables. -- **Secrets Management**: Integrates with tools like HashiCorp Vault, AWS KMS, and GCP KMS. -- **Pydantic Validation**: Built-in support for validating configurations with Pydantic models. - -Here's a quick look at how Dynaconf simplifies things: - -**1. Install Dynaconf:** -```bash -pip install dynaconf -``` - -**2. Organize your config files:** -Create a `config` directory with `default.toml` and `production.toml` (or `.yaml`). Dynaconf will automatically find and layer them. - -**3. Load settings in your code:** -```python -from dynaconf import Dynaconf - -settings = Dynaconf( - envvar_prefix="DYNACONF", - settings_files=['config/default.toml', 'config/production.toml'], - environments=True, -) - -# Set the environment (e.g., export DYNACONF_ENV=production) -# Dynaconf automatically loads the correct settings. -print(f"Model Name: {settings.model.name}") -print(f"N_Estimators: {settings.model.params.n_estimators}") - -``` -Dynaconf handles the complexity of merging, validation, and secrets, allowing you to focus on your application's logic. - ### πŸ”’ Managing Secrets and Environment Variables It's a critical security practice to **never** store sensitive information like API keys, database passwords, or other credentials directly in your configuration files. Instead, you should use environment variables. @@ -418,6 +378,6 @@ This approach keeps your secrets safe and your configurations clean. - **Never Commit Secrets**: Always use environment variables for sensitive data like API keys and passwords. Never store them in your configuration files or commit them to version control. - **Leverage Advanced Tools for Complex Projects**: For large projects, consider using libraries like Dynaconf to automate configuration management, including layering, secrets integration, and validation. -## πŸ“š Configs additional resources +## πŸ“š Additional resources - **[Configs example from the MLOps Python Package](https://github.com/fmind/mlops-python-package/blob/main/notebooks/prototype.ipynb)**