feat(course): Enhance Section 2.2 on Configurations

docs/2. Prototyping/2.2. Configs.md

@@ -2,9 +2,9 @@
 description: Understand the importance of configurations in notebooks and how to structure them effectively to manage project settings and parameters for easy experimentation and reproducibility.
 ---
 
-# 2.2. Configs
+# ⚙️ 2.2. Configs
 
-## What are configs?
+## 🤔 What are configs?
 
 [Configurations](https://en.wikipedia.org/wiki/Computer_configuration), often abbreviated as "configs," serve as a cornerstone in programming. They encapsulate a set of parameters or settings designed to adapt the behavior of your code. By employing configs, you introduce a layer of flexibility and customization, enabling easy adjustments of critical variables without the need to tamper with the core logic of your codebase. This strategy not only enhances code usability but also its adaptability across various scenarios.
 
@@ -33,7 +33,7 @@ PARAM_GRID = {
 }
 ```
 
-## Why should you create configs?
+## 👍 Why should you create configs?
 
 Incorporating configs into your projects is a reflection of best practices in software development. This approach ensures your code remains:
 
@@ -44,7 +44,7 @@ Incorporating configs into your projects is a reflection of best practices in so
 
 Effectively, configurations act as a universal "remote control" for your code, offering an accessible interface for fine-tuning its behavior.
 
-## Which configs can you provide out of the box?
+## 🎁 Which configs can you provide out of the box?
 
 When it comes to data science projects, several common configurations are frequently utilized, including:
 
@@ -63,7 +63,7 @@ TEST_SIZE = 0.2
 RANDOM_STATE = 0
 ```
 
-## How should you organize the configs in your notebook?
+## 🗂️ How should you organize the configs in your notebook?
 
 A logical and functional organization of your configurations can significantly enhance the readability and maintainability of your code. Grouping configs based on their purpose or domain of application is advisable:
 
@@ -87,7 +87,7 @@ Details on defining and executing model pipelines ...
 
 Such categorization makes it easier for both users and developers to navigate and modify configurations as needed.
 
-## What are options?
+## ⚙️ What are options?
 
 In the context of data science notebooks, options are akin to configurations but are specifically tied to the behavior and presentation of libraries such as [pandas](https://pandas.pydata.org/), [matplotlib](https://matplotlib.org/), and [scikit-learn](https://scikit-learn.org/stable/). These options offer a means to customize various aspects, including display settings and output formats, to suit individual needs or project requirements.
 
@@ -104,7 +104,7 @@ pd.options.display.max_columns = None
 sklearn.set_config(transform_output="pandas")
 ```
 
-## Why do you need to pass options?
+## ❓ Why do you need to pass options?
 
 Library defaults may not always cater to your specific needs or the demands of your project. For instance:
 
@@ -113,7 +113,7 @@ Library defaults may not always cater to your specific needs or the demands of y
 
 Adjusting these options helps tailor the working environment to better fit your workflow and analytical needs, ensuring that outputs are both informative and visually accessible.
 
-## How should you configure library options?
+## 🔧 How should you configure library options?
 
 To optimize your working environment, consider customizing the settings of key libraries according to your project's needs. Here are some guidelines:
 
@@ -147,6 +147,238 @@ import sklearn
 sklearn.set_config(transform_output='pandas')
 ```
 
-## Configs additional resources
+## 🚀 Beyond Notebooks: External Configuration Files
+
+While defining configurations directly within a notebook is great for quick prototypes, a more robust and scalable approach is to use external configuration files. This practice, known as "separation of concerns," decouples your settings from your code, making your project cleaner, more maintainable, and easier to transition to production.
+
+### Why Use External Config Files?
+
+- **Centralized Management**: All your settings are in one place.
+- **Environment-Specific Settings**: Easily manage different configurations for development, testing, and production environments.
+- **Improved Readability**: Keeps your notebooks and scripts focused on logic, not setup.
+- **Collaboration**: Team members can understand and modify configurations without digging through code.
+
+### 📄 Using YAML for Configurations
+
+[YAML](https://yaml.org/) (YAML Ain't Markup Language) is a popular choice for configuration files because it's human-readable and easy to write. It uses indentation to denote structure, much like Python.
+
+Here’s how you can structure a `config.yml` file for a typical machine learning project:
+
+```yaml
+# config.yml
+project_name: 'mlops-course-project'
+
+data:
+  raw_path: 'data/raw/bike_sharing.csv'
+  processed_path: 'data/processed/bike_sharing_processed.csv'
+  target_column: 'cnt'
+
+model:
+  name: 'RandomForestRegressor'
+  params:
+    n_estimators: 200
+    max_depth: 15
+    min_samples_leaf: 4
+    random_state: 42
+
+experiment:
+  tracking_uri: 'mlruns'
+  name: 'bike-sharing-demand'
+```
+
+### 🐍 Loading Configurations in Python
+
+To use your `config.yml` file, you need to load it into your Python script or notebook. The `PyYAML` library is the standard tool for this.
+
+First, install it:
+```bash
+pip install pyyaml
+```
+
+Then, you can load the configuration like this:
+
+```python
+import yaml
+
+def load_config(path='config.yml'):
+    with open(path, 'r') as f:
+        return yaml.safe_load(f)
+
+config = load_config()
+print(f"Project Name: {config['project_name']}")
+print(f"Model Name: {config['model']['name']}")
+```
+
+### ✅ Validating Configurations with Pydantic
+
+Hard-to-trace errors often arise from misconfigured settings (e.g., a typo in a key or a wrong data type). [Pydantic](https://docs.pydantic.dev/) helps prevent this by validating your configurations against a defined schema.
+
+First, install Pydantic:
+```bash
+pip install pydantic
+```
+
+Now, you can define Pydantic models that mirror your YAML structure and automatically parse and validate your config file.
+
+```python
+import yaml
+from pydantic import BaseModel, Field
+from typing import Dict, Any
+
+# Define Pydantic models for structured configuration
+class DataConfig(BaseModel):
+    raw_path: str
+    processed_path: str
+    target_column: str
+
+class ModelConfig(BaseModel):
+    name: str
+    params: Dict[str, Any]
+
+class ExperimentConfig(BaseModel):
+    tracking_uri: str
+    name: str
+
+class AppConfig(BaseModel):
+    project_name: str
+    data: DataConfig
+    model: ModelConfig
+    experiment: ExperimentConfig
+
+# Load and validate the configuration
+with open("config.yml", "r") as f:
+    config_dict = yaml.safe_load(f)
+
+config = AppConfig(**config_dict)
+
+# Now you can access config with autocompletion and type-safety
+print(f"Using model: {config.model.name} with {config.model.params['n_estimators']} estimators.")
+```
+Using Pydantic not only catches errors early but also provides modern Python features like type hints and autocompletion in your editor, making your code more robust and developer-friendly.
+
+### 🌐 Managing Multiple Environments
+
+As your project grows, you'll likely need different configurations for different environments, such as `development`, `staging`, and `production`. For example, your production environment might use a different database or larger machine learning models than your development setup.
+
+A common pattern is to have a base configuration file and override specific settings with environment-specific files.
+
+**1. Create a `default.yml` for base settings:**
+
+```yaml
+# config/default.yml
+project_name: 'mlops-course-project'
+
+data:
+  raw_path: 'data/raw/bike_sharing.csv'
+  processed_path: 'data/processed/bike_sharing_processed.csv'
+  target_column: 'cnt'
+
+model:
+  name: 'RandomForestRegressor'
+  params:
+    n_estimators: 10
+    max_depth: 5
+    random_state: 42
+```
+
+**2. Create an environment-specific file, e.g., `production.yml`:**
+
+You only need to specify the values that are different from the `default.yml`.
+
+```yaml
+# config/production.yml
+model:
+  params:
+    n_estimators: 300
+    max_depth: 20
+```
+
+**3. Load the configuration based on an environment variable:**
+
+You can use an environment variable (e.g., `APP_ENV`) to determine which configuration to load.
+
+```python
+import os
+import yaml
+
+def load_config():
+    env = os.getenv('APP_ENV', 'development') # Default to 'development'
+
+    with open('config/default.yml', 'r') as f:
+        config = yaml.safe_load(f)
+
+    env_config_path = f'config/{env}.yml'
+    if os.path.exists(env_config_path):
+        with open(env_config_path, 'r') as f:
+            env_config = yaml.safe_load(f)
+            # Deep merge the environment-specific config into the base config
+            for key, value in env_config.items():
+                if isinstance(value, dict) and key in config:
+                    config[key].update(value)
+                else:
+                    config[key] = value
+    return config
+
+# To load production settings, set the environment variable:
+# export APP_ENV=production
+config = load_config()
+print(f"Running in '{os.getenv('APP_ENV', 'development')}' environment.")
+print(f"Model estimators: {config['model']['params']['n_estimators']}")
+```
+
+### 🔒 Managing Secrets and Environment Variables
+
+It's a critical security practice to **never** store sensitive information like API keys, database passwords, or other credentials directly in your configuration files. Instead, you should use environment variables.
+
+You can load environment variables in Python using the `os` module.
+
+```python
+import os
+
+# Best practice: Load secrets from environment variables
+# For example, you might set this in your shell:
+# export WANDB_API_KEY='your-secret-key'
+
+wandb_api_key = os.getenv('WANDB_API_KEY')
+
+if wandb_api_key:
+    print("W&B API Key loaded successfully.")
+else:
+    print("W&B API Key not found. Please set the WANDB_API_KEY environment variable.")
+```
+
+Libraries like `python-dotenv` can also help manage environment variables in local development by loading them from a `.env` file.
+```bash
+pip install python-dotenv
+```
+
+Create a `.env` file (and add it to your `.gitignore`!):
+```
+# .env
+WANDB_API_KEY='your-secret-key-for-local-dev'
+```
+
+And load it in your script:
+```python
+from dotenv import load_dotenv
+import os
+
+load_dotenv() # loads variables from .env file
+
+wandb_api_key = os.getenv('WANDB_API_KEY')
+print(f"Loaded API key: {wandb_api_key}")
+```
+This approach keeps your secrets safe and your configurations clean.
+
+## 🔑 Key Takeaways
+
+- **Start Simple, Scale Up**: Begin with simple variable-based configurations in your notebooks for prototyping, but plan to move to external files as your project grows.
+- **Separate Code from Configuration**: Use external files like `config.yml` to keep your settings decoupled from your application logic. This improves maintainability, readability, and makes your code easier to adapt for different environments.
+- **Validate Your Configurations**: Use tools like Pydantic to validate your settings. This catches errors early, prevents bugs, and provides features like autocompletion and type safety.
+- **Manage Environments Explicitly**: Use different configuration files (e.g., `default.yml`, `production.yml`) for different environments (development, staging, production) to manage settings cleanly and avoid mistakes.
+- **Never Commit Secrets**: Always use environment variables for sensitive data like API keys and passwords. Never store them in your configuration files or commit them to version control.
+- **Leverage Advanced Tools for Complex Projects**: For large projects, consider using libraries like Dynaconf to automate configuration management, including layering, secrets integration, and validation.
+
+## 📚 Additional resources
 
 - **[Configs example from the MLOps Python Package](https://github.com/fmind/mlops-python-package/blob/main/notebooks/prototype.ipynb)**