Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/docker-build-test-push-template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ on:
type: string
default: 'okdp-examples'
publish_to_registry:
description: Wheter to push to the registry
description: Whether to push to the registry
required: false
type: string
default: "false"
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/docker-rebuild.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ on:
# At 05:00 AM, only on Tuesday
#- cron: "0 5 * * 2"
# https://crontab.cronhub.io/
# At 05:"0 AM, only on Tuesday
# At 05:00 AM, only on Tuesday
- cron: "0 5 * * 2"
# The release should be created manually (or with user token=pr approval/merge) in order to trigger the event
### https://github.com/orgs/community/discussions/25281
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/helm-lint-template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ on:
workflow_call:
inputs:
skip_chart_test:
description: Wether to deploy and test the chart
description: Whether to deploy and test the chart
required: false
type: boolean
default: false
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -301,7 +301,7 @@ Storage access is controlled at the service level, while user-level data access

</details>

# Know issues
# Known issues
1. [Polaris - Spark Iceberg REST Catalog refresh token](https://github.com/apache/iceberg/issues/12363)
> Long-running jobs may need more metadata calls to Polaris during execution, not just one initial call
2. [Polaris - OAuth 2 grant type "refresh_token" not implemented](https://github.com/apache/iceberg/issues/12196)
Expand Down
8 changes: 4 additions & 4 deletions airflow/dags/nyc_taxi_pipeline.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
"""
NYC Taxi Pipeline - Airflow + Spark Operator
Utilise l'API Kubernetes Python pour soumettre un SparkApplication.
Uses the Kubernetes Python API to submit a SparkApplication.
"""
import os
import re
Expand Down Expand Up @@ -165,12 +165,12 @@ def submit_and_wait_nyc_taxi_etl(run_suffix, timeout_seconds=1200):
.get("state", "SUBMITTED")
)
if last_state == "COMPLETED":
return f"Spark ETL terminé: {app_name}"
return f"Spark ETL completed: {app_name}"
if last_state in {"FAILED", "SUBMISSION_FAILED", "UNKNOWN"}:
raise RuntimeError(f"Spark ETL échoué: {app_name} state={last_state}")
raise RuntimeError(f"Spark ETL failed: {app_name} state={last_state}")
time.sleep(10)

raise TimeoutError(f"Spark ETL timeout après {timeout_seconds}s: {app_name} state={last_state}")
raise TimeoutError(f"Spark ETL timeout after {timeout_seconds}s: {app_name} state={last_state}")


with DAG(
Expand Down
2 changes: 1 addition & 1 deletion airflow/tests/run_integration_tests.sh
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ SCHEDULER_POD="$(find_scheduler_pod)"
WEBSERVER_POD="$(find_webserver_pod)"

if [[ -z "${SCHEDULER_POD}" || -z "${WEBSERVER_POD}" ]]; then
echo "Pods Airflow introuvables dans namespace ${NAMESPACE}"
echo "Airflow pods not found in namespace ${NAMESPACE}"
exit 1
fi

Expand Down
Loading