This repository contains an integration test suite designed to validate the correct behavior of the MCP server ignfab/geocontext across different language models.
The main goal is to ensure that end-to-end interactions with ignfab/geocontext remain stable and consistent when switching model providers or model versions.
- Agent creation using a configured model name
- Real model invocation through integration scenarios
- Functional response checks for expected outputs
See CODING.md for comprehensive setup instructions including:
- Environment configuration and API key setup
- Running tests with different models
- Testing a local version of geocontext
- Debugging and troubleshooting
Generated with geocontext v0.9.8 :
- reports/claude-haiku-4-5.md : OK
- reports/claude-sonnet-4-6.md : OK
- reports/gemini-3.5-flash.md : OK
- reports/gemini-3.1-flash-lite.md : 1 test KO (
gpf_wfs_describe_type tool was not called) - reports/gemma4-26b-moe.md : 2 tests KO (ERP,
gpf_wfs_describe_type tool was not called) - reports/qwen3-6-35b-moe.md : 2 test KO (ERP and detailed counts for test_count_lycees_2km_chateau_vincennes.py...)
- reports/ministral-14b-latest.md : 3 tests KO (ERP and prompt ignored for test_geocode_address_not_found.py)
- reports/mistral-small-latest.md : 1 test KO (ERP)
- reports/mistral-medium-latest.md : OK
- reports/albert-openai-gpt-oss-120b.md : 2 tests KO (ERP)
About ERP, see gpf-schema-store#48 and gpf-schema-store#56
Running the tests also writes the details of each conversation to reports/<model>/<test name>.txt
(not versioned, see CODING.md).
export ANTHROPIC_API_KEY=YourKey
# to run with some anthropic model
uv run scripts/run_tests.py config/models-anthropic.yaml
# to rerun with a single model
uv run scripts/run_tests.py config/models-anthropic.yaml --model=claude-haiku-4-5export MODEL_NAME="anthropic:claude-sonnet-4-6"
export ANTHROPIC_API_KEY=YourKey
uv run pytestexport MODEL_NAME="google_genai:gemini-3.5-flash"
export GOOGLE_API_KEY=YourKey
uv run pytestexport MODEL_NAME="google_genai:gemini-3.1-flash-lite-preview"
export GOOGLE_API_KEY=YourKey
uv run pytestTool names are given for geocontext 0.10.x (
GEOCONTEXT_DEV=1). With 0.9.x,gpf_*tools are namedgpf_wfs_*, counts rely ongpf_wfs_get_featuresand tests usingdistanceare skipped (see config/constants.py).
| File | Tool | Description |
|---|---|---|
| test_france_capital.py | none (LLM only) | Basic test without MCP: checks that the LLM answers "Paris" |
| test_tools_discovery.py | none (tool listing) | Checks that all the expected tools are exposed by the MCP server |
| test_geocode.py | geocode |
Coordinates of 1 rue de Rivoli, Paris |
| test_altitude.py | altitude |
Altitude at (6.87, 45.92), Chamonix area (900-1200 m) |
| test_adminexpress.py | adminexpress |
Commune and department at (2.35, 48.85): Paris, 75 |
| test_cadastre.py | cadastre |
Cadastral parcel at 73 avenue de Paris, Saint-Mandé |
| test_urbanisme.py | urbanisme |
Urban planning rules at 73 avenue de Paris, Saint-Mandé |
| test_assiette_sup.py | assiette_sup |
Public utility easements at (4.83, 45.76), Lyon |
| test_describe_type.py | gpf_describe_type |
Attributes of the BDTOPO_V3:batiment table |
| test_get_features.py | gpf_get_features |
BD TOPO buildings near Chamonix |
| test_get_feature_by_id.py | gpf_get_feature_by_id |
code_insee of a commune given its feature_id (first commune of the WFS) |
| test_search_batiment.py | gpf_search_types |
Search for tables about buildings |
| test_search_ecoles.py | gpf_search_types |
Search for tables about schools |
| File | Tool chain | Description |
|---|---|---|
| test_chaining_geocode_altitude.py | geocode → altitude |
Altitude of Chamonix town hall (900-1200 m) |
| test_chaining_geocode_adminexpress.py | geocode → adminexpress |
1 rue de Rivoli → commune and department |
| test_chaining_geocode_assiette_sup.py | geocode → assiette_sup |
10 place Bellecour, Lyon → public utility easements |
| test_chaining_cadastre_urbanisme.py | geocode → cadastre → urbanisme |
Address → parcel → urban planning rules |
| test_chaining_discovery.py | gpf_search_types → gpf_describe_type → gpf_get_features |
Full discovery: watercourse near the Eiffel Tower (Seine) |
| test_count_batiment_saint_mande.py | geocode → gpf_search_types → gpf_count_features |
Number of buildings in Saint-Mandé (1700-1800) |
| test_count_batiment_30m_angouleme.py | geocode → gpf_search_types → gpf_describe_type → gpf_count_features |
Number of buildings higher than 30 m in Angoulême (19) |
| test_count_lycees_2km_chateau_vincennes.py | geocode → gpf_search_types → gpf_describe_type → gpf_count_features |
Number of high schools within 2 km of the Château de Vincennes (14) |
| test_count_lycees_15min_mairie_saint_mande.py | gpf_search_types → gpf_describe_type → gpf_count_features / gpf_get_features (travel_time_filter) |
Number of high schools within 15 minutes on foot from the Saint-Mandé town hall, coordinates given (7) |
| test_area_lac_daumesnil.py | gpf_search_types → gpf_describe_type → gpf_get_features (spatial_extras: area) |
Area of the Lac Daumesnil (~101 000 m², ~10.1 ha) |
| test_nearest_lycee_chateau_vincennes.py | gpf_search_types → gpf_describe_type → gpf_get_features (dwithin_point_filter, spatial_extras: distance_to_filter_center) |
Nearest high school to the Château de Vincennes, coordinates, radius (2 km) and limit (20) given (Lycée Professionnel Jean Moulin) |
| test_hauteur_tour_montparnasse.py | gpf_search_types → gpf_describe_type → gpf_get_features (intersects_point_filter) |
Height of the building at a given point, the Tour Montparnasse (207.4 m ± 10 m) |
| test_count_batiment_bbox_chateau_vincennes.py | gpf_search_types → gpf_count_features (bbox_filter) |
Number of buildings in a bounding box around the Château de Vincennes, bbox given (238) |
| test_highest_batiment_bbox_chateau_vincennes.py | gpf_search_types → gpf_describe_type → gpf_get_features (bbox_filter) |
Highest building in a bounding box around the Château de Vincennes, bbox given (the donjon, 50.1 m ± 1 m) |
| test_map_lycees_2km_chateau_vincennes.py | geocode → gpf_search_types → gpf_describe_type → gpf_get_features_layer → show_map |
Map of the high schools within 2 km of the Château de Vincennes: show_map called with the data_url of the layer (14 features) |
| test_map_commune_saint_mande.py | adminexpress → gpf_get_feature_by_id_layer → show_map |
Map of the commune of Saint-Mandé: show_map called with the data_url of the layer (1 commune) |
| test_count_batiment_vendee.py | gpf_search_types → gpf_get_features → gpf_count_features (intersects_feature_filter) |
Number of buildings in the Vendée department (800 000-900 000). Skipped by default until #42 is fixed (SKIP_TEST_COUNT_BATIMENT_VENDEE=0 to run it) |
| test_chaining_geocode_distance.py | geocode → distance |
Walking distance between the Gare de Lyon and the Eiffel Tower (5-9 km) |
| test_chaining_distance_piscine_caen.py | geocode → gpf_search_types → gpf_describe_type → gpf_get_features → distance |
Walking time from the cinéma LUX (Caen) to the nearest swimming pool: Sivom, Mondeville (25-35 min) |
| File | Description |
|---|---|
| test_geocode_address_not_found.py | Non-existent address: checks that the agent reports "adresse non trouvée" |
| File | Description |
|---|---|
| test_mcp_concurrency.py | Concurrent tool calls must not be crossed. Skipped by default until #33 is fixed (SKIP_TEST_MCP_CONCURRENCY=0 to run it) |
| Tool | Description |
|---|---|
show_map |
show_map(title, data_url) mimics a map MCP (MCP Carto, ...) and returns "map displayed". It is added to the agent of all the tests (see conftest.py and #43) |
| Case | Criticality | Description |
|---|---|---|
| Ambiguous address | Medium | No test with an address matching several results (e.g. "rue de la République" without a city) |
| Overseas territories (DOM-TOM) | Medium | No test on overseas territories (Réunion, Guadeloupe) |
| Network error / tool error recovery | Low | No test of the agent's ability to retry after a tool error |