From ce6604ab374fd5548aa6a9d7908e61da1007c9d3 Mon Sep 17 00:00:00 2001 From: "Alina (Xi) Li" Date: Tue, 25 Nov 2025 11:51:45 -0800 Subject: [PATCH 1/2] ODBC Standalone MSVC CI Remove DataSet Fix cache key Use odbc-specific cache key Fix Lint and Add odbc registration step Link appropriate GitHub issues that blocks ODBC tests Disable test phase and add schedule Run everyday 7 am Vancouver time Enable ODBC build on MSVC CI Code Clean up and enable ODBC tests in CI Still need to modify ODBCUtilEnvironment if we decide to use it. Still need to add ODBC V2 support so a different env and conn is used for ODBC 2 tests. Draft enable ODBC global setup/teardown Run ODBC test once outside of test script If ODBC test is run using MinGW Shell, segfault occurs. I am not getting any seg fault on my local MSVC Windows when I run the tests without the bash script. But if Windows CI breaks from running the standalone exe then I will look into this Prepend vcpkg to search vcpkg before `/lib/cmake` Since we are installing dependencies on vcpkg, if `lib/cmake` is searched first, then cmake will look into that directory and use the wrong paths. Convert VCPKG Windows path to MSYS path Set `VCPKG_ROOT` in test phase Fix ODBC dll name Use `arrow_flight_sql_odbc.dll` instead of `libarrow_flight_sql_odbc.dll` which is the naming convention used on MinGW Windows Link ODBC library on all platforms On my local MSVC Windows machine, Visual Studio is used to build without needing to link ` ${ODBCINST}` explicitly, its behavior might be different from Ninja which is what CI uses. `${ODBCINST}` is likely needed by Linux as well, so adding it for all platforms. Attempt to resolve `arrow-compute-grouper-benchmark` build issue I think `if(ARROW_FLIGHT_TEST_LINKAGE STREQUAL "static")` might be more appropriate here, as I am seeing build issues due to both dynamic and static linking occurring. I see in https://github.com/apache/arrow/commit/59903d089ec5b1766e1622d94c1f618b539b205d, this check is used for `arrow-filesystem-s3fs-benchmark` Disable `UNITY_BUILD` for ODBC test due to conflict on `sqlite_sql_info.cc` On some workflows, unity build is set to ON to make the build faster. This is an attempt to resolve the build issue. Remove debug messages Fix lint Add `sql_info_undef.h` to sqlite_sql_info.cc Undefine duplicates in SQLGetInfo Add `#pragma once` to odbc_test_suite.h To avoid redefinition error Attempt to resolve conflict with sql/types.h Attempt to include Arrow headers before ODBC headers to avoid conflict with arrow/flight/sql/types.h [GH-48084] Replace boost::optional with std::optional Addresses comment https://github.com/apache/arrow/pull/40939#discussion_r2099118497 Replace boost::optional with std::optional in ODBC codebase https://github.com/apache/arrow/issues/48084 Fix lint Remove `BOOST_SOURCE=BUNDLED` to use vcpkg's dynamic link to boost Since we need to use link to boost dynamically, we need to remove the bundled boost library flag. Add debug messages. Remove `ARROW_BOOST_USE_SHARED = OFF` Since we have a release build, can enable boost as shared. With `ARROW_BOOST_USE_SHARED = OFF`, I am getting error ``` D:\a\arrow\arrow\build\cpp\vcpkg_installed\x64-windows\include\boost/filesystem/config.hpp(96): fatal error C1189: #error: Must not define both BOOST_FILESYSTEM_DYN_LINK and BOOST_FILESYSTEM_STATIC_LINK ``` Also increased time limit, since 2 hours don't seem to be enough to finish the vcpkg build Change to release build Getting ``` orc.lib(Exceptions.cc.obj) : error LNK2038: mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '0' doesn't match value '2' in unity_2_cxx.cxx.obj orc.lib(Exceptions.cc.obj) : error LNK2038: mismatch detected for 'RuntimeLibrary': value 'MD_DynamicRelease' doesn't match value 'MDd_DynamicDebug' in unity_2_cxx.cxx.obj ``` when building with debug and presumably ARROW_ORC Set ARROW_BOOST_USE_SHARED to OFF `ARROW_BOOST_USE_SHARED` is restored to `OFF`, which according to Windows doc should help with the debug build now. Retore value of `ARROW_BUILD_BENCHMARKS`, though noting it is set to `OFF` in GLib MSVC workflow. Add VCPKG set up Borrowed from the Glib workflow Add `/EHsc` flag * Add back `CMAKE_CXX_STANDARD: "17"` Remove `CMAKE_CXX_STANDARD` 17 * reason: gtest can't be used with C++17. Arrow project doesn't support C++ 17 yet. Extend run-time to 2hr windows-mingw also has timeout of 2hr Empty commit to trigger CI Specify `VCPKG_BINARY_SOURCES` and `VCPKG_DEFAULT_TRIPLET` Specify VCPKG_TARGET_TRIPLET as x64 windows Set ARROW_DEPENDENCY_SOURCE to VCPKG To make it easier to manage dependencies Enable static build on MSVC `ARROW_BUILD_BENCHMARKS` prevents Flight from being built Remove `ARROW_BOOST_USE_SHARED` Having static vs. shared issue Enable Flight & Flight SQL for ODBC Enable ODBC build on MSVC CI Enable regular ctest tests Remove undef items Add concurrency and permissions to odbc yml Create cpp_odbc.yml Fix architecture in CI --- .github/workflows/cpp.yml | 4 + .github/workflows/cpp_odbc.yml | 160 ++++++++++++++++++ ci/scripts/cpp_test.sh | 3 +- cpp/cmake_modules/SetupCxxFlags.cmake | 1 + .../flight/sql/odbc/odbc_impl/CMakeLists.txt | 10 +- .../sql/odbc/odbc_impl/get_info_cache.cc | 6 +- .../sql/odbc/odbc_impl/odbc_connection.cc | 1 + .../flight/sql/odbc/tests/CMakeLists.txt | 3 + .../flight/sql/odbc/tests/odbc_test_suite.h | 2 + 9 files changed, 180 insertions(+), 10 deletions(-) create mode 100644 .github/workflows/cpp_odbc.yml diff --git a/.github/workflows/cpp.yml b/.github/workflows/cpp.yml index c4ae7fe8b41a..e8b9c56b9d82 100644 --- a/.github/workflows/cpp.yml +++ b/.github/workflows/cpp.yml @@ -390,6 +390,10 @@ jobs: PIPX_BASE_PYTHON: ${{ steps.python-install.outputs.python-path }} run: | ci/scripts/install_gcs_testbench.sh default + - name: Register Flight SQL ODBC Driver + shell: cmd + run: | + call "cpp\src\arrow\flight\sql\odbc\tests\install_odbc.cmd" ${{ github.workspace }}\build\cpp\%ARROW_BUILD_TYPE%\libarrow_flight_sql_odbc.dll - name: Test shell: msys2 {0} run: | diff --git a/.github/workflows/cpp_odbc.yml b/.github/workflows/cpp_odbc.yml new file mode 100644 index 000000000000..0a97e6dfe1fc --- /dev/null +++ b/.github/workflows/cpp_odbc.yml @@ -0,0 +1,160 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +name: C++ ODBC + +on: + push: + branches: + - '**' + - '!dependabot/**' + tags: + - '**' + paths: + - '.github/workflows/cpp_odbc.yml' + - 'ci/scripts/cpp_*' + - 'cpp/src/arrow/flight/sql/odbc/*' + pull_request: + paths: + - '.github/workflows/cpp_odbc.yml' + - 'ci/scripts/cpp_*' + - 'cpp/src/arrow/flight/sql/odbc/*' + schedule: + - cron: '0 13 * * *' + +concurrency: + group: ${{ github.repository }}-${{ github.head_ref || github.sha }}-${{ github.workflow }} + cancel-in-progress: true + +permissions: + contents: read + +jobs: + windows: + runs-on: windows-2022 + timeout-minutes: 240 + env: + ARROW_BUILD_SHARED: ON + ARROW_BUILD_STATIC: ON + ARROW_BUILD_TESTS: ON + ARROW_BUILD_TYPE: release + ARROW_DEPENDENCY_SOURCE: VCPKG + ARROW_FLIGHT: ON + ARROW_FLIGHT_SQL: ON + ARROW_FLIGHT_SQL_ODBC: ON + ARROW_SIMD_LEVEL: AVX2 + CMAKE_CXX_STANDARD: "17" + CMAKE_GENERATOR: Ninja + CMAKE_INSTALL_PREFIX: /usr + VCPKG_BINARY_SOURCES: 'clear;nuget,GitHub,readwrite' + VCPKG_DEFAULT_TRIPLET: x64-windows + steps: + - name: Disable Crash Dialogs + run: | + reg add ` + "HKCU\SOFTWARE\Microsoft\Windows\Windows Error Reporting" ` + /v DontShowUI ` + /t REG_DWORD ` + /d 1 ` + /f + - name: Checkout Arrow + uses: actions/checkout@v5 + with: + fetch-depth: 0 + submodules: recursive + - name: Download Timezone Database + shell: bash + run: ci/scripts/download_tz_database.sh + - name: Install msys2 (for tzdata for ORC tests) + uses: msys2/setup-msys2@v2 + id: setup-msys2 + - name: Install cmake + shell: bash + run: | + ci/scripts/install_cmake.sh 4.1.2 /usr + - name: Install ccache + shell: bash + run: | + ci/scripts/install_ccache.sh 4.12.1 /usr + - name: Setup ccache + shell: bash + run: | + ci/scripts/ccache_setup.sh + - name: ccache info + id: ccache-info + shell: bash + run: | + echo "cache-dir=$(ccache --get-config cache_dir)" >> $GITHUB_OUTPUT + - name: Cache ccache + uses: actions/cache@v4 + with: + path: ${{ steps.ccache-info.outputs.cache-dir }} + key: cpp-odbc-ccache-windows-x64-${{ hashFiles('cpp/**') }} + restore-keys: cpp-odbc-ccache-windows-x64- + - name: Checkout vcpkg + uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0 + with: + fetch-depth: 0 + path: vcpkg + repository: microsoft/vcpkg + - name: Bootstrap vcpkg + run: | + vcpkg\bootstrap-vcpkg.bat + $VCPKG_ROOT = $(Resolve-Path -LiteralPath "vcpkg").ToString() + Write-Output ${VCPKG_ROOT} | ` + Out-File -FilePath ${Env:GITHUB_PATH} -Encoding utf8 -Append + Write-Output "VCPKG_ROOT=${VCPKG_ROOT}" | ` + Out-File -FilePath ${Env:GITHUB_ENV} -Encoding utf8 -Append + - name: Setup NuGet credentials for vcpkg caching + shell: bash + run: | + $(vcpkg fetch nuget | tail -n 1) \ + sources add \ + -source "https://nuget.pkg.github.com/$GITHUB_REPOSITORY_OWNER/index.json" \ + -storepasswordincleartext \ + -name "GitHub" \ + -username "$GITHUB_REPOSITORY_OWNER" \ + -password "${{ secrets.GITHUB_TOKEN }}" + $(vcpkg fetch nuget | tail -n 1) \ + setapikey "${{ secrets.GITHUB_TOKEN }}" \ + -source "https://nuget.pkg.github.com/$GITHUB_REPOSITORY_OWNER/index.json" + - name: Build + shell: cmd + run: | + set VCPKG_ROOT_KEEP=%VCPKG_ROOT% + call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + set VCPKG_ROOT=%VCPKG_ROOT_KEEP% + bash -c "ci/scripts/cpp_build.sh $(pwd) $(pwd)/build" + - name: Register Flight SQL ODBC Driver + shell: cmd + run: | + call "cpp\src\arrow\flight\sql\odbc\tests\install_odbc.cmd" ${{ github.workspace }}\build\cpp\%ARROW_BUILD_TYPE%\arrow_flight_sql_odbc.dll + # GH-48270 TODO: Resolve segementation fault during Arrow library unload + # GH-48269 TODO: Enable Flight & Flight SQL testing in MSVC CI + # TODO: enable ODBC tests after GH-48270 and GH-48269 are resolved. + # - name: Test + # shell: cmd + # run: | + # set VCPKG_ROOT_KEEP=%VCPKG_ROOT% + # call "C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Auxiliary\Build\vcvarsall.bat" x64 + # # For ORC + # set TZDIR=${{ steps.setup-msys2.outputs.msys2-location }}\usr\share\zoneinfo + + # # Convert VCPKG Windows path to MSYS path + # for /f "usebackq delims=" %%I in (`bash -c "cygpath -u \"$VCPKG_ROOT_KEEP\""` ) do set VCPKG_ROOT=%%I + + # bash -c "ci/scripts/cpp_test.sh $(pwd) $(pwd)/build" diff --git a/ci/scripts/cpp_test.sh b/ci/scripts/cpp_test.sh index a0b77b11be2e..729d2cea7878 100755 --- a/ci/scripts/cpp_test.sh +++ b/ci/scripts/cpp_test.sh @@ -144,8 +144,9 @@ if [ "${ARROW_USE_MESON:-OFF}" = "OFF" ] && \ CMAKE_PREFIX_PATH+="/lib/cmake/" ;; esac + # Search vcpkg before /lib/cmake. if [ -n "${VCPKG_ROOT}" ] && [ -n "${VCPKG_DEFAULT_TRIPLET}" ]; then - CMAKE_PREFIX_PATH+=";${VCPKG_ROOT}/installed/${VCPKG_DEFAULT_TRIPLET}" + CMAKE_PREFIX_PATH="${VCPKG_ROOT}/installed/${VCPKG_DEFAULT_TRIPLET};${CMAKE_PREFIX_PATH}" fi cmake \ -S "${source_dir}/examples/minimal_build" \ diff --git a/cpp/cmake_modules/SetupCxxFlags.cmake b/cpp/cmake_modules/SetupCxxFlags.cmake index 3c172aebdf8a..e0d1fa0dcca3 100644 --- a/cpp/cmake_modules/SetupCxxFlags.cmake +++ b/cpp/cmake_modules/SetupCxxFlags.cmake @@ -186,6 +186,7 @@ if(WIN32) # # ARROW-2986: Without /EHsc we get C4530 warning set(CXX_COMMON_FLAGS "/W3 /EHsc") + string(APPEND CMAKE_CXX_FLAGS " /EHsc") endif() # Disable C5105 (macro expansion producing 'defined' has undefined diff --git a/cpp/src/arrow/flight/sql/odbc/odbc_impl/CMakeLists.txt b/cpp/src/arrow/flight/sql/odbc/odbc_impl/CMakeLists.txt index ff716660a31b..22531d9638e0 100644 --- a/cpp/src/arrow/flight/sql/odbc/odbc_impl/CMakeLists.txt +++ b/cpp/src/arrow/flight/sql/odbc/odbc_impl/CMakeLists.txt @@ -129,13 +129,9 @@ if(WIN32) system_dsn.h) endif() -target_link_libraries(arrow_odbc_spi_impl PUBLIC arrow_flight_sql_shared - arrow_compute_shared Boost::locale) - -# Link libraries on MINGW64 and macOS -if(MINGW OR APPLE) - target_link_libraries(arrow_odbc_spi_impl PUBLIC ${ODBCINST}) -endif() +target_link_libraries(arrow_odbc_spi_impl + PUBLIC arrow_flight_sql_shared arrow_compute_shared Boost::locale + ${ODBCINST}) set_target_properties(arrow_odbc_spi_impl PROPERTIES ARCHIVE_OUTPUT_DIRECTORY diff --git a/cpp/src/arrow/flight/sql/odbc/odbc_impl/get_info_cache.cc b/cpp/src/arrow/flight/sql/odbc/odbc_impl/get_info_cache.cc index cd637be165eb..7d6239f24a7e 100644 --- a/cpp/src/arrow/flight/sql/odbc/odbc_impl/get_info_cache.cc +++ b/cpp/src/arrow/flight/sql/odbc/odbc_impl/get_info_cache.cc @@ -19,8 +19,6 @@ #include "arrow/flight/sql/odbc/odbc_impl/platform.h" -#include -#include #include "arrow/array.h" #include "arrow/array/array_nested.h" #include "arrow/flight/sql/api.h" @@ -32,6 +30,10 @@ #include "arrow/flight/sql/odbc/odbc_impl/scalar_function_reporter.h" #include "arrow/flight/sql/odbc/odbc_impl/util.h" +// Include ODBC headers after arrow headers to avoid conflicts with sql_info_undef.h +#include +#include + // Aliases for entries in SqlInfoOptions::SqlInfo that are defined here // due to causing compilation errors conflicting with ODBC definitions. #define ARROW_SQL_IDENTIFIER_CASE 503 diff --git a/cpp/src/arrow/flight/sql/odbc/odbc_impl/odbc_connection.cc b/cpp/src/arrow/flight/sql/odbc/odbc_impl/odbc_connection.cc index 755151fcc0bf..a2464aa00e36 100644 --- a/cpp/src/arrow/flight/sql/odbc/odbc_impl/odbc_connection.cc +++ b/cpp/src/arrow/flight/sql/odbc/odbc_impl/odbc_connection.cc @@ -30,6 +30,7 @@ #include "arrow/flight/sql/odbc/odbc_impl/spi/statement.h" #include "arrow/flight/sql/odbc/odbc_impl/util.h" +// Include ODBC headers after arrow headers to avoid conflicts with sql_info_undef.h #include #include #include diff --git a/cpp/src/arrow/flight/sql/odbc/tests/CMakeLists.txt b/cpp/src/arrow/flight/sql/odbc/tests/CMakeLists.txt index 1f487fc0cd1d..977975a7378c 100644 --- a/cpp/src/arrow/flight/sql/odbc/tests/CMakeLists.txt +++ b/cpp/src/arrow/flight/sql/odbc/tests/CMakeLists.txt @@ -52,3 +52,6 @@ add_arrow_test(flight_sql_odbc_test ${ODBCINST} ${SQLite3_LIBRARIES} arrow_odbc_spi_impl) + +# Disable unity build due to sqlite_sql_info.cc conflict with sql.h and sqlext.h headers. +set_target_properties(arrow-flight-sql-odbc-test PROPERTIES UNITY_BUILD OFF) diff --git a/cpp/src/arrow/flight/sql/odbc/tests/odbc_test_suite.h b/cpp/src/arrow/flight/sql/odbc/tests/odbc_test_suite.h index e043a459f0a1..823fc3d46141 100644 --- a/cpp/src/arrow/flight/sql/odbc/tests/odbc_test_suite.h +++ b/cpp/src/arrow/flight/sql/odbc/tests/odbc_test_suite.h @@ -15,6 +15,8 @@ // specific language governing permissions and limitations // under the License. +#pragma once + #include "arrow/testing/gtest_util.h" #include "arrow/util/io_util.h" #include "arrow/util/utf8.h" From 2f4cf1c954f1435a877ed11c04d817e8d327556e Mon Sep 17 00:00:00 2001 From: "Alina (Xi) Li" Date: Mon, 1 Dec 2025 11:33:35 -0800 Subject: [PATCH 2/2] Add ODBC installer MSVC CI --- .github/workflows/cpp_odbc.yml | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/.github/workflows/cpp_odbc.yml b/.github/workflows/cpp_odbc.yml index 0a97e6dfe1fc..b7d2afc683d6 100644 --- a/.github/workflows/cpp_odbc.yml +++ b/.github/workflows/cpp_odbc.yml @@ -56,6 +56,7 @@ jobs: ARROW_FLIGHT: ON ARROW_FLIGHT_SQL: ON ARROW_FLIGHT_SQL_ODBC: ON + ARROW_FLIGHT_SQL_ODBC_INSTALLER: ON ARROW_SIMD_LEVEL: AVX2 CMAKE_CXX_STANDARD: "17" CMAKE_GENERATOR: Ninja @@ -158,3 +159,22 @@ jobs: # for /f "usebackq delims=" %%I in (`bash -c "cygpath -u \"$VCPKG_ROOT_KEEP\""` ) do set VCPKG_ROOT=%%I # bash -c "ci/scripts/cpp_test.sh $(pwd) $(pwd)/build" + + - name: Install WiX Toolset + shell: pwsh + run: | + Invoke-WebRequest -Uri https://github.com/wixtoolset/wix/releases/download/v6.0.0/wix-cli-x64.msi -OutFile wix-cli-x64.msi + Start-Process -FilePath wix-cli-x64.msi -ArgumentList '/quiet', 'Include_freethreaded=1' -Wait + echo "C:\Program Files\WiX Toolset v6.0\bin\" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append + - name: Build MSI ODBC installer + shell: pwsh + run: | + # Verify WiX version + wix --version + cd "${{ github.workspace }}\build\cpp" + cpack + - name: Upload the artifacts to the job + uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0 + with: + name: flight-sql-odbc-msi-installer + path: ${{ github.workspace }}\build\cpp\Apache Arrow Flight SQL ODBC-1.0.0-win64.msi