Overhaul gaiaCore to Postgres-Centric#23
Conversation
|
Just some notes/observations from here! First, tried out using your branch here to get things working but got these errors initially: I found you can fix this by disabling postgres if it is running locally on your machine outside of Docker. Second, quick_ingest_datasource is not doing anything on my computer at the moment. It just hangs indefinitely... Third, I cannot get the Julia client to work. Whether in example or in the pipeline, I get errors in the try-catch block. |
|
Hey @jshoughtaling , some more tinkering and results from experimentation on my side:
It seems like there is still a docker version mismatch with
|
|
Hey @TheCedarPrince, First, thank you so much for the time you've spent going through and testing this out. I will try and push some updates as soon as I can to resolve the issues you're having. Meanwhile, I've had a few nice conversations with @tibbben about where certain components of this structure should live, and it could be that I partition this pull request out into multiple pieces that affect gaiaDB and gaiaCore. We will discuss at the meeting tomorrow in detail. Thanks again! |
|
@jshoughtaling wow, great work! I owe you a review but will not have time to do anything soon (like this week), but may have a chance to look with a little more care sometime next week and then meet the week after (week of the 17th). If that sounds ok, I will dig in next week. I am optimistic that we can bring together our approaches, modify the architectural approach a little to give things good homes, and then move forward ... I look forward to it. |
|
Thanks @TheCedarPrince and @tibbben for the feedback! What Was MigratedTo gaiaDB RepositorySQL Files (sql/ directory)All SQL initialization and function files have been migrated to
Enhanced init.sqlThe
Enhanced DockerfileThe
To gaiaDocker RepositoryPostgREST ServiceAdded a new
Environment ConfigurationUpdated
Database Health CheckEnhanced What Remained in gaiaCoreConnector Examples (connectors/ directory)The language-specific connector implementations remain in gaiaCore:
Rationale: These are demonstration/reference implementations showing how to interact with the PostgREST API. They are not core infrastructure and belong with the gaiaCore repository as examples. Documentation
Test Data
Updated Init Script
Key Schema Changes in gaiaDBBreaking Changes
How to Use the New Code1. Build and Run gaiaDB Standalonecd gaiaDB
docker build -t gaia-db .
docker run -d -p 5432:5432 -e POSTGRES_PASSWORD=secret gaia-db2. Run Full GAIA Stack with gaiaDockercd gaiaDocker
docker compose --profile gaia up -dThis will start:
3. Access PostgREST API# View API documentation
curl http://localhost:3000/
# Query data sources
curl http://localhost:3000/data_source
# Call a function
curl -X POST http://localhost:3000/rpc/quick_ingest_datasource \
-H "Content-Type: application/json" \
-d '{"dataset_name": "My Dataset"}'4. Load Test Data (Client-Side)# From gaiaCore repository root
cd gaiaCore
./scripts/init_gaiacore.sh
# Or with custom database connection
DB_HOST=localhost DB_PORT=5432 ./scripts/init_gaiacore.sh5. Use Connector Examplescd gaiaCore/connectors/python
python pipeline_example.py http://localhost:3000Documentation Updates NeededgaiaDB Repository
gaiaDocker Repository
gaiaCore Repository
|
3 participants
@tibbben & @rtmill - this Pull Request does the following:
Key next steps: