Auto update workflow fix#51
Merged
Merged
Conversation
- Add GitHub Actions workflow for auto-updating pkl and pkl-go - Workflow runs daily and can be triggered manually - Auto-creates PRs with detailed changelogs - Includes test validation before PR creation - Add workflow documentation and README badge - Update OFFLINE.md with automation section Related to kdeps/.github/workflows/auto-update-dependencies.yml
- Add comprehensive testing checklist - Add quick start guide for workflow - Add local test script for workflow logic - Provides step-by-step validation procedures
- Add comprehensive testing checklist - Add quick start guide for workflow - Add local test script for workflow logic - Provides step-by-step validation procedures
…into auto-update-workflow
- Fix 'deps/pkl/external' not found error - download_deps.sh creates assets/pkl/external directly - Only need to copy PKL files from deps/pkl to assets/pkl - External dependencies already in correct location - Add better error messaging and validation
- Resolved conflicts in auto-update-dependencies.yml - Kept fixed version with proper error handling - Kept correct asset copying logic (no deps/pkl/external)
- Add exclusion logic to download_deps.sh - Exclude com.circleci.v2/configuration.pkl (conflicts with Config.pkl) - Fixes Go embed error on case-insensitive filesystems - Remove existing conflicting file from assets
- Change from EXCLUDE_FILES to RENAME_FILES approach - Rename configuration.pkl → CircleCI_configuration.pkl - Preserves both files with unique names - More maintainable than deletion
- Automatic detection of ALL case-insensitive filename conflicts - Works for pkl-go, pkl-pantry, and any future dependencies - Automatically renames conflicting files with parent dir prefix - Fixes all import/reference statements to renamed files - Cross-platform (macOS/Linux sed compatibility) - Bash 3.2+ compatible (macOS default bash) How it works: 1. detect_conflicts() - Scans all .pkl files, finds conflicts 2. apply_renames() - Renames conflicting files automatically 3. fix_references() - Updates all imports to use new names Example: Config.pkl vs configuration.pkl → Renames to: Config.pkl and com.circleci.v2_configuration.pkl → Updates all files that import configuration.pkl
Dramatically improve performance of fix_references() function: - Build sed script once instead of per-file execution - Pre-filter files using grep -l to only process files with matches - Single sed pass per file instead of nested loops - Reduce complexity from O(n×m) to O(n×grep + k×sed) This addresses the "updating references takes a lot of time" issue by avoiding unnecessary processing of files that don't reference renamed dependencies. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Resolved conflict in scripts/download_deps.sh by keeping the optimized fix_references() function with performance improvements (pre-filtering and batch sed operations) over the slower version from main.
Remove hardcoded error checks and counter arithmetic that could cause exit code 1 with set -e. Simplify to: build → filter → apply → cleanup. - Remove count variable and arithmetic expansion - Use xargs on Linux for better performance - Explicit return 0 for success - Let bash handle errors naturally 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Refactor conflict resolution to handle each conflict atomically: - Upon detection: rename file immediately - Fix references for that file immediately - Then move to next conflict Benefits: - Simpler flow (single function instead of 3) - Easier to debug (know exactly which file causes issues) - No intermediate state files needed - Cleaner code Removed: - detect_conflicts() - apply_renames() - fix_references() Added: - detect_and_resolve_conflicts() - handles everything atomically - fix_references_for_file() - fixes refs for single file 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
After all atomic conflict resolutions, run a final global sweep to catch any references that might have been missed. Flow: 1. Detect conflict → rename → fix references (atomic) 2. Track all renames in /tmp/all_renames.txt 3. After all conflicts: global check for remaining refs 4. Fix any missed references 5. Clean up This provides belt-and-suspenders safety - atomic fixes plus a final safety net. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Resolved conflict by keeping atomic conflict resolution approach (detect_and_resolve_conflicts with final global check) over the old three-function approach from main.
Change reference fixing to only replace the basename, not the entire path. This preserves relative import structures. Before: import "../foo/config.pkl" → import "pkl-pantry/.../com.circleci.v2_config.pkl" ❌ (breaks) After: import "../foo/config.pkl" → import "../foo/com.circleci.v2_config.pkl" ✅ (correct) Changes: - Only replace the filename part: /config.pkl → /new_name.pkl - Preserve directory structure from original imports - Apply to both atomic fixes and final global check 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Critical fixes: 1. Record NEW filename's lowercase, not old (was the bug!) 2. Check if generated name will conflict before renaming 3. Add numeric suffix (_2, _3, etc.) if new name conflicts Before: Config.pkl → com.circleci.v2_Config.pkl (lowercase: ...config.pkl) config.pkl → com.circleci.v2_config.pkl (lowercase: ...config.pkl) Result: STILL CONFLICTS! ❌ After: Config.pkl → com.circleci.v2_Config.pkl (records: ...config.pkl) config.pkl → com.circleci.v2_config_2.pkl (detects conflict, adds _2) Result: No conflicts ✅ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Resolved conflicts by keeping our version which includes: - Atomic conflict resolution - Basename-only reference fixing (preserves paths) - New name conflict checking with numeric suffixes - Correct lowercase tracking of renamed files
Error: 'local: can only be used in a function' The 'local' keyword at line 222 was in the main script body (inside a while loop), not inside a function. Removed 'local' to fix the bash error. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Resolved conflict by keeping our version without 'local' keyword (main had the buggy version with 'local' outside function).
Major improvements to dependency management: 1. Fixed download_deps.sh to only download files listed in versions.json - Was downloading ALL .pkl files (~600+ files) - Now downloads only specified files (~70 files) - Reduced unnecessary files by ~90% 2. Updated versions.json with correct filenames - Fixed CircleCI.pkl → Config.pkl - Fixed Prometheus.pkl → Configuration.pkl, PrometheusObject.pkl, Rule.pkl - Fixed Spark.pkl → Properties.pkl, PropertiesBase.pkl, utils.pkl - Fixed CSV.pkl → csv.pkl, Network.pkl → net.pkl, etc. 3. Optimized pkl-pantry download to use monorepo structure - pkl-pantry uses a monorepo where all packages share releases - Download once instead of once per package - Extract specific files from the correct package subdirectories Before: - Downloaded ~600+ files from pkl-pantry - Case-insensitive conflicts required mass renaming - Massive git diffs with hundreds of deleted/renamed files After: - Download exactly 70 specified files from pkl-pantry - No conflicts, clean directory structure - Each package contains only its required files 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
- Remove 500+ unnecessary pkl-pantry files that were downloaded previously - Keep only the 70 files specified in versions.json - Remove renamed files from conflict resolution (no longer needed) - Clean directory structure with each package containing only required files This completes the fix for the dependency download issue where all .pkl files were being downloaded instead of just the specified ones. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Changed from local file paths to package:// URLs for external dependencies. This is the correct way to reference pkl-pantry packages. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
…resolution Removed 183 lines of complexity that's no longer needed: Before: - Downloaded all files → conflicts on case-insensitive filesystems - Complex rename logic with RENAME_MAP - Reference fixing across all files - Bash 4.0 requirement for associative arrays - 276 lines total After: - Download only specified files → no conflicts - Each package in its own directory - Clean, straightforward logic - Works with bash 3.2+ - 93 lines total (66% reduction) Why this is safe: - We now download only 70 pkl-pantry files (not 600+) - Files are isolated by package directory - No filename collisions possible - Case-insensitive filesystems work fine 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Updated imports from package:// URLs to local file paths to ensure offline compatibility. Tests require all embedded assets to use local paths. Changes: - Item.pkl, Memory.pkl, Session.pkl, Tool.pkl - URI.pkl imports: package:// → external/pkl-pantry/packages/... All tests now pass ✅ 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
Now that we only download specified files (no conflicts), we can restore the original filenames without prefixes. Changes: - Renamed fixtures_ClassGen.pkl → ClassGen.pkl - Renamed tests_gatherer.pkl → gatherer.pkl - Renamed pkl-go_generator-settings.pkl → generator-settings.pkl - Renamed com.circleci.v2_Config.pkl → Config.pkl - Renamed org.apache.spark_utils.pkl → utils.pkl - Renamed internal_Type.pkl → Type.pkl - etc. All files now have their original, clean names since there are no case-insensitive conflicts. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.