Skip to content

Conversation

@smuzaffar
Copy link
Collaborator

@smuzaffar smuzaffar commented Dec 18, 2025

This PR proposes the following changes

  • Relocate all text files which have hard-coded build directory paths.
    • This change moves the relocation logic in a separate script which does not contain any hard coded paths and can be deployed directly in the package install directory.
    • etc/profile.d/.bits-relocate file int he package contains the list of all text files with hard-coded build path
    • etc/profile.d/.bits-pkginfo containe the package information e.g. it now contains the package build directory path which can be used to properly relocate all text files.
  • As build time path is stored in etc/profile.d/.bits-pkginfo so there is no need to keep the extra *.unrelocated copies of files to be relocated

I have tested this using dummy packages by building/uploading them from different build directory and it has properly relocated the text files [a-d]. I will run more tests e.g. building few externals with long dependency chain

[a] Building/uploading pkg1 (hich depemnds on pkg0) in PKG1 directory

> bits  build  -c cms.bits --remote-store b3://cmsrep::rw -w PKG1 --builder 1 dummy-pkg1
==> Configured directory:
    /build/muz/bits-works/cms.bits

==> Package Recipe will be searched in the following order 
    
==> Packages will be built in the following order:
     - dummy-pkg0@1.0
     - dummy-pkg1@1.0
==> Compiling defaults-release@v1 (use --debug for full output): done
==> Compiling dummy-pkg0@1.0 (use --debug for full output): done
==> Compiling dummy-pkg1@1.0 (use --debug for full output): done

==> Build of dummy-pkg1 successfully completed on `cmsdev40.cern.ch'.
    Your software installation is at:    
      /build/muz/bits-works/PKG1/slc9_x86-64
    You can use this package by loading the environment:
      bits enter dummy-pkg1/latest-release

[b] Building/uploading pkg2 (hich depends on pkg1, so downloaded the prebuild binaries) in PKG2 directory

(venv) Singularity> bits  build  -c cms.bits --remote-store b3://cmsrep::rw -w PKG2 --builder 1 dummy-pkg2
==> Configured directory:
    /build/muz/bits-works/cms.bits

==> Package Recipe will be searched in the following order 
    
==> Packages will be built in the following order:
     - dummy-pkg0@1.0
     - dummy-pkg1@1.0
     - dummy-pkg2@1.0
==> Downloading tarball for defaults-release@v1: done  
==> Unpacking defaults-release@v1: done
==> Downloading tarball for dummy-pkg0@1.0: done  
==> Unpacking dummy-pkg0@1.0: done
==> Downloading tarball for dummy-pkg1@1.0: done  
==> Unpacking dummy-pkg1@1.0: done
==> Compiling dummy-pkg2@1.0 (use --debug for full output): done

==> Build of dummy-pkg2 successfully completed on `cmsdev40.cern.ch'.
    Your software installation is at:
      /build/muz/bits-works/PKG2/slc9_x86-64
    You can use this package by loading the environment:
      bits enter dummy-pkg2/latest-release

[c] Building/uploading pkg3 (hich depends on pkg2, so downloaded the prebuild binaries) in PKG3 directory

(venv) Singularity> bits  build  -c cms.bits --remote-store b3://cmsrep::rw -w PKG3 --builder 1 dummy-pkg3
==> Configured directory:
    /build/muz/bits-works/cms.bits

==> Package Recipe will be searched in the following order 
   
==> Packages will be built in the following order:
     - dummy-pkg0@1.0
     - dummy-pkg1@1.0
     - dummy-pkg2@1.0
     - dummy-pkg3@1.0
==> Downloading tarball for defaults-release@v1: done  
==> Unpacking defaults-release@v1: done
==> Downloading tarball for dummy-pkg0@1.0: done  
==> Unpacking dummy-pkg0@1.0: done
==> Downloading tarball for dummy-pkg1@1.0: done  
==> Unpacking dummy-pkg1@1.0: done
==> Downloading tarball for dummy-pkg2@1.0: done  
==> Unpacking dummy-pkg2@1.0: done
==> Compiling dummy-pkg3@1.0 (use --debug for full output): done

==> Build of dummy-pkg3 successfully completed on `cmsdev40.cern.ch'.
    Your software installation is at: 
      /build/muz/bits-works/PKG3/slc9_x86-64
    You can use this package by loading the environment:
      bits enter dummy-pkg3/latest-release

[d] bin/pkg-config in each package contains paths to its dependencies which were properly relocated

> cat PKG3/slc9_x86-64/dummy-pkg3/1.0-1/bin/pkg-config 
#!/bin/bash
THIS_PKG_BIN=/build/muz/bits-works/PKG3/slc9_x86-64/dummy-pkg3/1.0-1/bin
DUMMY_PKG0_ROOT=/build/muz/bits-works/PKG3/slc9_x86-64/dummy-pkg0/1.0-1
DUMMY_PKG1_ROOT=/build/muz/bits-works/PKG3/slc9_x86-64/dummy-pkg1/1.0-1
DUMMY_PKG2_ROOT=/build/muz/bits-works/PKG3/slc9_x86-64/dummy-pkg2/1.0-1
DEFAULTS_RELEASE_ROOT=/build/muz/bits-works/PKG3/slc9_x86-64/defaults-release/v1-1
ALL_PKGS=/build/muz/bits-works/PKG3/slc9_x86-64/defaults-release/v1-1:/build/muz/bits-works/PKG3/slc9_x86-64/dummy-pkg2/1.0-1:/build/muz/bits-works/PKG3/slc9_x86-64/dummy-pkg1/1.0-1:/build/muz/bits-works/PKG3/slc9_x86-64/dummy-pkg0/1.0-1:
> cat PKG3/slc9_x86-64/dummy-pkg1/1.0-1/bin/pkg-config 
#!/bin/bash
THIS_PKG_BIN=/build/muz/bits-works/PKG3/slc9_x86-64/dummy-pkg1/1.0-1/bin
DUMMY_PKG0_ROOT=/build/muz/bits-works/PKG3/slc9_x86-64/dummy-pkg0/1.0-1
DEFAULTS_RELEASE_ROOT=/build/muz/bits-works/PKG3/slc9_x86-64/defaults-release/v1-1
ALL_PKGS=/build/muz/bits-works/PKG3/slc9_x86-64/defaults-release/v1-1:/build/muz/bits-works/PKG3/slc9_x86-64/dummy-pkg0/1.0-1:

cd "$WORK_DIR/INSTALLROOT/$PKGHASH/$PKGPATH"
# Find which files need relocation.
{ grep -I -H -l -R "\(INSTALLROOT/$PKGHASH\|[@][@]PKGREVISION[@]$PKGHASH[@][@]\)" . || true; } | sed -e 's|^\./||' > "$INSTALLROOT/.original-unrelocated"
{ grep -I -H -l -R "\($WORK_DIR\|[@][@]PKGREVISION[@]$PKGHASH[@][@]\)" . || true; } | sed -e 's|^\./||' > "$INSTALLROOT/etc/profile.d/.bits-relocate"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why $WORK_DIR and not INSTALLROOT/$PKGHASH?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

INSTALLROOT/$PKGHASH is for the package itself but the whole idea of this change was to find all text files which has $WORKDIR (note that this will also match files with INSTALLROOT/$PKGHASH as it is under $WORKDIR directory ... right?) in them. If we search for INSTALLROOT/$PKGHASH then we might miss those text files which have hard coded paths of package dependencies ( e.g. root-config)

echo "sed -e \"s|/[^ ;:]*INSTALLROOT/\$PH/\$OP|\$WORK_DIR/\$PP|g; s|[@][@]PKGREVISION[@]\$PH[@][@]|$PKGREVISION|g\"" \
"\$PP/$unrelocated.unrelocated > \$PP/$unrelocated"
done < "$INSTALLROOT/.original-unrelocated" >> "$INSTALLROOT/relocate-me.sh"
cp "${BITS_SCRIPT_DIR}/relocate-me.sh" "$INSTALLROOT/"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should use install, to make sure the copied file has the right mode?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done in #75

INSTALL_BASE=$(echo $THISDIR | sed "s|/$PP$||")
if [[ -s ${THISDIR}/etc/profile.d/.bits-relocate ]] ; then
for f in $(cat ${THISDIR}/etc/profile.d/.bits-relocate) ; do
sed -i -e "s|${PKG_DIR}/INSTALLROOT/$PH|$INSTALL_BASE|g;s|${PKG_DIR}|$INSTALL_BASE|g" "${THISDIR}/$f"
Copy link
Collaborator

@ktf ktf Dec 22, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

-i should take an argument (the extension of the unmodified file) and the backup file should be deleted afterwards.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done in #75

@ktf
Copy link
Collaborator

ktf commented Dec 22, 2025

Merging once the last test rerun completes. Please consider the comments and see if anything applies for some further PR.

@ktf ktf merged commit 4b645fc into main Dec 22, 2025
42 of 48 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants