Greylilac09/fix mem tracking

.dockerignore

@@ -10,3 +10,6 @@
 !rust-toolchain.toml
 !scripts
 !vdev
+
+# Exclude target directories to avoid copying build artifacts
+target

.github/ISSUE_TEMPLATE/minor-release.md

@@ -11,10 +11,11 @@ labels: "domain: releasing"
 Note the preparation steps are now automated. First, alter/create release.env
 
 ```shell
+#!/usr/bin/env bash
 export NEW_VECTOR_VERSION=<new Vector version> # replace this with the actual new version (e.g.: 0.50.0)
 export NEW_VRL_VERSION=<new VRL version> # replace this with the actual new VRL version (e.g.: 0.30.0)
 export MINOR_VERSION=$(echo "$NEW_VECTOR_VERSION" | cut -d. -f2)
-export PREP_BRANCH=prepare-v-0-"${MINOR_VERSION}"-"${NEW_VECTOR_VERSION}"-website
+export PREP_BRANCH=prepare-v-"${NEW_VECTOR_VERSION//./-}"-website
 export RELEASE_BRANCH=v0."${MINOR_VERSION}"
 ```
 
@@ -24,7 +25,7 @@ and then source it by running `source ./release.env`
 
 ## 1. Manual Steps
 
-- [ ] Cut a new release of [VRL](https://github.com/vectordotdev/vrl) if needed
+- [ ] Cut a new release of [VRL](https://github.com/vectordotdev/vrl) if needed.
   - VRL release steps: https://github.com/vectordotdev/vrl/blob/main/release/README.md
 
 ## 2. Automated Steps
@@ -37,12 +38,11 @@ cargo vdev release prepare --version "${NEW_VECTOR_VERSION}" --vrl-version "${NE
 
 Automated steps include:
 - [ ] Create a new release branch from master to freeze commits
-  - `git fetch && git checkout origin/master && git checkout -b "{RELEASE_BRANCH}" && git push -u`
+  - `git fetch && git checkout origin/master && git checkout -b "${RELEASE_BRANCH}" && git push -u`
 - [ ] Create a new release preparation branch from `master`
   - `git checkout -b "${PREP_BRANCH}" && git push -u`
 - [ ] Pin VRL to latest released version rather than `main`
-- [ ] Check if there is a newer version of [Alpine](https://alpinelinux.org/releases/) or
-      [Debian](https://www.debian.org/releases/) available to update the release images in
+- [ ] Check if there is a newer version of [Alpine](https://alpinelinux.org/releases/) or [Debian](https://www.debian.org/releases/) available to update the release images in
       `distribution/docker/`. Update if so.
 - [ ] Run `cargo vdev build release-cue` to generate a new cue file for the release
   - [ ] Copy VRL changelogs from the VRL version in the last Vector release as a new changelog entry
@@ -60,47 +60,44 @@ Automated steps include:
 - [ ] Edit `website/cue/reference/releases/"${NEW_VECTOR_VERSION}".cue`
   - [ ] Add description key to the generated cue file with a description of the release (see
         previous releases for examples).
-  - [ ] Ensure any breaking changes are highlighted in the release upgrade guide
-  - [ ] Ensure any deprecations are highlighted in the release upgrade guide
-  - [ ] Review generated changelog entries to ensure they are understandable to end-users
+  - [ ] Ensure any breaking changes are highlighted in the release upgrade guide.
+  - [ ] Ensure any deprecations are highlighted in the release upgrade guide.
+  - [ ] Review generated changelog entries to ensure they are understandable to end-users.
   - [ ] Ensure the date matches the scheduled release date.
   - [ ] Add a link to pending deprecation items from [DEPRECATIONS.md](https://github.com/vectordotdev/vector/blob/master/docs/DEPRECATIONS.md).
-- [ ] PR review & approval
+- [ ] PR review & approval.
 
 # On the day of release
 
-- [ ] Make sure the release branch is in sync with origin/master and has only one squashed commit with all commits from the prepare branch. If you made a PR from the prepare branch into the release branch this should already be the case
+- [ ] Make sure the release branch is in sync with origin/master and has only one squashed commit with all commits from the prepare branch. If you made a PR from the prepare branch into the release branch this should already be the case.
   - [ ] `git checkout "${RELEASE_BRANCH}"`
-  - [ ] `git show --stat HEAD` - This should show the squashed prepare commit
+  - [ ] `git show --stat HEAD` - This should show the squashed prepare commit.
   - [ ] Ensure release date in `website/cue/reference/releases/0.XX.Y.cue` matches current date.
-    - If this needs to be updated commit and squash it in the release branch
+    - If this needs to be updated commit and squash it in the release branch.
   - Follow these steps if the release branch needs to be updated
-    - [ ] Rebase the release preparation branch on the release branch
+    - [ ] Rebase the release preparation branch on the release branch.
       - [ ] Squash the release preparation commits (but not the cherry-picked commits!) to a single
           commit. This makes it easier to cherry-pick to master after the release.
-    - [ ] Merge release preparation branch into the release branch
+    - [ ] Merge release preparation branch into the release branch.
         - `git switch "${RELEASE_BRANCH}" && git merge --ff-only "${PREP_BRANCH}"`
 
 - [ ] Tag new release
   - [ ] `git tag v"${NEW_VECTOR_VERSION}" -a -m v"${NEW_VECTOR_VERSION}"`
   - [ ] `git push origin v"${NEW_VECTOR_VERSION}"`
-- [ ] Wait for release workflow to complete
+- [ ] Wait for release workflow to complete.
   - Discoverable via [release.yml](https://github.com/vectordotdev/vector/actions/workflows/release.yml)
 - [ ] Reset the `website` branch to the `HEAD` of the release branch to update https://vector.dev
   - [ ] `git switch website && git reset --hard origin/"${RELEASE_BRANCH}" && git push`
   - [ ] Confirm that the release changelog was published to https://vector.dev/releases/
-    - The deployment is done by Amplify. You can see
-      the [deployment logs here](https://dd-corpsite.datadoghq.com/logs?query=service%3Awebsites-vector%20branch%3Awebsite&agg_m=count&agg_m_source=base&agg_t=count&cols=host%2Cservice&fromUser=true&messageDisplay=inline&refresh_mode=sliding&storage=hot&stream_sort=time%2Casc&viz=stream).
-- [ ] Release Linux packages. See [`vector-release` usage](https://github.com/DataDog/vector-release#usage).
-  - Note: the pipeline inputs are the version number `v"${NEW_VECTOR_VERSION}"` and a personal GitHub token.
-  - [ ] Manually trigger the `trigger-package-release-pipeline-prod-stable` job.
-- [ ] Release updated Helm chart. See [releasing Helm chart](https://github.com/vectordotdev/helm-charts#releasing).
-- [ ] Once Helm chart is released, updated Vector manifests
-    - Run `cargo vdev build manifests` and open a PR with changes
-- [ ] Add docker images to [https://github.com/DataDog/images](https://github.com/DataDog/images/tree/master/vector) to have them available internally. ([Example PR](https://github.com/DataDog/images/pull/7104))
+    - Refer to the internal releasing doc to monitor the deployment.
+- [ ] Release Linux packages. Refer to the internal releasing doc.
+- [ ] Release updated Helm chart. See [releasing Helm chart](https://github.com/vectordotdev/helm-charts/blob/develop/RELEASING.md).
+- [ ] Release Homebrew. Refer to the internal releasing doc.
+- [ ] Create internal Docker images. Refer to the internal releasing doc.
+- [ ] Update the latest [release tag](https://github.com/vectordotdev/vector/release) description with the release announcement.
 - [ ] Create a new PR with title starting as `chore(releasing):`
-  - [ ] Cherry-pick any release commits from the release branch that are not on `master`, to `master`
+  - [ ] Cherry-pick any release commits from the release branch that are not on `master`, to `master`.
+  - [ ] Run `cargo vdev build manifests` and commit changes.
   - [ ] Bump the release number in the `Cargo.toml` on master to the next minor release.
-  - [ ] Also, update `Cargo.lock` with: `cargo update -p vector`
+  - [ ] Also, update `Cargo.lock` with: `cargo update -p vector`.
   - [ ] If there is a VRL version update, revert it and make it track the git `main` branch and then run `cargo update -p vrl`.
-- [ ] Kick-off post-mortems for any regressions resolved by the release

.github/actions/install-vdev/action.yml

@@ -5,12 +5,19 @@ branding:
   icon: tool
   color: purple
 
+inputs:
+  skip-cache:
+    description: "Skip cache lookup and force compilation"
+    required: false
+    default: 'false'
+
 runs:
   using: "composite"
   steps:
     - name: Cache vdev binary
       id: cache-vdev
-      uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+      if: ${{ inputs.skip-cache != 'true' }}
+      uses: actions/cache@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
       with:
         path: ~/.cargo/bin/vdev
         key: ${{ runner.os }}-vdev-${{ hashFiles('vdev/**', 'Cargo.toml', 'Cargo.lock') }}
@@ -23,3 +30,15 @@ runs:
       run: |
         echo "Building vdev from source (cache miss)"
         cargo install --path vdev --locked --force
+
+    - name: Save vdev to cache
+      if: ${{ inputs.skip-cache == 'true' }}
+      uses: actions/cache/save@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
+      with:
+        path: ~/.cargo/bin/vdev
+        key: ${{ runner.os }}-vdev-${{ hashFiles('vdev/**', 'Cargo.toml', 'Cargo.lock') }}
+
+    - name: Set VDEV environment variable
+      shell: bash
+      run: |
+        echo "VDEV=$(which vdev)" >> "$GITHUB_ENV"

.github/actions/setup/action.yml

@@ -9,7 +9,7 @@ inputs:
   cargo-cache:
     required: false
     default: true
-    description: "Cache Cargo registry, index, git DB, and ~/.cargo/bin. Enabled automatically if any Rust tools are requested."
+    description: "Cache Cargo registry, index, and git DB. Enabled automatically if any Rust tools are requested."
 
   mold:
     required: false
@@ -24,6 +24,10 @@ inputs:
     required: false
     default: false
     description: "Install cue."
+  libsasl2:
+    required: false
+    default: false
+    description: "Install libsasl2." # Required to fully build Vector
 
   # prepare.sh - rust
   rust: # rustup module
@@ -60,7 +64,7 @@ inputs:
     description: "Install wasm-pack for Rust to WASM bridge."
   vdev:
     required: false
-    default: false
+    default: true
     description: "Install vdev CLI tool (cached by vdev/ directory changes)."
 
   # prepare.sh - npm
@@ -76,13 +80,24 @@ inputs:
 runs:
   using: "composite"
   steps:
+    - name: Check vdev cache status
+      id: check-vdev-cache
+      if: ${{ inputs.vdev == 'true' }}
+      uses: actions/cache@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
+      with:
+        path: ~/.cargo/bin/vdev
+        key: ${{ runner.os }}-vdev-${{ hashFiles('vdev/**', 'Cargo.toml', 'Cargo.lock') }}
+        restore-keys: |
+          ${{ runner.os }}-vdev-
+        lookup-only: true
+
     - id: needs
       name: Compute if rust is needed
       shell: bash
       run: |
         rust="false"
 
-        # If any rust-related inputs are true, enable cache regardless
+        # If any rust-related inputs are true (excluding vdev), enable rust
         for val in "${{ inputs.mold }}" \
                    "${{ inputs.rust }}" \
                    "${{ inputs.cargo-deb }}" \
@@ -91,15 +106,22 @@ runs:
                    "${{ inputs.cargo-deny }}" \
                    "${{ inputs.cargo-msrv }}" \
                    "${{ inputs.dd-rust-license-tool }}" \
-                   "${{ inputs.wasm-pack }}" \
-                   "${{ inputs.vdev }}"; do
+                   "${{ inputs.wasm-pack }}"; do
           if [[ "$val" == "true" ]]; then
             rust="true"
             break
           fi
         done
 
+        # If vdev needs compilation (cache miss), enable rust
+        vdev_needs_compile="false"
+        if [[ "${{ inputs.vdev }}" == "true" && "${{ steps.check-vdev-cache.outputs.cache-hit }}" != "true" ]]; then
+          rust="true"
+          vdev_needs_compile="true"
+        fi
+
         echo "NEEDS_RUST=$rust" >> "$GITHUB_ENV"
+        echo "VDEV_NEEDS_COMPILE=$vdev_needs_compile" >> "$GITHUB_ENV"
 
         if [[ "$rust" == "true" ]]; then
           echo "RUST_BACKTRACE=full" >> "$GITHUB_ENV"
@@ -124,12 +146,11 @@ runs:
         EOF
       shell: bash
 
-    - name: Cache Cargo registry + index
-      if: ${{ inputs.cargo-cache == 'true' || env.NEEDS_RUST == 'true' }}
-      uses: actions/cache@5a3ec84eff668545956fd18022155c47e93e2684 # v4.2.3
+    - name: Cache Cargo registry, index, and git DB
+      if: ${{ inputs.cargo-cache == 'true' || env.NEEDS_RUST == 'true' || env.VDEV_NEEDS_COMPILE == 'true' }}
+      uses: actions/cache@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
       with:
         path: |
-          ~/.cargo/bin/
           ~/.cargo/registry/index/
           ~/.cargo/registry/cache/
           ~/.cargo/git/db/
@@ -138,7 +159,7 @@ runs:
           ${{ runner.os }}-cargo-
 
     - name: Install mold
-      if: ${{ inputs.mold == 'true' }}
+      if: ${{ runner.os == 'Linux' && env.DISABLE_MOLD != 'true' && (inputs.mold == 'true' || env.VDEV_NEEDS_COMPILE == 'true') }}
       shell: bash
       run: |
         echo "Installing mold"
@@ -189,7 +210,11 @@ runs:
       shell: bash
       run: |
         echo "Installing protoc"
-        sudo bash ./scripts/environment/install-protoc.sh
+        if [[ "${{ runner.os }}" == "macOS" ]]; then
+          sudo bash ./scripts/environment/install-protoc.sh /usr/local/bin
+        else
+          sudo bash ./scripts/environment/install-protoc.sh
+        fi
 
     - name: Install cue
       if: ${{ inputs.cue == 'true' }}
@@ -206,9 +231,34 @@ runs:
         sudo cp "${TEMP}/cue" /usr/bin/cue
         rm -rf "$TEMP"
 
+    - name: Install libsasl2
+      if: ${{ inputs.libsasl2 == 'true' }}
+      shell: bash
+      run: sudo apt-get update && sudo apt-get install -y libsasl2-dev
+
     - name: Install vdev
       if: ${{ inputs.vdev == 'true' }}
       uses: ./.github/actions/install-vdev
+      with:
+        skip-cache: ${{ env.VDEV_NEEDS_COMPILE == 'true' }}
+
+    - name: Cache prepare.sh binaries
+      id: cache-prepare-binaries
+      uses: actions/cache@cdf6c1fa76f9f475f3d7449005a359c84ca0f306 # v5.0.3
+      with:
+        path: |
+          ~/.cargo/bin/cargo-deb
+          ~/.cargo/bin/cross
+          ~/.cargo/bin/cargo-nextest
+          ~/.cargo/bin/cargo-deny
+          ~/.cargo/bin/cargo-msrv
+          ~/.cargo/bin/dd-rust-license-tool
+          ~/.cargo/bin/wasm-pack
+          /usr/local/bin/markdownlint
+          /usr/local/bin/datadog-ci
+        key: ${{ runner.os }}-prepare-binaries-${{ hashFiles('scripts/environment/*') }}
+        restore-keys: |
+          ${{ runner.os }}-prepare-binaries-
 
     - name: Run prepare.sh
       shell: bash
@@ -226,4 +276,6 @@ runs:
         [[ "${{ inputs.datadog-ci }}" == "true" ]] && mods+=("datadog-ci")
         csm=$(IFS=,; echo "${mods[*]}")
 
-        [[ "$csm" != "" ]] && ./scripts/environment/prepare.sh --modules="${csm}"
+        if [[ "$csm" != "" ]]; then
+          ./scripts/environment/prepare.sh --modules="${csm}"
+        fi

.github/actions/spelling/allow.txt

@@ -355,6 +355,7 @@ netcat
 netdata
 Netflix
 netlify
+netlink
 Neue
 neuronull
 Nextbook

.github/actions/spelling/expect.txt

@@ -430,6 +430,7 @@ pgo
 PIDs
 PII
 plainify
+plds
 ple
 podspec
 Ponge