fix(import,update): stop writing .dvc metadata that disagrees with dvc (0.24.0)

docs/doctor.md

@@ -27,6 +27,7 @@ dt doctor -v        # Verbose output, includes dvc doctor
 | Archived remotes | No configured DVC remote has been archived (carries an `ARCHIVED.yaml` signpost) |
 | Git repo | The current directory is inside a git repository |
 | DVC repo | The current directory is inside a DVC repository |
+| dvcignore | `.dvcignore` excludes `.gitignore`, so this repo's generated ignore files cannot be hashed into the payload of anyone importing a directory from it ([why](init.md#why-dvcignore-starts-with-gitignore)) |
 
 ### Verbose-only checks
 
@@ -55,8 +56,9 @@ DVC Tools version: 0.1.0
 ✓ No archived remotes detected
 ✓ In git repository (/scratch/a56/me/my-project)
 ✓ In DVC repository (/scratch/a56/me/my-project)
+✓ .dvcignore excludes .gitignore files
 
-All 10 checks passed.
+All 11 checks passed.
 ```
 
 With issues:

docs/import.md

@@ -20,6 +20,39 @@ Imports files or directories from another DVC repository by:
 
 `dt import` first tries a local-cache workflow that does not require network access to remote object storage. If the source repository has no locally-accessible remote/cache, it falls back to `dvc import`.
 
+## The path must be a single DVC out
+
+`dt import` builds its manifest from object hashes, so every file under the
+imported path needs one. That holds when the path is itself a DVC out (or lives
+inside one). It does not hold for a plain directory that merely *contains* outs:
+DVC's repo filesystem presents git-tracked files there as part of the same tree,
+and `dvc import` hashes them from the git worktree into the payload — including
+the `.gitignore` files DVC generated in that directory.
+
+Given such a path, `dt import` refuses and names the offending files rather than
+writing a manifest that disagrees with `dvc import` in hash, `nfiles` and
+`size`. The fixes, in order of preference:
+
+1. Import the out itself, e.g. `data/annotations/in_house` rather than
+   `data/annotations`.
+2. If the only stray files are DVC's own generated ignore files, add
+   `.gitignore` to the **source** repo's `.dvcignore` and commit
+   (`dt init` seeds this; `dt doctor` flags repos that lack it). The payload
+   then contains only real data and `dt import` matches `dvc import` exactly.
+3. Make the path a single out upstream (`dvc add <path>`).
+4. Use plain `dvc import`, which hashes the git-tracked files too — at the cost
+   of a payload that includes them, and that your own remote cannot serve
+   (DVC excludes repo-import stages from `dvc push`).
+
+## Recorded size
+
+`size` comes from stat'ing the cache objects, in both the `files/md5/<xx>/` v3
+layout and the legacy `<xx>/` v2 one — a remote part-way through migration holds
+some objects in each. If any object cannot be found, `dt import` records **no**
+`size` and says so on stderr, rather than counting the miss as zero: a `size: 0`
+on a multi-GB import reads as an empty payload and silently undercounts every
+storage estimate downstream.
+
 ## Options
 
 - `--out, -o <path>`: Destination path for imported files (default: basename of source path)

docs/init.md

@@ -15,10 +15,24 @@ my-project/
 │   ├── config            # DVC settings (tracked)
 │   ├── config.local      # Local DVC settings (not tracked)
 │   └── ...
-├── .dvcignore            # DVC ignore patterns
+├── .dvcignore            # DVC ignore patterns, seeded with `.gitignore`
 └── .gitignore            # Updated with DVC patterns
 ```
 
+### Why `.dvcignore` starts with `.gitignore`
+
+DVC writes a `.gitignore` next to everything it tracks. Its repo filesystem
+excludes only `*.dvc`, `dvc.yaml`, `dvc.lock` and `.dvcignore` when another repo
+reads through it — so `dvc import <this repo> <some/dir>`, where `some/dir` is
+not itself an out, hashes those generated ignore files and folds them into the
+importer's payload, `nfiles` and `size`. One pattern stops this repo's
+bookkeeping from becoming someone else's data.
+
+DVC still creates and maintains those `.gitignore` files; it just stops treating
+them as content. `dt doctor` flags repos that predate this, and adding the
+pattern later is safe — but note it changes the hash, `nfiles` and `size` of any
+directory import of a path that contains one.
+
 ## Usage
 
 ```bash

docs/update.md

@@ -165,6 +165,21 @@ dt update imported/dir.dvc
 `dt update` does not check files out into the workspace — run `dvc checkout`
 afterwards if you need the workspace copies linked.
 
+## Imports of a path that is not a single out
+
+Step 5 needs a hash for every file under the path. A plain directory that merely
+*contains* outs also contains git-tracked files, which the source repo records
+no hash for — including the `.gitignore` files DVC generated there. `dvc import`
+hashes those from the git worktree and counts them; `dt update` cannot, so its
+rebuilt `.dir` would differ from DVC's in hash, `nfiles` and `size`, and step 7
+would publish that difference into a *shared* upstream remote as an object no
+DVC operation would ever produce.
+
+`dt update` therefore refuses such a target, names the files, and pushes
+nothing. See [dt import](import.md#the-path-must-be-a-single-dvc-out) for the
+fixes — most often adding `.gitignore` to the source repo's `.dvcignore`, after
+which dt and dvc agree exactly.
+
 ## Metadata population
 
 `dt update` produces first-class `.dvc` files with complete metadata:
@@ -188,7 +203,36 @@ outs:
   size: 52428800      # File size (50 MiB)
 ```
 
-This enables `dt du` to report accurate sizes and file counts. Note that size information is only available if the source repository's `.dvc` files also contain size metadata.
+This enables `dt du` to report accurate sizes and file counts.
+
+### Where `size` comes from
+
+`dvc list --size` against a repository *URL* usually reports no sizes at all: a
+`.dir` manifest records only an md5 and a relpath per file, so the only place a
+size actually exists is the object itself. When the listing is silent,
+`dt update` sizes each object by stat'ing it in the source repository's remote
+(if that remote is on this filesystem) or in the local cache — both the
+`files/md5/<xx>/` v3 layout and the legacy `<xx>/` v2 one.
+
+If even one object cannot be sized, `dt update` writes **no** `size` field
+rather than a partial total, and removes any `size` left over from the previous
+hash. An absent size means "not known"; a stale or partial one is a false
+statement that `dt du` and every downstream storage estimate will believe.
+Re-run `dt update` once the objects are locally readable, or `dvc update` to let
+DVC record it.
+
+### `rev` and `rev_lock`
+
+`deps.repo.rev` is the revision *spec* you track (a branch, tag, or pinned
+commit) and `rev_lock` is its resolution. `dt update` keeps the two consistent:
+
+- `--rev <branch|tag|sha>` records the spec in `rev` and the resolved 40-char
+  commit in `rev_lock`.
+- Without `--rev`, an import that tracks a branch advances `rev_lock` to the tip
+  of *that branch*, not to whatever the clone has checked out.
+- A `rev` pinned to a commit that the new `rev_lock` contradicts is removed,
+  with a message. Left in place, the next plain `dvc update` would resolve `rev`
+  and roll the import back to the commit you just updated away from.
 
 ## Import detection
 

dt/__init__.py

@@ -1,3 +1,3 @@
 """DVC Tools - Convenient tools for working with DVC in HPC environments."""
 
-__version__ = "0.23.0"
+__version__ = "0.24.0"

dt/cache_ops.py

@@ -300,6 +300,40 @@ def find_source_file(
     return None
 
 
+def object_size(md5: str, *cache_roots: Optional[str]) -> Optional[int]:
+    """Size of a cache object, searched across roots and both layouts.
+
+    The object *is* the file content, so its own size is authoritative -- no
+    need to ask the source repo. Uses :func:`find_source_file` so a remote in
+    the legacy v2 layout (or a half-migrated one holding some objects at
+    ``<xx>/<rest>`` and others under ``files/md5/``) answers as readily as a v3
+    one. That fallback is the whole point: sizing only the v3 path against a
+    mixed remote misses silently, and a miss that is then treated as zero
+    understates a multi-GB import (issue #182).
+
+    Args:
+        md5: The MD5 hash (with optional .dir suffix).
+        *cache_roots: Cache/remote roots to search, in priority order.
+            None/empty entries are skipped so callers can pass optional paths.
+
+    Returns:
+        Size in bytes, or None if the object was not found in any root.
+    """
+    for root in cache_roots:
+        if not root:
+            continue
+        path = find_source_file(md5, Path(root))
+        if path is None:
+            continue
+        try:
+            return path.stat().st_size
+        except OSError:
+            # Vanished or unreadable between the lookup and the stat -- try
+            # the next root rather than reporting a size we did not observe.
+            continue
+    return None
+
+
 def populate_cache_file(
     md5: str,
     source_cache: str,

dt/doctor.py

@@ -521,6 +521,43 @@ def check_dvc_repo() -> DiagnosticResult:
         )
 
 
+def check_dvcignore() -> DiagnosticResult:
+    """Check that .dvcignore keeps this repo's .gitignore files out of imports.
+
+    DVC writes a ``.gitignore`` beside everything it tracks, and its repo
+    filesystem does not exclude those files -- so anyone importing a *directory*
+    from this repo that is not itself an out gets our bookkeeping hashed into
+    their payload, ``nfiles`` and ``size`` (issue #182). ``dt init`` seeds the
+    pattern; repos created before that need it adding.
+    """
+    from . import utils
+
+    in_repo, root = check_in_dvc_repo()
+    if not in_repo or root is None:
+        return DiagnosticResult(
+            "dvcignore", True, "Not in a DVC repository (skipped)"
+        )
+
+    dvcignore = Path(root) / '.dvcignore'
+    patterns = set()
+    if dvcignore.exists():
+        patterns = {l.strip() for l in dvcignore.read_text().splitlines()}
+
+    if utils.DVCIGNORE_GITIGNORE_PATTERN in patterns:
+        return DiagnosticResult(
+            "dvcignore", True, ".dvcignore excludes .gitignore files"
+        )
+
+    return DiagnosticResult(
+        "dvcignore", False,
+        ".dvcignore does not exclude .gitignore files",
+        "DVC's own generated .gitignore files can be hashed into the payload "
+        "of anyone importing a directory from this repo. Add a '.gitignore' "
+        "line to .dvcignore and commit it. Note this changes the hash, nfiles "
+        "and size of any such directory import."
+    )
+
+
 def check_network() -> DiagnosticResult:
     """Check network connectivity (for dt doctor output)."""
     has_network = check_network_connectivity(timeout=2.0)
@@ -625,6 +662,7 @@ def run_diagnostics(verbose: bool = False) -> list[DiagnosticResult]:
     # Repository context checks
     results.append(check_git_repo())
     results.append(check_dvc_repo())
+    results.append(check_dvcignore())
 
     # Environment checks (may be slow)
     if verbose: