The ed2k hash calculation result differ from other commonly used tools

[nanoia]

The calculation results do not match those obtained using other tools such as Amule , emule LinkCreator. ChatGPT says Write() will never handle “that last incomplete chunk.”

hashit/vendor/go.felesatra.moe/hash/ed2k/ed2k.go

Lines 52 to 65 in dab96e6

	func (h *Hash) Write(p []byte) (int, error) {
	total := len(p)
	for len(p) > 0 {
	p2 := h.limitNextChunk(p)
	n, _ := h.subhash.Write(p2)
	h.written += int64(n)
	p = p[n:]
	if h.written%ChunkSize == 0 {
	h.hashlist = h.subhash.Sum(h.hashlist)
	h.subhash.Reset()
	}
	}
	return total, nil
	}

hashit/vendor/go.felesatra.moe/hash/ed2k/ed2k.go

Lines 78 to 88 in dab96e6

	func (h *Hash) Sum(b []byte) []byte {
	if h.written < ChunkSize {
	return h.subhash.Sum(b)
	}
	if h.written == ChunkSize {
	return append(b, h.hashlist...)
	}
	h2 := md4.New()
	h2.Write(h.hashlist)
	return h2.Sum(b)
	}

// Provided by ChatGPT
func (h *Hash) Sum(b []byte) []byte {
	if h.written == 0 {
		return h.subhash.Sum(b)
	}

	// uncommited chunk
	if h.written%ChunkSize != 0 {
		h.hashlist = h.subhash.Sum(h.hashlist)
	}

	if h.written <= ChunkSize {
		return append(b, h.hashlist...)
	}

	h2 := md4.New()
	h2.Write(h.hashlist)
	return h2.Sum(b)
}

Reference:
https://wiki.anidb.net/Ed2k-hash#How_is_an_ed2k_hash_calculated_exactly?

[boyter]

Well good thing I have not made it a release yet.

[boyter]

Yes it is indeed a bug in the library where a file larger than 9,728,000 bytes and not a multiple of 9,728,000 bytes would not finish off.

I have merged a fix for it and raised a bug report in the repository which hopefully results in an upstream fix.

[darkfeline]

(FYI, I think the AI patch is "incorrect" because it modifies the internal state when Sum() is called. This will cause incorrect values after the first Sum() call, though it's not a problem if Sum() is only ever called once.)

(Also FYI, there's an alternate implementation of ed2k hash that is generally considered wrong but still somewhat in use, where an extra zero block is appended if the data is an exact multiple of the block size. My lib only implements the "correct" version.)

[boyter]

Yeah in this case only a single sum is called. More to ensure I had it working, but will be moving to yours soon @darkfeline

Knowing about the other bug is useful, I did consider swapping but the fix was fairly trivial to implement.

[boyter]

TODO upgrade to latest version of dependancy and verify

[nanoia]

Hello, the upstream dependency has released a patch. Can we pull in the latest dependency and release a new version of this project?

[boyter]

Yes... its been on my todo list, but work has gotten in the way.

Note the current release from master will have the patch in there that resolves it still.