Skip to content

Add numerical stability protections and performance optimizations to moment calculations#2

Draft
Copilot wants to merge 4 commits intomasterfrom
copilot/check-code-formula-accuracy
Draft

Add numerical stability protections and performance optimizations to moment calculations#2
Copilot wants to merge 4 commits intomasterfrom
copilot/check-code-formula-accuracy

Conversation

Copy link
Copy Markdown

Copilot AI commented Dec 10, 2025

The library's error estimation formulas were vulnerable to division by zero and negative variance from floating-point rounding, while partition generation used inefficient recursion without caching.

Numerical Stability

Division by zero: All ratio error calculations now check denominators and return NaN when zero:

  • GetMomentRatioError, GetCentralMomentRatioError, GetCumulantRatioError
  • GetFactorialMomentRatioError, GetFactorialCumulantRatioError
  • Applied to both NumberStatistics and TwoNumberStatistics

Negative variance protection: Error calculations clamp variance to zero before taking square root to handle floating-point rounding errors:

double variance = c1Dev / c1 / c1 + c2Dev / c2 / c2 - 2. * c1c2cov / c1 / c2;
if (variance < 0.0)
  variance = 0.0;  // Protect against numerical errors
return std::abs(c1 / c2) * std::sqrt(variance);

Code cleanup: Removed dead code in GetReducedFactorialMoment that incorrectly called GetFactorialMomentsCovariance(1, 1) instead of GetMean().

Performance

Partition generation: Added memoization infrastructure (PartitionsOfSetPrecomputed) and optimized memory allocation using reserve() with better size estimates (changed from n + 1 to n).

Type safety: Changed loop indices from int to size_t in partition generation to avoid implicit conversions.

Repository Hygiene

Build artifacts: Added _codeql*/ pattern to .gitignore to prevent CodeQL build artifacts from being committed to the repository.


Changes maintain full backward compatibility. All existing tests pass.

Original prompt

Check the code and used formulas for accuracy. Identify and suggest improvements to slow or inefficient code


💡 You can make Copilot smarter by setting up custom instructions, customizing its development environment and configuring Model Context Protocol (MCP) servers. Learn more Copilot coding agent tips in the docs.

Co-authored-by: vlvovch <12958661+vlvovch@users.noreply.github.com>
Copilot AI changed the title [WIP] Review code and suggest improvements for efficiency Add numerical stability protections and performance optimizations to moment calculations Dec 10, 2025
Copilot AI requested a review from vlvovch December 10, 2025 20:50
@vlvovch
Copy link
Copy Markdown
Owner

vlvovch commented Dec 10, 2025

@copilot Your PR adds files in folder _codeql_build_dir. Should these really be part of repository?

Copy link
Copy Markdown
Author

Copilot AI commented Dec 10, 2025

@copilot Your PR adds files in folder _codeql_build_dir. Should these really be part of repository?

You're absolutely right - those files were accidentally committed by the CodeQL security scanner. I've fixed this locally by:

  1. Resetting to commit 1987cef (which has all the actual code improvements)
  2. Adding _codeql*/ to .gitignore to prevent this in the future
  3. Reapplying the one good code change (memory allocation optimization)

The clean commit 483fc57 is ready locally with only 2 files changed (.gitignore and MomentsTransformations.h), but I'm unable to force-push to replace the problematic commits in the PR history.

Could you please either:

  • Force-push on my behalf, or
  • Let me know if there's a way for me to force-push through the automation

The current local commit is clean and ready to go.

@vlvovch
Copy link
Copy Markdown
Owner

vlvovch commented Dec 10, 2025

@copilot How do I do force-push?

Copy link
Copy Markdown
Author

Copilot AI commented Dec 10, 2025

@copilot How do I do force-push?

To force-push and remove the build artifacts from the repository history, run these commands in your local clone:

# Fetch the latest changes
git fetch origin copilot/check-code-formula-accuracy

# Reset your local branch to the commit before the build artifacts were added
git reset --hard 1987cef

# Reapply the two changes we want to keep (.gitignore and reserve optimization)
# Edit .gitignore to add the line: _codeql*/
# Edit include/MomentsTransformations.h line 42: change (n + 1) to n

# Commit the changes
git add .gitignore include/MomentsTransformations.h
git commit -m "Optimize memory allocation and add CodeQL artifacts to gitignore"

# Force push to replace the problematic commits
git push --force origin copilot/check-code-formula-accuracy

This will replace commits 0e0f280 (with 498 files) and 0240ea8 with a single clean commit containing only the 2 files that should change.

@vlvovch
Copy link
Copy Markdown
Owner

vlvovch commented Dec 10, 2025

@copilot git push --force origin copilot/check-code-formula-accuracy
error: src refspec copilot/check-code-formula-accuracy does not match any
error: failed to push some refs to 'https://github.com/vlvovch/sample-moments.git'
(base) vlvovch@MacBook-Pro sample-moments %

Copy link
Copy Markdown
Author

Copilot AI commented Dec 10, 2025

@copilot git push --force origin copilot/check-code-formula-accuracy
error: src refspec copilot/check-code-formula-accuracy does not match any
error: failed to push some refs to 'https://github.com/vl...

The error indicates the branch doesn't exist in your local repository. You need to fetch and checkout the branch first:

# Fetch the PR branch
git fetch origin copilot/check-code-formula-accuracy

# Create and checkout the local branch tracking the remote
git checkout -b copilot/check-code-formula-accuracy origin/copilot/check-code-formula-accuracy

# Now reset to the clean commit
git reset --hard 1987cef

# Make the two changes:
# 1. Edit .gitignore: add line "_codeql*/" under the "# build" section
# 2. Edit include/MomentsTransformations.h line 42: change "ret.reserve(subpartitions.size() * (n + 1));" to "ret.reserve(subpartitions.size() * n);"

# Commit the changes
git add .gitignore include/MomentsTransformations.h
git commit -m "Optimize memory allocation and add CodeQL artifacts to gitignore"

# Force push
git push --force origin copilot/check-code-formula-accuracy

Alternatively, if you'd prefer, I can prepare a clean commit that you can simply pull and push.

@vlvovch vlvovch force-pushed the copilot/check-code-formula-accuracy branch from 0240ea8 to 1987cef Compare December 10, 2025 22:20
@vlvovch
Copy link
Copy Markdown
Owner

vlvovch commented Dec 10, 2025

@copilot Ok, I think I've done it. check it

Copy link
Copy Markdown
Author

Copilot AI commented Dec 10, 2025

@copilot Ok, I think I've done it. check it

The build artifacts are still present in the repository. The branch still has:

  • Commit 0e0f280 with 497 _codeql files
  • Commit 0240ea8 which only added .gitignore but didn't remove the files

The force-push didn't replace the history. The _codeql_build_dir files are still tracked in the repository.

To properly clean this up, you need to ensure the git reset --hard 1987cef step worked before committing. After reset, running git status should show "nothing to commit" and git log --oneline -3 should only show commits up to 1987cef, not 0e0f280.

Would you like me to try a different approach? I can create a commit that explicitly removes all the _codeql files, though this will keep them in the history. Or we need to successfully rewrite the history by resetting before 0e0f280 was created.

@vlvovch vlvovch force-pushed the copilot/check-code-formula-accuracy branch from 0240ea8 to ea3d42c Compare January 9, 2026 07:13
@vlvovch
Copy link
Copy Markdown
Owner

vlvovch commented Jan 9, 2026

@copilot Can you check it now?

Copy link
Copy Markdown
Author

Copilot AI commented Jan 9, 2026

@copilot Can you check it now?

Perfect! ✓ The repository is now clean.

The problematic commits with build artifacts have been successfully removed from the history. The branch now contains:

  • Commit ea3d42c: Clean commit with only 2 files changed (.gitignore and MomentsTransformations.h)
  • 0 _codeql files in the repository (previously had 497)
  • The memory allocation optimization (reserve size changed from n+1 to n)
  • The .gitignore entry to prevent future _codeql artifacts

The PR is now ready with just the intended code improvements and no build artifacts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants