Stability

[bonomani]

Stability and Memory Leak / Increase

Given the complexity of the Perl program, a slight memory leak or increase is expected and has been observed. However, minimizing this is a primary objective.

• On Linux: Work in Progress
• On BSD: TBD
• Xymon-rrd Issue: A previously identified issue has been resolved (Which version?).

Plan

Step #1: Detect/Identify Sources of Problems (WIP)

• Valgrind: Initial testing on Xymon rrd suggests no leaks, although Xymon does not free memory as expected when stopped.
• Additional Tools:
  • Devel::Cycle: Testing for circular references (no output indicates no leak).
  • Devel::MAT: [User Guide](https://metacpan.org/dist/Devel-MAT/view/lib/Devel/MAT/UserGuide.pod) (WIP).
  • Other tools: Are there additional tools to consider?
• Approach:
  • Divide the program into smaller segments to pinpoint issues.
  • Explore Pure Perl (PP) versions of external libraries.
• Compilation Warnings: Xymon rrd emits multiple warnings, some of which are fixed in version 4.4.
  • Therabithia uses rrd-cache, planned for inclusion in Xymon 4.4. Will this impact the outcome?

Step #2: Fix/Apply Corrections (WIP)

• Devmon Errors: Execute Devmon in the foreground and monitor logs (largely DONE).
• Perl Best Practices: Work in Progress.

Step #3: Control Results (WIP)

---

Current Status

Step #1: Problem Sources Identified

1. Stale Forks: When forks terminate, memory is not properly released, causing leaks.
2. Memory Duplication with Forks: Forks replicate the main process and are not memory-optimized. For example, with 10 children and 1 master process, memory usage becomes 11 times that of the main process, contrary to expectations based on Copy-On-Write efficiency.
   • Currently using a pool of 10 processes (why this limitation?). Increasing the number could potentially accommodate more requests.
   • Implemented a Boss-Worker model with worker processes being re-forked upon unexpected termination.

Step #2: Implement Corrections

1. Prevent Fork Termination: Rewrote SNMPWalk Engine for Retry Management (COMPLETED in v0.25.03 for SNMP_Session and NET_SNMP_C lib).

   • Partial Retries: The engine continues polling using base primitives (e.g., getbulk, getnext, get) until all data is gathered. If a timeout occurs (max 4 seconds per primitive, configurable), the device is reinjected into the pool to complete pending OIDs. If the global timeout is reached, retries are abandoned, and partial results are shown. This ensures polling completes without unnecessary delays or prematurely killing forks.

   • Dynamic Timeout & Retry Optimization: The retry mechanism dynamically adjusts based on network conditions and device responsiveness, improving reliability and efficiency.

2. Memory Duplication with Forks: A comprehensive review of the forking process is in progress (WIP).

Step #3: Control (WIP)

1. Avoid Fork Termination: Rewriting of the SNMPWalk engine to manage retries successfully completed.
   • Unexpected Process Termination: Significant improvement observed, stable over several months.
   • Memory Increment: Some memory increment persists, though reduced compared to before. It's unclear whether this is related to Devmon or an SNMP polling issue.
     • The issue seems sporadic and likely arises during network outages, suggesting incomplete SNMP polling.
     • Interpreting memory data from Xymon and pinpointing the process responsible remains unclear.
     • RRD remains under suspicion. The next step is to remove all graphs for further investigation. However, the introduction of RRD-cache in Xymon 4.4 may postpone this troubleshooting.

---

Number of Forks

• Description: The number of forks indicates the count of SNMP polling processes running concurrently.
• Default: 10 forks.
• Increased Forks (>10): Experimental. Increasing the number appears to cause process blockages (exact cause TBD).

---

This version presents the same information with enhanced clarity, streamlined structure, and precise technical language.