refactor: enhance token estimation and tool result handling in conversation service

[chenyme]

Summary

Fix oversized MCP tool results being truncated by fixed character limits before reaching the model.

Tool results now use the selected model’s effective context budget. Readable results remain complete when they fit; oversized batches are fairly rebalanced across the current tool round, preserving useful head and tail content. Older complete conversation turns are removed only when necessary.

Full raw MCP output remains persisted. Opaque binary/base64 payloads stay excluded from model context. OpenAI Responses stateful continuation is disabled when local history adjustments would make the stored upstream state inconsistent.

Change type

• Bug fix
• Feature
• Documentation
• Refactor
• Configuration / deployment
• Security hardening
• Other

Affected areas

• Frontend / UI
• Backend / API
• Authentication / authorization
• Conversations / streaming
• Files / RAG / extraction
• Model routing / providers
• MCP / tools
• Billing / payments
• Admin console
• Deployment / Docker / configuration
• Documentation

Verification

• cd backend && go test ./...
• cd backend && go test -race -count=1 ./internal/application/conversation
• cd backend && go vet ./...
• git diff --check

Screenshots, API examples, or logs

Not applicable. This change affects backend MCP result processing and model context assembly.

Configuration, migration, and compatibility notes

No configuration, database migration, or public API changes are required.

The existing MCP transport size limit remains unchanged. Full tool output continues to be stored in conversation tool-call records; only the model-visible copy is constrained by the effective context budget.

Documentation

• Documentation is not needed for this change.
• Documentation was updated.
• Documentation still needs to be updated.

Security and privacy

• No secrets, tokens, credentials, local config, or personal data are included.
• User data access remains scoped by authenticated user context unless an admin-only path explicitly requires broader access.
• Security-sensitive behavior was reviewed, including authentication, authorization, provider routing, file processing, billing, and admin APIs where relevant.

Checklist

• I searched existing issues and pull requests.
• Changes are focused and do not include unrelated refactors.
• Tests or static verification were run where practical.
• User-facing behavior, deployment steps, API contracts, or configuration changes are documented.
• Generated artifacts are included only when this project explicitly requires them.
• Caches, build output, .pyc files, .env files, and local storage data are not committed.