[RED TEAM] PR #112 universal grammar loader cannot load tsx grammar (wrong import module name)

[Wolfvin]

Temuan

PR #112 (universal grammar loader) added PACKAGE_NAME_OVERRIDES to map language → PyPI distribution name (e.g. tsx → tree-sitter-typescript). But there is no equivalent override for the Python import module name. _import_module_name() always returns tree_sitter_<language>:

# scripts/universal_grammar_loader.py
def _import_module_name(language: str) -> str:
    return "tree_sitter_" + _normalize_language_name(language)

For tsx, this returns tree_sitter_tsx — but no such module exists. The actual module is tree_sitter_typescript (which exposes both language_typescript() and language_tsx() entry points).

_try_import("tsx") therefore always returns None, load_grammar("tsx") always returns None, and GrammarLoader.get_language("tsx") always returns None — even when tree-sitter-typescript is fully installed and importable.

The downstream effect: scripts/parsers/ts_backend_parser.py calls loader.get_language('tsx') in __init__, gets None, raises RuntimeError("tree-sitter-typescript not installed"), which is then caught by scripts/commands/scan.py:710 and logged as the misleading warning:

[codelens] WARNING: TSBackendParser init failed, using JS fallback: tree-sitter-typescript not installed

This warning is false — the package IS installed. The scan then falls back to JSBackendParser for ALL .ts files, missing TypeScript-specific syntax (interfaces, type aliases, generics, decorators, enum declarations, etc.).

Cara reproduce

pip install tree-sitter-typescript tree-sitter-python
python -c "
import sys; sys.path.insert(0, 'scripts')
from universal_grammar_loader import load_grammar, _try_import, _import_module_name
print('module name for tsx:', _import_module_name('tsx'))   # → tree_sitter_tsx  (wrong)
print('_try_import(tsx):', _try_import('tsx'))              # → None             (wrong)
print('load_grammar(tsx):', load_grammar('tsx'))            # → None             (wrong)
print('load_grammar(typescript):', load_grammar('typescript'))  # → <Language ...> (works)
import tree_sitter_typescript  # works fine
print('package is importable:', tree_sitter_typescript)
"

Then run a scan and observe the false warning:

PYTHONUTF8=1 python scripts/codelens.py scan . --verbose 2>&1 | head -5

Output aktual

module name for tsx: tree_sitter_tsx
_try_import(tsx): None
load_grammar(tsx): None
load_grammar(typescript): <Language id=140075551898912, version=14, name=None>
package is importable: <module 'tree_sitter_typescript' from '/home/z/.venv/lib/python3.12/site-packages/tree_sitter_typescript/__init__.py'>

Scan output (first 5 lines):

[CodeLens] Auto-detected workspace: /home/z/my-project/workerA/CodeLens
[codelens] WARNING: TSBackendParser init failed, using JS fallback: tree-sitter-typescript not installed
[codelens] WARNING: populate_graph_tables: batch write error: UNIQUE constraint failed: graph_nodes.node_id
[CodeLens] Scan results persisted to SQLite database.
{

Output yang diharapkan

• _import_module_name("tsx") should return tree_sitter_typescript (mirroring PACKAGE_NAME_OVERRIDES).
• load_grammar("tsx") should return a non-None Language object (calling tree_sitter_typescript.language_tsx()).
• TSBackendParser should initialize successfully when tree-sitter-typescript is installed.
• The misleading warning "tree-sitter-typescript not installed" should not appear when the package is in fact installed.

Likely fix: add a MODULE_NAME_OVERRIDES dict (or extend PACKAGE_NAME_OVERRIDES to a single LANGUAGE_OVERRIDES dict covering both PyPI name and import module name) with entries like "tsx": "tree_sitter_typescript".

Severity

HIGH — TypeScript / TSX (React) parsing is silently broken on every install. Users see a false "tree-sitter-typescript not installed" warning even after installing the package, and all .ts files fall back to the JS parser, losing type-system constructs from the call graph.

PR terkait

#112 (universal grammar loader — closes #18)