refactor: Extract shared connector utilities to @dashframe/engine

[youhaowei]

Summary

After reviewing the codebase, there's significant code duplication across connectors (CSV, JSON, Notion) that should be extracted to @dashframe/engine.

Duplicated Code (~110 lines per connector)

1. Type Inference (inferType, parseValue)

• CSV: packages/connector-csv/src/index.ts (lines 32-66)
• JSON: packages/connector-json/src/index.ts (lines 50-126)
• Nearly identical logic for detecting number, boolean, date, string types
• Same parseValue function for converting raw values to typed values

2. Arrow DataFrame Creation

• CSV: packages/connector-csv/src/index.ts (lines 172-215)
• JSON: packages/connector-json/src/index.ts (lines 260-301)
• Same pattern: create Arrow vectors by type, build Table, convert to IPC, create DataFrame

3. Field Generation

• CSV: packages/connector-csv/src/index.ts (lines 151-170)
• JSON: packages/connector-json/src/index.ts (lines 239-258)
• Notion: packages/connector-notion/src/index.ts (lines 92-142)
• Same _rowIndex computed field pattern, UUID generation, SourceSchema creation

4. Primary Key Detection

• Both CSV and JSON use: /^_?id$/i.test(col.name)
• Falls back to _rowIndex if no ID column found

Proposed Solution

Add shared utilities to @dashframe/engine:

packages/engine/src/
├── connectors/
│   ├── FileSourceConnector.ts  # (existing)
│   ├── type-inference.ts       # inferType, parseValue, detectPrimaryKey
│   ├── arrow-helpers.ts        # createArrowColumns, createDataFrame  
│   ├── field-generation.ts     # generateFields, generateSourceSchema
│   └── index.ts

Exports

// type-inference.ts
export function inferColumnType(value: unknown): ColumnType
export function parseValueByType(value: unknown, type: ColumnType): unknown
export function detectPrimaryKeyColumn(columnNames: string[]): string

// arrow-helpers.ts
export function createArrowColumnsFromRows(
  rows: Record<string, unknown>[],
  columns: { name: string; type: ColumnType }[]
): Record<string, Vector<DataType>>

// field-generation.ts
export function generateFieldsFromColumns(
  columns: TableColumn[],
  tableId: UUID,
  systemFields?: { name: string; type: ColumnType }[]
): Field[]
export function generateSourceSchema(columns: TableColumn[]): SourceSchema

Benefits

• ~110 lines removed from each connector
• Consistency - single source of truth for type inference
• Easier to add connectors - Excel, Parquet, XML connectors get utilities for free
• Better testing - centralized logic = comprehensive tests in one place
• Simpler dependency graph - connectors already depend on @dashframe/engine

Tasks

• Create packages/engine/src/connectors/type-inference.ts
• Create packages/engine/src/connectors/arrow-helpers.ts
• Create packages/engine/src/connectors/field-generation.ts
• Add comprehensive tests for shared utilities
• Refactor @dashframe/connector-csv to use shared utilities
• Refactor @dashframe/connector-json to use shared utilities
• Refactor @dashframe/connector-notion to use shared utilities
• Update exports in @dashframe/engine

Related

• PR feat: Add JSON file connector for importing JSON data #20 - Add JSON file connector (introduced the duplication pattern)

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Similar Issues

Related Issues

• Migrate VegaChart to vgplot for large dataset optimization #2

🔗 Related PRs

#16 - Add test for queryBuilder [merged]

👤 Suggested Assignees

• youhaowei

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!