feat: admin meta CRUD API#299
Draft
silveirado wants to merge 90 commits intomainfrom
Draft
Conversation
Bumps [glob](https://github.com/isaacs/node-glob) from 10.3.10 to 10.5.0. - [Changelog](https://github.com/isaacs/node-glob/blob/main/changelog.md) - [Commits](isaacs/node-glob@v10.3.10...v10.5.0) --- updated-dependencies: - dependency-name: glob dependency-version: 10.5.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [body-parser](https://github.com/expressjs/body-parser) from 1.20.2 to 2.2.1. - [Release notes](https://github.com/expressjs/body-parser/releases) - [Changelog](https://github.com/expressjs/body-parser/blob/master/HISTORY.md) - [Commits](expressjs/body-parser@1.20.2...v2.2.1) --- updated-dependencies: - dependency-name: body-parser dependency-version: 2.2.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [nodemailer](https://github.com/nodemailer/nodemailer) from 6.9.9 to 7.0.11. - [Release notes](https://github.com/nodemailer/nodemailer/releases) - [Changelog](https://github.com/nodemailer/nodemailer/blob/master/CHANGELOG.md) - [Commits](nodemailer/nodemailer@v6.9.9...v7.0.11) --- updated-dependencies: - dependency-name: nodemailer dependency-version: 7.0.11 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
Bumps [jws](https://github.com/brianloveswords/node-jws) from 3.2.2 to 3.2.3. - [Release notes](https://github.com/brianloveswords/node-jws/releases) - [Changelog](https://github.com/auth0/node-jws/blob/master/CHANGELOG.md) - [Commits](auth0/node-jws@v3.2.2...v3.2.3) --- updated-dependencies: - dependency-name: jws dependency-version: 3.2.3 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
… streaming - Create findStream function with record-by-record processing - Extract common logic to findUtils.ts (DRY principle) - Create Transform streams for field permissions and date conversion - Add ObjectToJsonTransform for HTTP streaming - Add new endpoint /rest/stream/:document/findStream - Register streamApi in routes/index.ts - Add unit tests for Transform streams and findUtils - Add integration, E2E, and benchmark tests - Add confidence test to validate data consistency - All tests execute directly in Node (no Jest dependency) - Benchmark shows 82% memory reduction and 99% faster TTFB for 55k records TODO: Refactor and cleanup
- Extract magic numbers to streamConstants.ts (DRY)
- Replace let with const (const-pref)
- Replace forEach/for loops with functional methods (.map, .filter, .reduce)
- Extract helper functions from findUtils.ts (buildSortOptions, buildAccessConditionsForField, buildAccessConditionsMap, calculateConditionsKeys)
- Extract parseFilterFromQuery to eliminate duplication in streamApi.ts
- Create streamTestHelpers.ts with reusable test functions
- Use BluebirdPromise.map with concurrency limits in all promise operations
- Add default sort { _id: 1 } to findStream for consistent ordering
- Match find.ts behavior in findUtils.ts for query construction consistency
- Refactor test files to use helpers and functional methods
- Fix test variable references (testResults.allPassed)
All tests passing:
- Unit and integration tests: 7/7 passed
- Benchmark: 99.3% faster TTFB, 45% faster total time, 81.8% better throughput
- Confidence test: All datasets match exactly with find paginated endpoint
- Add comprehensive documentation for /rest/stream/:document/findStream endpoint - Document streaming format (newline-delimited JSON) - Include client-side processing examples (JavaScript) - Add advantages comparison with traditional find endpoint - Add usage guidelines and best practices - Update Postman collection with 3 new requests: - Find Stream (main request with all parameters) - Find Stream - Contact (simple example) - Find Stream - With Filter (complex filter example) - Documentation available in pt-BR and en - Include response examples and error handling
…ntation
- ADR-0001: HTTP Streaming para Busca de Dados
Documents decision to implement HTTP streaming endpoint
Includes performance metrics (68% memory reduction, 99.3% faster TTFB)
- ADR-0002: Extração de Lógica Comum para findUtils
Documents DRY principle application
Explains shared logic extraction between find and findStream
- ADR-0003: Node.js Transform Streams para Processamento Sequencial
Documents use of Transform streams for record-by-record processing
Explains pipeline architecture
- ADR-0004: Ordenação Padrão para Consistência
Documents default sorting decision ({ _id: 1 })
Explains consistency requirements for confidence tests
All ADRs available in pt-BR and en
Includes README files with index
- Add hasSecondaryNodes() function to check for available secondary nodes - Implement dynamic read preference selection: - Uses 'secondary' when secondaries are available (maximum isolation) - Falls back to 'secondaryPreferred' when no secondaries (no errors) - Add performance optimizations: - STREAM_BATCH_SIZE: 1000 documents per batch - STREAM_MAX_TIME_MS: 5 minutes max query time - Apply same read preference to countDocuments for consistency - Update ADR-0005 to reflect smart fallback approach - Works in all environments (dev without secondaries, prod with secondaries) See ADR-0005 for detailed rationale
- Remove KonectyResult (not used) - Remove errorReturn (not used) - Remove successReturn (not used) - Remove DataDocument (not used directly, only in streamTransforms) All imports are now used, lint passes without errors
- Add hierarchical pivot table structure with nested children - Enrich pivot config with metadata from MetaObject.Meta - Implement lookup field formatting with formatPattern - Add recursive field metadata resolution for nested lookups - Concatenate parent labels in nested fields (e.g., 'Grupo > Nome') - Calculate subtotals per hierarchy level - Calculate grand totals for all data - Update Python script to build hierarchical structure - Support Accept-Language header for multilingual labels - Update integration and unit tests for new structure Breaking changes: - Pivot API response format changed from flat array to hierarchical structure - Response now includes metadata, data (hierarchical), and grandTotals
…rmat - Update API documentation (en/pt-BR) with new hierarchical response structure - Add examples showing metadata, nested children, subtotals, and grandTotals - Document lookup formatting rules and nested field label concatenation - Add ADR-0007 documenting hierarchical pivot output format decision - Update ADR READMEs to include new ADR Breaking changes documented: - Response format changed from flat array to hierarchical structure - New metadata field with enriched field information - Nested children arrays for multi-level hierarchies - Subtotals per level and grand totals
- Update Postman collection example response to show new hierarchical structure - Include metadata, nested children, subtotals, and grandTotals in example - Reflect breaking change in response format
- Add Rust, cargo, and musl-dev for building polars from source on Alpine - Fix ENV format to use key=value syntax (removes warning) - Fix COPY paths to use absolute paths (/app instead of app) - Add python3-dev and py3-pip for Python development dependencies - Ensure konecty user has access to build tools - Note: polars will compile on first execution (takes ~2-5 minutes), then cached Alpine Linux (musl) doesn't have precompiled polars wheels, so compilation from source is required. This is handled automatically by uv when the script runs for the first time.
- Add GET /rest/data/:document/graph endpoint for SVG chart generation - Implement graphStream function orchestrating findStream + Python - Create graph_generator.py script using Polars for aggregations and pandas/matplotlib for visualization - Support 6 chart types: bar, line, pie, scatter, histogram, timeSeries - Add collectSVGFromPython function to pythonStreamBridge for SVG collection - Add GraphConfig and GraphStreamParams TypeScript types - Create unit and integration tests for graph endpoint - Add ADR-0008 documenting Polars+Pandas decision (pt-BR and en) - Update API documentation with graph endpoint examples (pt-BR and en) - Update Postman collection with graph examples using Opportunity document - Performance: Polars is 3-10x faster than Pandas for aggregations - Convert only aggregated results to Pandas (memory efficient) - Add pyarrow dependency for Polars to_pandas() conversion
- Fix wrong variable name in runFindStreamTests.ts (failed++ -> testResults.failed++) - Change BENCHMARK_ITERATION_CONCURRENCY from 3 to 1 for accurate memory measurements - Fix TypeScript linting errors (any type, empty line, type guards)
- Replace echo with printf in polars pre-build step - BusyBox ash doesn't interpret \n in echo, causing malformed input - printf correctly interprets \n as newline character - This ensures polars is properly pre-compiled during Docker build - Prevents multi-minute delay on first pivot/graph request
Bumps [@smithy/config-resolver](https://github.com/smithy-lang/smithy-typescript/tree/HEAD/packages/config-resolver) from 4.0.1 to 4.4.5. - [Release notes](https://github.com/smithy-lang/smithy-typescript/releases) - [Changelog](https://github.com/smithy-lang/smithy-typescript/blob/main/packages/config-resolver/CHANGELOG.md) - [Commits](https://github.com/smithy-lang/smithy-typescript/commits/@smithy/config-resolver@4.4.5/packages/config-resolver) --- updated-dependencies: - dependency-name: "@smithy/config-resolver" dependency-version: 4.4.5 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
- Add support for hierarchical column headers in pivotStream response - Update tests to validate presence and structure of columnHeaders - Modify API documentation to reflect new columnHeaders field - Implement logic in Python script to handle and return column headers - Ensure backward compatibility with existing pivot functionality Breaking changes: - Response format now includes columnHeaders, enhancing the pivot table structure.
- Pie charts: corrigido para sempre usar categoryField + aggregation, ignorando series - Bar charts: ajustado para usar xAxis.field quando disponível (em vez de sempre categoryField) - Campos de sistema lookup: melhorada projeção para _user, _createdBy, etc. (incluem _id para população) - Normalização: graph_type normalizado para lowercase em todas as comparações - Mensagens de erro: melhoradas com listagem de campos disponíveis para debug - Logs de debug: adicionados para rastrear identificação de lookups e campos disponíveis - Resolução de campos: melhorada para campos de sistema que são lookups
- graphMetadata: gera título instrutivo quando vazio
- Inclui nome do módulo/documento
- Usa campo de agrupamento principal (categoryField ou xAxis)
- Formato: '{Módulo} por {Campo}' (pt-BR) ou '{Module} by {Field}' (en)
- Preserva títulos existentes (não sobrescreve)
- graphMetadata: adicionado comentário sobre ADR-0016 e ADR-0008 - graphStream: adicionado comentário sobre ADR-0016 e ADR-0008 - Documentação: código segue decisões arquiteturais documentadas
- Adiciona seção 'graph' no pivot Activity:pivot:Default - Configura gráfico de pizza (pie chart) por status - Agrega por contagem (count) de atividades - Título i18n: "Activities by Status" / "Atividades por Situação" - Legendas habilitadas, grade desabilitada Documentação: - docs/pt-BR/metadata-pivot-graph.md: guia completo de configuração - Estrutura JSON detalhada - Exemplos para cada tipo de gráfico - Referência de campos e agregações - Bucketing temporal (year, quarter, month, week, day, hour) - Boas práticas e validação Esta configuração permite que usuários visualizem o gráfico automaticamente ao alternar para o modo "Gráfico" na interface, sem necessidade de configuração manual. Ref: ADR-0020 (frontend)
Bumps [undici](https://github.com/nodejs/undici) from 5.28.3 to 6.23.0. - [Release notes](https://github.com/nodejs/undici/releases) - [Commits](nodejs/undici@v5.28.3...v6.23.0) --- updated-dependencies: - dependency-name: undici dependency-version: 6.23.0 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com>
- sqlToRelationsParser: SQL to IQR with node-sql-parser, unit tests - RFC EN/PT-BR: status, open questions, Phase 2 details - crossModuleQueryValidator.test: default includeTotal/includeMeta assertions - package.json/yarn.lock: node-sql-parser dependency Co-authored-by: Cursor <cursoragent@cursor.com>
…iance - Use parent field for compound types (address, money, etc.) in projection to avoid clearProjectionPathCollision stripping sibling subfields (fixes address.district not appearing on graph X axis) - Honor Cache-Control: no-cache in withBlobCache for widget refresh - Validate graph config fields exist before stream; return 400 for GRAPH_FIELD_NOT_FOUND - Strip legacy yAxis when series present (graphStream, graphMetadata) - GRAPH_FIELD_NOT_FOUND message empathetic, non-technical (ADR-0024) - validateGraphFieldsExist: use .find() instead of for...of (ADR-0012) Co-authored-by: Cursor <cursoragent@cursor.com>
- Introduced a new section in the filters documentation explaining the `exists` operator's behavior regarding absent fields, `null`, and empty arrays. - Updated relevant documents in both English and Portuguese to clarify how `exists: false` only matches documents where the field is absent, not those with empty arrays or `null`. - Added a workaround for filtering absent or empty fields and noted limitations for the `file` field type. - Updated the changelog to reflect these changes. This documentation aims to provide users with a clear understanding of the expected behavior when using the `exists` operator.
…ction - Enhanced the userInfo function to include customTheme and customLogo properties from MetaObject.Namespace. - This addition improves user customization options for themes and logos. This change aligns with the ongoing efforts to enhance user interface flexibility.
Adds support for limiting the number of categories (xAxisLimit) and series (yAxisLimit) displayed in graphs, with configurable sort order (limitOrder: desc/asc). Applied after aggregation in graph_generator.py. Made-with: Cursor
Pass strict=False to pl.DataFrame() so columns with mixed Int64/Float64 values (e.g. 29.5 in an integer column) are coerced instead of raising TypeError. Made-with: Cursor
Null, empty string and "null" values in the X-axis category column now display as "(Vazio)" (pt) or "(Empty)" (en) instead of blank bars. Made-with: Cursor
…Stream (#293) Made-with: Cursor
…ns (#294) * chore: wip kpi aggregator com contagem unica e ajustes no dataApi/kpiStream Made-with: Cursor * feat(charts): backend support for data labels and full picklist options - GraphConfig: showDataLabels, includeAllPicklistOptions, xAxisPicklistOptions (enriched) - graphMetadata: getPicklistOptionValues + enrich xAxisPicklistOptions when includeAllPicklistOptions and X field is picklist - graph_generator.py: bar_label for bars, autopct for pie, annotate for line/scatter/timeseries; left join with picklist options and fill_null(0) Made-with: Cursor
Reverts the retry/cache mount added in 370632a. The original simple yarn install is sufficient; the 500 error was a transient registry issue. Made-with: Cursor
- Implemented POST /rest/access/:document for creating access entries. - Implemented DELETE /rest/access/:document/:accessName for deleting access entries. - Registered admin API in the main routes file for enhanced access management.
* chore(data-explorer): wip - interface de query builder validada Made-with: Cursor * feat(data-explorer): execução de query e exportação Excel finalizadas - Relações opcionais: cross-module query sem relação retorna só o módulo - Python: aceita relations vazias; extract_nested_value trata array (ex: email.address) - Export API: colunas explícitas (body.fields), CSV/XLSX com colunas ordenadas - TransformFlattenData: initialHeaders para ordem estável nas exportações Made-with: Cursor * chore(data-explorer): align crossModuleQuery and queryExportApi with ADR-12, ADR-14, ADR-24 ADR-12: prefer-const, named constants (LARGE_DATASET_WARNING_THRESHOLD, DEFAULT_EXPORT_LIMIT), functional style (flatMap, filter), no magic numbers. ADR-14: error and warning messages as translation keys (dataExplorer.errors.*, dataExplorer.export.errors.*). ADR-24: user-facing wording via keys; UI translates at display time. Made-with: Cursor * fix(crossmodulequery): normalize relation aggregator field for Python (email[0].address) - When building Python config for each relation, strip relation prefix from aggregator field so Python reads correct path from child document (e.g. email.address -> email[0].address) Made-with: Cursor * fix(cross-module query): resolve isList reverse lookups (e.g. Contact.staff) - resolveRelationLookup: for self-referential isList, find forward FK field (e.g. mainContact) and use parentKey=_id, childKey=mainContact._id - buildAugmentedFields: skip relation-prefixed fields in groupBy/aggregators - Use unique dataset name (Contact:staff) for self-referential relations - Augment relation findStream with childKey field for join - Python: expand records for relation-prefixed groupBy, add prefix to config Fixes staff.name.full and first_staff_email_address returning null when data exists. Changelog: 2026-03-13_crossmodule-isList-fix.md Closes #295 Made-with: Cursor * feat(cross-module query): remove duplicated data from aggregated children * fix(cross-module query): remove internal prop from result --------- Co-authored-by: Leonardo Viva <leonardo.leal@konecty.com>
- add **/__pycache__/ and *.pyc to .gitignore - remove src/scripts/python/__pycache__/pivot_table.cpython-314.pyc from index (was committed by mistake) Made-with: Cursor
* fix(auth): otp email delivery without message create permission - insert OTP email message directly into collection instead of data.create() so sending is not blocked by user create permission on Message - expose delivery error in API response (errors[0].debug) when NODE_ENV !== production - map 'Message collection not found' to user-facing message Made-with: Cursor * fix(auth): improve error handling for OTP email delivery - Refactor error handling in OTP email delivery to map specific error messages to user-friendly responses. - Introduce MESSAGE_COLLECTION_NOT_FOUND_ERROR constant for better readability and maintainability. - Ensure that users receive appropriate feedback when email resources are unavailable. Made-with: Cursor --------- Co-authored-by: Wagner Terry <goncalveswagner15@gmail.com>
- Introduced MESSAGE_COLLECTION_NOT_FOUND_ERROR constant for better readability. - Updated error handling to provide user-friendly messages when the message collection is unavailable. - Ensured proper type for message data when inserting into the collection. This change enhances the maintainability and clarity of error responses in the OTP email delivery process.
Implements admin-only /api/admin/meta/* endpoints for reading and editing MetaObjects. All endpoints require user.admin === true. Endpoints: GET list, GET by document, GET by type/name, GET hook, PUT upsert, DELETE, PUT hook, DELETE hook, POST reload. See docs/adr/0006-meta-crud-api.md for architecture decision. Made-with: Cursor
Made-with: Cursor
- Zod validation on all PUT; MetaObjects.History before update/delete - GET /:metaId/history, GET /:metaId/history/:version, POST /:metaId/rollback - POST /doctor validates schema, hooks, lookups (all documents) - POST /hook/validate for dry-run hook validation - Centralized hook rules: no comments, blocked APIs, required return - loadMetaObjects: try/catch per meta, skip invalid instead of crash - Docs EN/PT-BR, Postman, ADR 0012, changelog updated Made-with: Cursor
Resolve conflicts: combine admin meta API with access-overview routes, keep MCP build/widgets and graph xAxisSort from main, ADR pt-BR meta CRUD renumbered to 0020, unify changelog and ADR indexes, Postman JSON. Made-with: Cursor
- Updated the `dbLoad` function to improve logging for schema validation issues. - Introduced a new counter for schema warnings, allowing for better tracking of potential issues while still loading valid meta objects. - Adjusted logging levels to differentiate between schema warnings and invalid meta objects, enhancing clarity in error reporting. - This change aims to improve observability and maintainability of the meta object loading process.
- Updated ServerStorage.upload() to send the hashed filename as the multipart filename, ensuring alignment between the database key and the remote storage object key. - Removed the unnecessary decodeFilenameForBlob function as it is no longer needed in the upload flow. - Added changelog entry to document this fix, which resolves issues with image URLs based on persisted hashed keys. This change improves the consistency of file storage and retrieval in the application.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
/api/admin/meta/*endpoints for reading and editing MetaObjectsuser.admin === true(enforced via shared preHandler hook)loadMetaObjects()after changesEndpoints
/api/admin/meta/api/admin/meta/:document/api/admin/meta/:document/:type/:name/api/admin/meta/:document/hook/:hookName/api/admin/meta/:document/:type/:name/api/admin/meta/:document/:type/:name/api/admin/meta/:document/hook/:hookName/api/admin/meta/:document/hook/:hookName/api/admin/meta/reloadADR
See
docs/adr/0006-meta-crud-api.mdTest plan
Made with Cursor