Expected normaliser errors?

[michaelwood]

Expected normaliser errors?

@rhiaro Do you think both these exceptions are expected in the normaliser? The second one looks to be a data quality issue that emanates from the publisher's data, the first one I'm not so sure about...

2021-03-01T17:31:57.606527945Z app[normaliser-worker.3]: Error running data through pipe NormaliseSchedulePipe 
2021-03-01T17:31:57.606566024Z app[normaliser-worker.3]:  TypeError: eventSchedule is not iterable
2021-03-01T17:31:57.606574680Z app[normaliser-worker.3]: TypeError: eventSchedule is not iterable
2021-03-01T17:31:57.606579369Z app[normaliser-worker.3]:     at NormaliseSchedulePipe.run (file:///app/src/lib/pipes/normalise-schedule-pipe.js:22:28)
2021-03-01T17:31:57.606583251Z app[normaliser-worker.3]:     at PipeLine.run (file:///app/src/lib/pipeline.js:15:46)
2021-03-01T17:31:57.606586695Z app[normaliser-worker.3]:     at runMicrotasks (<anonymous>)
2021-03-01T17:31:57.606591002Z app[normaliser-worker.3]:     at processTicksAndRejections (internal/process/task_queues.js:97:5)
2021-03-01T17:31:57.606594882Z app[normaliser-worker.3]:     at async normalise_data_publisher_feed (file:///app/src/lib/normalise-data.js:95:34)
2021-03-01T17:31:57.606598423Z app[normaliser-worker.3]: Saved as done and error saved

2021-03-01T17:31:57.032778982Z app[normaliser-worker.3]:     at encodeURIComponent (<anonymous>)
2021-03-01T17:31:57.032783018Z app[normaliser-worker.3]:     at Object.encode (/app/node_modules/uritemplate/bin/uritemplate.js:148:29)
2021-03-01T17:31:57.032802288Z app[normaliser-worker.3]:     at Object.encodeCharacter (/app/node_modules/uritemplate/bin/uritemplate.js:179:27)
2021-03-01T17:31:57.032807277Z app[normaliser-worker.3]:     at Object.encodeLiteral (/app/node_modules/uritemplate/bin/uritemplate.js:346:111)
2021-03-01T17:31:57.032812139Z app[normaliser-worker.3]:     at new LiteralExpression (/app/node_modules/uritemplate/bin/uritemplate.js:439:39)
2021-03-01T17:31:57.032816137Z app[normaliser-worker.3]:     at Function.parse (/app/node_modules/uritemplate/bin/uritemplate.js:615:30)
2021-03-01T17:31:57.032819717Z app[normaliser-worker.3]:     at Field.detectType (/app/node_modules/@openactive/data-model-validator/src/classes/field.js:165:40)
2021-03-01T17:31:57.032823562Z app[normaliser-worker.3]:     at FieldsCorrectTypeRule.validateField (/app/node_modules/@openactive/data-model-validator/src/rules/core/fields-correct-type-rule.js:209:34)
2021-03-01T17:31:57.032827080Z app[normaliser-worker.3]:     at FieldsCorrectTypeRule.validate (/app/node_modules/@openactive/data-model-validator/src/rules/rule.js:34:40)
2021-03-01T17:31:57.032830548Z app[normaliser-worker.3]:     at runMicrotasks (<anonymous>)
2021-03-01T17:31:57.032832802Z app[normaliser-worker.3]: Saved as done and error saved

2021-03-03T18:09:22.370978858Z app[normaliser-worker.2]: ERROR profile_normalised_data_for_item_for_profile
2021-03-03T18:09:22.371100642Z app[normaliser-worker.2]: error: duplicate key value violates unique constraint "normalised_data_profile_results_pkey"
2021-03-03T18:09:22.371109161Z app[normaliser-worker.2]:     at Parser.parseErrorMessage (/app/node_modules/pg-protocol/dist/parser.js:278:15)
2021-03-03T18:09:22.371114003Z app[normaliser-worker.2]:     at Parser.handlePacket (/app/node_modules/pg-protocol/dist/parser.js:126:29)
2021-03-03T18:09:22.371117576Z app[normaliser-worker.2]:     at Parser.parse (/app/node_modules/pg-protocol/dist/parser.js:39:38)
2021-03-03T18:09:22.371121347Z app[normaliser-worker.2]:     at Socket.<anonymous> (/app/node_modules/pg-protocol/dist/index.js:8:42)
2021-03-03T18:09:22.371125888Z app[normaliser-worker.2]:     at Socket.emit (events.js:321:20)
2021-03-03T18:09:22.371130053Z app[normaliser-worker.2]:     at Socket.EventEmitter.emit (domain.js:485:12)
2021-03-03T18:09:22.371141091Z app[normaliser-worker.2]:     at addChunk (_stream_readable.js:297:12)
2021-03-03T18:09:22.371144280Z app[normaliser-worker.2]:     at readableAddChunk (_stream_readable.js:273:9)
2021-03-03T18:09:22.371147124Z app[normaliser-worker.2]:     at Socket.Readable.push (_stream_readable.js:214:10)
2021-03-03T18:09:22.371150394Z app[normaliser-worker.2]:     at TCP.onStreamRead (internal/stream_base_commons.js:186:23) {
2021-03-03T18:09:22.371153780Z app[normaliser-worker.2]:   length: 300,
2021-03-03T18:09:22.371157142Z app[normaliser-worker.2]:   name: 'error',
2021-03-03T18:09:22.371160538Z app[normaliser-worker.2]:   severity: 'ERROR',
2021-03-03T18:09:22.371163504Z app[normaliser-worker.2]:   code: '23505',
2021-03-03T18:09:22.371167011Z app[normaliser-worker.2]:   detail: 'Key (normalised_data_id, profile_name)=(5399527, core) already exists.',
2021-03-03T18:09:22.371170385Z app[normaliser-worker.2]:   hint: undefined,
2021-03-03T18:09:22.371173517Z app[normaliser-worker.2]:   position: undefined,
2021-03-03T18:09:22.371176400Z app[normaliser-worker.2]:   internalPosition: undefined,
2021-03-03T18:09:22.371179595Z app[normaliser-worker.2]:   internalQuery: undefined,
2021-03-03T18:09:22.371182852Z app[normaliser-worker.2]:   where: undefined,
2021-03-03T18:09:22.371186540Z app[normaliser-worker.2]:   schema: 'public',
2021-03-03T18:09:22.371189689Z app[normaliser-worker.2]:   table: 'normalised_data_profile_results',
2021-03-03T18:09:22.371192887Z app[normaliser-worker.2]:   column: undefined,
2021-03-03T18:09:22.371195736Z app[normaliser-worker.2]:   dataType: undefined,
2021-03-03T18:09:22.371198705Z app[normaliser-worker.2]:   constraint: 'normalised_data_profile_results_pkey',
2021-03-03T18:09:22.371201387Z app[normaliser-worker.2]:   file: 'nbtinsert.c',
2021-03-03T18:09:22.371204628Z app[normaliser-worker.2]:   line: '563',
2021-03-03T18:09:22.371207754Z app[normaliser-worker.2]:   routine: '_bt_check_unique'
2021-03-03T18:09:22.371210741Z app[normaliser-worker.2]: }

[Lathrisk]

I looked at some of these a while back, not sure how much progress I made, but I think the first one arises because sometimes eventSchedule is an object not an array of objects - in the case where there is only one event in the schedule. I think this might just need to be wrapped in an array, iirc.