Skip to content

Remove usage of index_schema_history instead use pg_indexes #244

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
vishalhaven wants to merge 54 commits into
base: master
Choose a base branch
from
Open

Remove usage of index_schema_history instead use pg_indexes #244

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
b56bb46
target es2020
Jun 22, 2022
6c7a1c2
full change tree
Nov 29, 2022
9998379
updated syntax
Dec 6, 2022
9d20201
es2020 updates
Jan 25, 2023
37badab
fix test case on supertypelogger clone
Jan 25, 2023
65e5fb6
Merge branch 'master' into cap/feature/HL-00000-es2020-typescript-upg…
kamranHaven Jan 25, 2023
ee4e4d6
types update
Jan 25, 2023
30c1959
fix supertype cases
Jan 26, 2023
eca8b68
test solutions
Jan 27, 2023
2373a29
semotus changes
Jan 29, 2023
bff5ef0
add persistor changes
Jan 30, 2023
d816578
persistor changes
Jan 30, 2023
b5b1e1f
remove irrelavant logs
Jan 30, 2023
de0e176
merge with supertype changes
Jan 30, 2023
3661737
add updated supertype packages
Jan 30, 2023
2d17955
update packages
Jan 30, 2023
646635e
update supertype export
Jan 30, 2023
084bcb1
supertype updates
Jan 31, 2023
39adb17
refactor supertype branch
Jan 31, 2023
8f0d1fc
update packages
Jan 31, 2023
c76dbec
pakage locks updates
Jan 31, 2023
cb48ab7
semtus refactor
Jan 31, 2023
18284ce
semotus updates
Jan 31, 2023
8621810
update semotus and persistor
Jan 31, 2023
40f4562
ts config changes
Jan 31, 2023
b11b043
update supertype decorators
Feb 6, 2023
f1ad62a
bump packags
Feb 6, 2023
3cfc7fb
pakage updates
Feb 13, 2023
6e05f0b
add extentions to file to support esm
Mar 23, 2023
71ff907
push updates
Mar 23, 2023
e2ad026
merge latest feature
Mar 23, 2023
07d01b3
Presistor: Move q dev-dep, remove tv4 and add ajv for json schema val…
vishalhaven Mar 23, 2023
51cff34
add extentions to code base in semotus
Mar 24, 2023
2ed42cf
merge latest
Mar 24, 2023
9f8103c
bump supertype
Apr 7, 2023
067880e
merge latest supertype
Apr 18, 2023
29e07a9
remove unncessary import
Apr 18, 2023
d8ab747
remove .js extensions
Apr 18, 2023
4e28498
remove useDefineForClassFields flag
Apr 18, 2023
4b3ddb5
remove .js extensions
Apr 18, 2023
1b4f3ec
update formatting
Apr 18, 2023
7e56fd9
update HISTORY values
Apr 18, 2023
3aefe51
bump supertype on root
Apr 18, 2023
dd67292
switch ajv from require to import
Apr 19, 2023
b2c1db1
Remove usage of index_schema_history instead using pg_indexes
vishalhaven Apr 26, 2023
ee72600
Clean up console.log and remove commented code from knex/db.ts
vishalhaven Apr 26, 2023
1e830a8
Resolve conflict
vishalhaven Apr 27, 2023
10a26aa
Clean up all tests
vishalhaven Apr 27, 2023
2fa82ea
Revert unrelated changes
vishalhaven Apr 28, 2023
bdbb95e
Merge master into branch
vishalhaven May 1, 2023
5de80d6
Fix package-lock.json after conflict
vishalhaven May 1, 2023
ed2405b
Fix package-lock.json after conflict
vishalhaven May 1, 2023
a903095
Bumped persistor version
vishalhaven May 4, 2023
38fd99e
Added additional test scenario as per comments
vishalhaven May 4, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
228 changes: 104 additions & 124 deletions components/persistor/lib/knex/db.ts
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 2 additions & 2 deletions components/persistor/package-lock.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion components/persistor/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"name": "@haventech/persistor",
"description": "A subclass of supertype that serializes to and reconstitutes from MongoDB or SQL databases",
"homepage": "https://github.com/haven-life/persistor",
"version": "12.0.1",
"version": "12.1.0",
"main": "dist/index.js",
"types": "dist/index.d.ts",
Copy link
Copy Markdown
Contributor

@kamranHaven kamranHaven Apr 29, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like we have a test case failure (pipeline seems to be failing)

can we make sure to add unit tests around:

  1. no warning message is created if we are deleting an index that is already deleted
  2. no warning message is created if we insert an index that was already created
  3. If a new index is added in schema file, then pg_indexes has that index.
  4. If that same index is deleted from schema file, pg_indexes removes it without impacting any other index
  5. Create invalid index where column doesn't exist
  6. update index (valid and invalid case)
  7. delete index where column is deleted.

Regression:
We would also test regression against some large repo like haven and run their startup.

  1. We would need to test above listed scenarios for a blank db (history table empty to mimic fresh db). In a fresh db, history table wont be created. However the pg_indexes initially without the user added indexes, would get populated with the indexes from schema and the ones created by amorphic.

  2. Then we would also want to test with an existing history table (uat / prod scenario) that this feature, stops incrementing the rows in history table, and not delete history table or its contents. However, any adds, updates and deletes to indexes are correctly reflecting.

Copy link
Copy Markdown
Contributor Author

@vishalhaven vishalhaven May 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@kamranHaven I have covered all the scenarios you mention in the comments.

About the regression:

  1. I have tested with blank db -> works as expected. We need to double the indexes created by the amorphic one.
  2. Not sure how I should test it. I'll check with the PA repo if there are any changes in the schema/indexes/columns and maybe try to run a startup.

"dependencies": {
Expand Down
14 changes: 6 additions & 8 deletions components/persistor/test/persist_banking_pgsql.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -380,7 +380,7 @@ describe('Banking from pgsql Example persist_banking_pgsql', function () {
sandbox.restore();
});

it ('opens the database Postgres', function () {
it('opens the database Postgres', function () {
return Promise.resolve()
.then(function () {
knex = require('knex')({
Expand All @@ -402,13 +402,11 @@ describe('Banking from pgsql Example persist_banking_pgsql', function () {
PersistObjectTemplate.performInjections(); // Normally done by getTemplates
}).catch(function(e) {throw e;});
});
var schemaTable = 'index_schema_history';
it ('clears the bank', function () {

it('clears the bank', function () {
this.timeout(4000);
return knex.schema.dropTableIfExists(schemaTable)
.then(function () {
return clearCollection(Role);
}).then(function (count) {
return clearCollection(Role)
.then(function (count) {
expect(count).to.equal(0);
return clearCollection(Account)
}).then(function (count) {
Expand Down Expand Up @@ -465,7 +463,7 @@ describe('Banking from pgsql Example persist_banking_pgsql', function () {
// });
// });

it ('can create the data', function () {
it('can create the data', function () {
// Setup customers and addresses
sam = new Customer('Sam', 'M', 'Elsamman');
karen = new Customer('Karen', 'M', 'Burke');
Expand Down
3 changes: 0 additions & 3 deletions components/persistor/test/persist_banking_s3.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,9 +81,6 @@ describe('Banking from pgsql Example persist_banking_s3', function () {
PersistObjectTemplate.performInjections(); // Normally done by getTemplates

this.timeout(4000);

const schemaTable = 'index_schema_history';
await knex.schema.dropTableIfExists(schemaTable)
const count = await clearCollection(Customer);
expect(count).to.equal(0);

Expand Down
7 changes: 2 additions & 5 deletions components/persistor/test/persist_fetch.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,6 @@ var knex;


var schema = {};
var schemaTable = 'index_schema_history';
var Phone, Address, Employee, empId, addressId, phoneId, Role, AddressType;
var PersistObjectTemplate, ObjectTemplate;
describe('persist fetch', function () {
Expand Down Expand Up @@ -41,8 +40,7 @@ describe('persist fetch', function () {
return knex.schema.dropTableIfExists('tx_role')
}).then(function () {
return knex.schema.dropTableIfExists('tx_address_type')
}),
knex.schema.dropTableIfExists(schemaTable)]);
})]);
})
after('closes the database', function () {
return knex.destroy();
Expand Down Expand Up @@ -196,8 +194,7 @@ describe('persist fetch', function () {
return knex.schema.dropTableIfExists('tx_role')
}).then(function () {
return knex.schema.dropTableIfExists('tx_address_type')
}),
knex.schema.dropTableIfExists(schemaTable)]);
})])
});

it('check basic fetch without fetch spec should not return the records', function () {
Expand Down
7 changes: 2 additions & 5 deletions components/persistor/test/persist_fetch_children.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ var knexInit = require('knex');
var knex;

var schema = {};
var schemaTable = 'index_schema_history';
var Employee, Department, Role, roleId, EmployeeRef;
var PersistObjectTemplate, ObjectTemplate;
describe('persistor fetch children', function () {
Expand All @@ -36,8 +35,7 @@ describe('persistor fetch children', function () {
return knex.schema.dropTableIfExists('tx_department')
}).then(function () {
return knex.schema.dropTableIfExists('tx_employee_ref')
}),
knex.schema.dropTableIfExists(schemaTable)]);
})])
})
after('closes the database', function () {
return knex.destroy();
Expand Down Expand Up @@ -165,8 +163,7 @@ describe('persistor fetch children', function () {
return knex.schema.dropTableIfExists('tx_role')
}).then(function () {
return knex.schema.dropTableIfExists('tx_employee_ref')
}),
knex.schema.dropTableIfExists(schemaTable)]);
})])
});

it('load intermediate objects first and then try to load the parents ', function () {
Expand Down
12 changes: 4 additions & 8 deletions components/persistor/test/persist_newapi_extend.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,12 @@ var knexInit = require('knex');
var knex;

var schema = {};
var schemaTable = 'index_schema_history';
var Employee, Person, Manager, empId, Address;
var PersistObjectTemplate, ObjectTemplate;

describe('persist newapi extend', function () {
// this.timeout(5000);
before('drop schema table once per test suit', function() {
before('drop schema table once per test suit', async function() {
knex = knexInit({
client: 'pg',
connection: {
Expand All @@ -28,8 +27,7 @@ describe('persist newapi extend', function () {
password: process.env.dbPassword,
}
});
return Promise.all([knex.schema.dropTableIfExists('tx_person'),
knex.schema.dropTableIfExists(schemaTable)]);
return await knex.schema.dropTableIfExists('tx_person')
})
after('closes the database', function () {
return knex.destroy();
Expand Down Expand Up @@ -117,10 +115,8 @@ describe('persist newapi extend', function () {
}
});

afterEach('remove tables and after each test', function() {
return Promise.all([
knex.schema.dropTableIfExists('tx_person'),
knex.schema.dropTableIfExists(schemaTable)]);
afterEach('remove tables and after each test', async function() {
return await knex.schema.dropTableIfExists('tx_person')
});

it('persistorFetchById without fetch spec should not return the records', function () {
Expand Down
7 changes: 2 additions & 5 deletions components/persistor/test/persist_newapi_tests.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,6 @@ var knexInit = require('knex');
var knex;

var schema = {};
var schemaTable = 'index_schema_history';
var Phone, Address, Employee, empId, addressId, phoneId, Role;
var PersistObjectTemplate, ObjectTemplate;

Expand All @@ -39,8 +38,7 @@ describe('persist newapi tests', function () {
return knex.schema.dropTableIfExists('tx_department')
}).then(function () {
return knex.schema.dropTableIfExists('tx_role')
}),
knex.schema.dropTableIfExists(schemaTable)]);
})])
})
after('closes the database', function () {
return knex.destroy();
Expand Down Expand Up @@ -178,8 +176,7 @@ describe('persist newapi tests', function () {
return knex.schema.dropTableIfExists('tx_department')
}).then(function () {
return knex.schema.dropTableIfExists('tx_role')
}),
knex.schema.dropTableIfExists(schemaTable)]);
})])
});

it('persistorFetchById without fetch spec should not return the records', function () {
Expand Down
5 changes: 1 addition & 4 deletions components/persistor/test/persist_parent_subset.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,6 @@ var knexInit = require('knex');
var knex;

var schema = {};
var schemaTable = 'index_schema_history';


describe('persist parent subset', function () {
before('arrange', function (done) {
Expand Down Expand Up @@ -44,8 +42,7 @@ describe('persist parent subset', function () {
}),
knex.schema.dropTableIfExists('tx_employee_parentchild_subset').then(function () {
return knex.schema.dropTableIfExists('tx_employee_parentchild_subset');
}),
knex.schema.dropTableIfExists(schemaTable)
})
]).should.notify(done);
});
after('closes the database', function () {
Expand Down
2 changes: 0 additions & 2 deletions components/persistor/test/persist_polymorphic.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -485,7 +485,6 @@ var schema = {

var knexInit = require('knex');
var knex;
var schemaTable = 'index_schema_history';
describe('type mapping tests for parent/child relations', function () {
before('arrange', function (done) {
knex = knexInit({
Expand Down Expand Up @@ -516,7 +515,6 @@ describe('type mapping tests for parent/child relations', function () {
knex.schema.dropTableIfExists('NewTableWithComments1'),
knex.schema.dropTableIfExists('ExistingTableWithComments'),
knex.schema.dropTableIfExists('ExistingTableWithAField'),
knex.schema.dropTableIfExists(schemaTable)
]).should.notify(done);
})
after('closes the database', function () {
Expand Down
Loading