The down half of a migration is not applied: golang-migrate down files and goose, sql-migrate and dbmate down sections

[avison9]

What it changes

cdclint applied every *.sql file in name order, and every statement in each file. A forward migrate never runs a migration's down half, so the schema cdclint read could differ from the one the connector sees. Now the down half is left out:

• golang-migrate: {version}_{title}.down.sql files are skipped.
• goose (-- +goose Up / -- +goose Down), sql-migrate (-- +migrate Up / -- +migrate Down), dbmate (-- migrate:up / -- migrate:down): the down section of a file is blanked line for line before the file is split, so every remaining statement keeps its line number. Options after a marker (notransaction, transaction:false) are allowed. Markers checked against each tool's README.

How bad it was, measured rather than assumed

• goose, sql-migrate, dbmate: every migration undone. The down section comes after the up one in the same file, so a table was created and dropped again. goose-down-section on main:

  error topic-table-mapping goose-down-section/sink/0001_orders.sql:1
  

  for a table that exists. Every goose repository with down sections read as an empty schema.
• golang-migrate: occasional, and real. A down file sorts just before its own up file (.down < .up), so it usually undoes something that has not happened yet and does nothing. Not always: Mattermost v10.11.0's 000092_add_createat_to_teammembers.down.sql drops CreateAt from Reactions instead of TeamMembers, and reading it deleted a real column. On Mattermost's 140 MySQL migrations, 1 of 71 tables came out different with the down files read. golang-migrate-down-files on main:

  warning sink-column-unknown golang-migrate-down-files/sink/0001_reactions.sql:5
  

I had earlier described the golang-migrate case as "any golang-migrate repository reads wrong". That was too strong; the measurement above is the accurate version.

How

A new internal/migrate package: Down(path) for golang-migrate files, which postgres.ReadFiles skips (one choke point, so the working tree and the --base read agree), and Up(text), which postgres.Apply runs before splitting. The MySQL reader in #16 will call the same two when it is rebased.

Verified

• internal/migrate tests: golang-migrate names (case, backslash paths, countdown.sql is not a down file), each tool's markers with options, down then up again (dbmate), CRLF line endings, a column named down, text with no markers unchanged.
• Corpus: golang-migrate-down-files (reduced from Mattermost's 000092) and goose-down-section. Both fail on main with the findings above and pass here; checked by running them against main's reader. Every existing entry unchanged.
• gofmt -l . clean, go vet, go test ./... pass.
• RefuseRadar (plain numbered migrations, no down halves): 0 error(s), 0 warning(s), 159 info, unchanged.

Not included

Flyway undo migrations (U<version>__*.sql): a paid Flyway feature, and I did not verify the naming from Flyway's documentation.

Merging

Independent of #13, #14, #15. #16 (MySQL) also touches file reading; I will add the two calls to the MySQL reader when rebasing it.