Skip to content

Markdown #7

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sipec wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Markdown #7

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7cd6f57
convert to markdown (WIP)
sipec Apr 8, 2025
5e115cc
wip
sipec Apr 8, 2025
1706275
wip wip
sipec Apr 17, 2025
2e26589
fix some type stuff. migration script for comments
sipec Apr 20, 2025
beb0e66
remove tiptap dependency on backend
sipec Apr 25, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 1 addition & 8 deletions backend/api/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,14 +29,6 @@
"@google-cloud/secret-manager": "4.2.1",
"@react-email/components": "0.0.33",
"@supabase/supabase-js": "2.38.5",
"@tiptap/core": "2.3.2",
"@tiptap/extension-image": "2.3.2",
"@tiptap/extension-link": "2.3.2",
"@tiptap/extension-mention": "2.3.2",
"@tiptap/html": "2.3.2",
"@tiptap/pm": "2.3.2",
"@tiptap/starter-kit": "2.3.2",
"@tiptap/suggestion": "2.3.2",
"colors": "1.4.0",
"cors": "2.8.5",
"dayjs": "1.11.4",
Expand All @@ -62,6 +54,7 @@
"zod": "3.21.4"
},
"devDependencies": {
"@tiptap/core": "2.3.2",
"@types/cors": "2.8.17",
"@types/marked": "4.0.7",
"@types/twilio": "3.19.3",
Expand Down
4 changes: 2 additions & 2 deletions backend/api/src/create-comment.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@ import { getNotificationDestinationsForUser } from 'common/user-notification-pre
import { Notification } from 'common/notifications'
import { insertNotificationToSupabase } from 'shared/supabase/notifications'
import { User } from 'common/user'
import { richTextToString } from 'common/util/parse'
import * as crypto from 'crypto'
import { sendNewEndorsementEmail } from 'email/functions/helpers'
import { type Row } from 'common/supabase/utils'
import { broadcastUpdatedComment } from 'shared/websockets/helpers'
import { convertComment } from 'common/supabase/comment'
import { tiptapToMarkdown } from 'common/util/tiptap-to-markdown'

export const MAX_COMMENT_JSON_LENGTH = 20000

Expand Down Expand Up @@ -49,7 +49,7 @@ export const createComment: APIHandler<'create-comment'> = async (
await createNewCommentOnLoverNotification(
onUser,
creator,
richTextToString(content),
tiptapToMarkdown(content),
comment.id,
Comment on lines 49 to 53
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Mar 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Don’t send Markdown to notification copy.

This value becomes Notification.sourceText, and backend/email/emails/functions/helpers.tsx:1-30 forwards the same string unchanged into the email flow. Using tiptapToMarkdown(content) here will surface raw Markdown like **bold** and # heading in notifications/emails whenever a comment uses formatting. Use a plain-text serializer for notification text.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@backend/api/src/create-comment.ts` around lines 49 - 53, The notification is
being built with Markdown via tiptapToMarkdown(content) in the call to
createNewCommentOnLoverNotification which ends up in Notification.sourceText and
gets sent to emails; replace tiptapToMarkdown(content) with a plain-text
serialization of the tiptap content (e.g., a tiptapToPlainText or plainText
serializer / markdown-stripper) so Notification.sourceText contains only plain
text (no **bold**, # headings, or other markup); update the call site in
createNewCommentOnLoverNotification to pass the plain-text string instead of
tiptapToMarkdown(content).

pg
)
Expand Down
9 changes: 2 additions & 7 deletions backend/api/src/create-private-user-message.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +1,18 @@
import { APIError, APIHandler } from 'api/helpers/endpoint'
import { getUser } from 'shared/utils'
import { createSupabaseDirectClient } from 'shared/supabase/init'
import { MAX_COMMENT_JSON_LENGTH } from 'api/create-comment'
import { createPrivateUserMessageMain } from 'api/junk-drawer/private-messages'

export const createPrivateUserMessage: APIHandler<
'create-private-user-message'
> = async (body, auth) => {
const { content, channelId } = body
if (JSON.stringify(content).length > MAX_COMMENT_JSON_LENGTH) {
throw new APIError(
400,
`Message JSON should be less than ${MAX_COMMENT_JSON_LENGTH}`
)
}

const pg = createSupabaseDirectClient()
const creator = await getUser(auth.uid)
if (!creator) throw new APIError(401, 'Your account was not found')
if (creator.isBannedFromPosting) throw new APIError(403, 'You are banned')

return await createPrivateUserMessageMain(
creator,
channelId,
Expand Down
30 changes: 7 additions & 23 deletions backend/api/src/junk-drawer/private-messages.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,6 @@ import { User } from 'common/user'
import { first } from 'lodash'
import { log } from 'shared/monitoring/log'
import { getPrivateUser, getUser } from 'shared/utils'
import { type JSONContent } from '@tiptap/core'
import { APIError } from 'common/api/utils'
import { broadcast } from 'shared/websockets/server'
import { track } from 'shared/analytics'
Expand All @@ -17,26 +16,11 @@ import timezone from 'dayjs/plugin/timezone'
dayjs.extend(utc)
dayjs.extend(timezone)

export const leaveChatContent = (userName: string) => ({
type: 'doc',
content: [
{
type: 'paragraph',
content: [{ text: `${userName} left the chat`, type: 'text' }],
},
],
})
export const joinChatContent = (userName: string) => {
return {
type: 'doc',
content: [
{
type: 'paragraph',
content: [{ text: `${userName} joined the chat!`, type: 'text' }],
},
],
}
}
export const leaveChatContent = (userName: string) =>
`${userName} left the chat`

export const joinChatContent = (userName: string) =>
`${userName} joined the chat!`

export const insertPrivateMessage = async (
content: Json,
Expand Down Expand Up @@ -82,7 +66,7 @@ export const addUsersToPrivateMessageChannel = async (
export const createPrivateUserMessageMain = async (
creator: User,
channelId: number,
content: JSONContent,
content: string,
pg: SupabaseDirectClient,
visibility: ChatVisibility
) => {
Expand All @@ -101,7 +85,7 @@ export const createPrivateUserMessageMain = async (
await insertPrivateMessage(content, channelId, creator.id, visibility, pg)

const privateMessage = {
content: content as Json,
content: content,
channel_id: channelId,
user_id: creator.id,
}
Expand Down
59 changes: 59 additions & 0 deletions backend/scripts/messages-to-markdown.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
import { runScript } from 'run-script'
import { type Row } from 'common/supabase/utils'
import { tiptapToMarkdown } from 'common/util/tiptap-to-markdown'
import { chunk } from 'lodash'
import { type SupabaseTransaction } from 'shared/supabase/init'
import { pgp } from 'shared/supabase/init'

runScript(async ({ pg }) => {
const messages = await pg.manyOrNone<Row<'private_user_messages'>>(
`select * from private_user_messages`
)

const updates = messages.map((m) => ({
id: m.id,
content: tiptapToMarkdown(m.content as any),
}))

Comment on lines +8 to +17
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 20, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Improve type safety for content conversion.

The script uses as any to bypass type checking when converting content from JSON to Markdown. This could lead to runtime errors if the content structure doesn't match expectations.

- const updates = messages.map((m) => ({
-   id: m.id,
-   content: tiptapToMarkdown(m.content as any),
- }))
+ const updates = messages.map((m) => ({
+   id: m.id,
+   content: typeof m.content === 'object' && m.content !== null
+     ? tiptapToMarkdown(m.content)
+     : String(m.content),
+ }))
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
runScript(async ({ pg }) => {
const messages = await pg.manyOrNone<Row<'private_user_messages'>>(
`select * from private_user_messages`
)
const updates = messages.map((m) => ({
id: m.id,
content: tiptapToMarkdown(m.content as any),
}))
runScript(async ({ pg }) => {
const messages = await pg.manyOrNone<Row<'private_user_messages'>>(
`select * from private_user_messages`
)
const updates = messages.map((m) => ({
id: m.id,
content: typeof m.content === 'object' && m.content !== null
? tiptapToMarkdown(m.content)
: String(m.content),
}))
// …
})

Copy link
Copy Markdown
Owner Author

@sipec sipec Apr 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you're wrong

Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sipec You're right, I apologize for my incorrect suggestion. I misunderstood the context of your migration script. The as any type assertion is appropriate here since this is a one-time migration script where all content is expected to be in Tiptap JSON format. My suggested type checking would have added unnecessary complexity.

await pg.tx(async (tx) => {
await tx.none(
`alter table private_user_messages
drop column content,
add column content text not null default ''`
)

for (const c of chunk(updates, 1000)) {
console.log('updating chunk')
await bulkUpdate(tx, c)
}

await tx.none(
`alter table private_user_messages
alter column content drop default`
)

const values = await tx.many(
`select * from private_user_messages
limit 10`
)

console.log(values)

throw new Error(`don't commit`)
})
Comment on lines +18 to +43
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Apr 20, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Add error handling and consider making the transaction commit optional.

The script:

  1. Deliberately throws an error to prevent committing the transaction
  2. Lacks progress reporting for large message counts
  3. Has no error handling for the conversion process

Consider:

  1. Making the transaction commit configurable via a parameter
  2. Adding error handling and better progress reporting
  3. Providing a more descriptive error message
  await pg.tx(async (tx) => {
    await tx.none(
      `alter table private_user_messages
      drop column content,
      add column content text not null default ''`
    )

+   let processedCount = 0;
    for (const c of chunk(updates, 1000)) {
-     console.log('updating chunk')
+     console.log(`Updating chunk ${processedCount + 1} to ${processedCount + c.length} of ${updates.length}`)
      await bulkUpdate(tx, c)
+     processedCount += c.length;
    }

    await tx.none(
      `alter table private_user_messages
      alter column content drop default`
    )

    const values = await tx.many(
      `select * from private_user_messages
      limit 10`
    )

    console.log(values)

-   throw new Error(`don't commit`)
+   // Default to not committing for safety
+   if (process.env.COMMIT_CHANGES !== 'true') {
+     console.log('Dry run completed. Set COMMIT_CHANGES=true to commit changes.')
+     throw new Error('Migration completed but not committed (dry run)')
+   }
+   console.log('Migration completed and committed successfully.')
  })
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
await pg.tx(async (tx) => {
await tx.none(
`alter table private_user_messages
drop column content,
add column content text not null default ''`
)
for (const c of chunk(updates, 1000)) {
console.log('updating chunk')
await bulkUpdate(tx, c)
}
await tx.none(
`alter table private_user_messages
alter column content drop default`
)
const values = await tx.many(
`select * from private_user_messages
limit 10`
)
console.log(values)
throw new Error(`don't commit`)
})
await pg.tx(async (tx) => {
await tx.none(
`alter table private_user_messages
drop column content,
add column content text not null default ''`
)
let processedCount = 0;
for (const c of chunk(updates, 1000)) {
console.log(`Updating chunk ${processedCount + 1} to ${processedCount + c.length} of ${updates.length}`)
await bulkUpdate(tx, c)
processedCount += c.length;
}
await tx.none(
`alter table private_user_messages
alter column content drop default`
)
const values = await tx.many(
`select * from private_user_messages
limit 10`
)
console.log(values)
// Default to not committing for safety
if (process.env.COMMIT_CHANGES !== 'true') {
console.log('Dry run completed. Set COMMIT_CHANGES=true to commit changes.')
throw new Error('Migration completed but not committed (dry run)')
}
console.log('Migration completed and committed successfully.')
})

})

const bulkUpdate = (
tx: SupabaseTransaction,
updates: { id: number; content: string }[]
) => {
const values = updates
.map((u) => pgp.as.format('($(id), $(content))', u))
.join(',')

return tx.none(
`update private_user_messages p set content = v.content
from (values ${values}) as v(id, content)
where p.id = v.id`
)
}
2 changes: 0 additions & 2 deletions backend/shared/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,6 @@
"@google-cloud/monitoring": "4.0.0",
"@google-cloud/secret-manager": "4.2.1",
"@stdlib/math-base-special-betaincinv": "0.2.1",
"@tiptap/core": "2.3.2",
"@tiptap/html": "2.3.2",
"colors": "1.4.0",
"dayjs": "1.11.4",
"expo-server-sdk": "3.6.0",
Expand Down
22 changes: 22 additions & 0 deletions backend/supabase/migrations/20250415011659_dm_to_markdown.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
-- To preserve existing chat messages, instead run messages-to-markdown.ts
--
-- If this errors content is already no longer jsonb
assert (
select
count(*) > 0
from
private_user_messages
where
content ->> 'type' = 'doc'
),
'No Tiptap messages found';

alter table private_user_messages
drop column content;

alter table private_user_messages
add column content text not null default '';

alter table private_user_messages
alter column content
drop default
8 changes: 1 addition & 7 deletions common/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,19 +12,13 @@
"sideEffects": false,
"dependencies": {
"@supabase/supabase-js": "2.38.5",
"@tiptap/core": "2.3.2",
"@tiptap/extension-image": "2.3.2",
"@tiptap/extension-link": "2.3.2",
"@tiptap/extension-mention": "2.3.2",
"@tiptap/pm": "2.3.2",
"@tiptap/starter-kit": "2.3.2",
"@tiptap/suggestion": "2.3.2",
"dayjs": "1.11.4",
"lodash": "4.17.21",
"string-similarity": "4.0.4",
"zod": "3.21.4"
},
"devDependencies": {
"@tiptap/core": "2.3.2",
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Mar 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Verify whether `@tiptap/core` is imported as a value (non-type) in common.
# Expected: no matches if safe to keep `@tiptap/core` only in devDependencies.

rg -nP --type=ts "import\s+\{(?!\s*type\b)[^}]+\}\s+from\s+['\"]@tiptap/core['\"]" common/src
rg -nP --type=ts "import\s+[^;]*from\s+['\"]@tiptap/core['\"]" common/src -C2

Repository: sipec/polylove

Length of output: 1478

🏁 Script executed:

cat -n common/src/util/parse.ts | head -30

Repository: sipec/polylove

Length of output: 1394

@tiptap/core must remain in dependencies, not devDependencies.

common/src/util/parse.ts imports JSONContent as a non-type import (import { JSONContent } from '@tiptap/core' on line 1), which causes TypeScript to emit a runtime require() statement in the compiled JavaScript even though JSONContent is used only in type annotations (line 28). Moving this to devDependencies will cause downstream packages to fail with MODULE_NOT_FOUND at runtime.

Restore @tiptap/core to dependencies:

Fix 
  "dependencies": {
+   "@tiptap/core": "2.3.2",
    "@supabase/supabase-js": "2.38.5",
    "dayjs": "1.11.4",
    "lodash": "4.17.21",
    "string-similarity": "4.0.4",
    "zod": "3.21.4"
  },
  "devDependencies": {
-   "@tiptap/core": "2.3.2",
    "@types/jest": "29.2.4",
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"@tiptap/core": "2.3.2",
"dependencies": {
"@tiptap/core": "2.3.2",
"@supabase/supabase-js": "2.38.5",
"dayjs": "1.11.4",
"lodash": "4.17.21",
"string-similarity": "4.0.4",
"zod": "3.21.4"
},
"devDependencies": {
"@types/jest": "29.2.4",
🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@common/package.json` at line 21, Restore "@tiptap/core" from devDependencies
back into dependencies in package.json because common/src/util/parse.ts performs
a non-type import (import { JSONContent } from '@tiptap/core') which emits a
runtime require; ensure package.json lists "@tiptap/core" under "dependencies"
so consumers can resolve it at runtime and downstream packages do not get
MODULE_NOT_FOUND.

"@types/jest": "29.2.4",
"@types/lodash": "4.14.178",
"@types/string-similarity": "4.0.0",
Expand Down
3 changes: 2 additions & 1 deletion common/src/api/schema.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ import { DisplayUser, FullUser } from './user-types'
import { PrivateMessageChannel } from 'common/supabase/private-messages'
import { Notification } from 'common/notifications'
import { arrify } from 'common/util/array'
import { MAX_COMMENT_LENGTH } from 'common/comment'

// mqp: very unscientific, just balancing our willingness to accept load
// with user willingness to put up with stale data
Expand Down Expand Up @@ -404,7 +405,7 @@ export const API = (_apiTypeCheck = {
authed: true,
returns: {} as any,
props: z.object({
content: contentSchema,
content: z.string().max(MAX_COMMENT_LENGTH),
channelId: z.number(),
}),
},
Expand Down
2 changes: 1 addition & 1 deletion common/src/chat-message.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ export type ChatMessage = {
id: string
userId: string
channelId: string
content: JSONContent
content: JSONContent | string
createdTime: number
visibility: ChatVisibility
}
Expand Down
55 changes: 3 additions & 52 deletions common/src/util/parse.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,19 +1,7 @@
import {
getText,
getSchema,
getTextSerializersFromSchema,
Node,
JSONContent,
} from '@tiptap/core'
import { Node as ProseMirrorNode } from '@tiptap/pm/model'
import { StarterKit } from '@tiptap/starter-kit'
import { Image } from '@tiptap/extension-image'
import { Link } from '@tiptap/extension-link'
import { Mention } from '@tiptap/extension-mention'
import Iframe from './tiptap-iframe'
import { JSONContent } from '@tiptap/core'
import { find } from 'linkifyjs'
import { uniq } from 'lodash'
import { compareTwoStrings } from 'string-similarity'
import { tiptapToMarkdown } from './tiptap-to-markdown'

/** get first url in text. like "notion.so " -> "http://notion.so"; "notion" -> null */
export function getUrl(text: string) {
Expand All @@ -37,43 +25,6 @@ const checkAgainstQuery = (query: string, corpus: string) =>
export const searchInAny = (query: string, ...fields: string[]) =>
fields.some((field) => checkAgainstQuery(query, field))

/** @return user ids of all \@mentions */
export function parseMentions(data: JSONContent): string[] {
const mentions = data.content?.flatMap(parseMentions) ?? [] //dfs
if (data.type === 'mention' && data.attrs) {
mentions.push(data.attrs.id as string)
}
return uniq(mentions)
}

export const extensions = [
StarterKit,
Link,
Image.extend({ renderText: () => '[image]' }),
Mention, // user @mention
Iframe.extend({
renderText: ({ node }) =>
'[embed]' + node.attrs.src ? `(${node.attrs.src})` : '',
}),
]

const extensionSchema = getSchema(extensions)
const extensionSerializers = getTextSerializersFromSchema(extensionSchema)

export function richTextToString(text?: JSONContent) {
if (!text) return ''
try {
const node = ProseMirrorNode.fromJSON(extensionSchema, text)
return getText(node, {
blockSeparator: '\n\n',
textSerializers: extensionSerializers,
})
} catch (e) {
console.error('error parsing rich text', `"${text}":`, e)
return ''
}
}

export function parseJsonContentToText(content: JSONContent | string) {
return typeof content === 'string' ? content : richTextToString(content)
return typeof content === 'string' ? content : tiptapToMarkdown(content)
Comment on lines 28 to +29
Copy link
Copy Markdown

@coderabbitai coderabbitai Bot Mar 18, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Keep parseJsonContentToText a text serializer.

common/src/util/tiptap-to-markdown.ts:3-6 returns Markdown, not plain text. That silently changes this helper’s contract, so any existing caller that still uses it for previews, search, or other plain-string output will now receive formatting tokens like ** and #. Either restore plain-text serialization here or rename the API and migrate callers.

🤖 Prompt for AI Agents 
Verify each finding against the current code and only fix it if needed.

In `@common/src/util/parse.ts` around lines 28 - 29, parseJsonContentToText
currently returns Markdown because it forwards non-string input to
tiptapToMarkdown, violating its "text serializer" contract; change
parseJsonContentToText to return plain text by either calling a plain-text
serializer (e.g., tiptapToPlainText) if available or by post-processing
tiptapToMarkdown output to strip Markdown formatting (remove headings, emphasis,
code fences, links, etc.) before returning, keeping the function name and
signature so callers get plain strings; if you prefer returning Markdown
instead, rename parseJsonContentToText to something like
parseJsonContentToMarkdown and update all callers to use the new name.

}
Loading