Allow custom prompts

[thealternator89]

Currently the prompts are hardcoded. We should allow users to customize the prompt to get whatever output they want from the AI tool.

Provide substitution handling to allow the custom prompt to be populated as the user wants, e.g.

Generate a set of comprehensive test cases for the following user story/ticket.

Ticket ID: %%Id%%
Title: %%Title%%
Description: %%Description%%
Acceptance Criteria: %%AcceptanceCriteria%%

Additional Context: %%Context%%

Please format the output in a Markdown table, including:
- Test Case ID
- Description
- Pre-conditions
- Steps
- Expected Result
- Priority

Note the above %%...%% substitutions are just a suggestion - we can consider using something like Handlebars or other templating engines, or other substitution characters for handling the substitutions, however consider:

• Whether the substitutions are likely to cause issues with someone writing a prompt (e.g. you could use parentheses, but someone possibly would want to include those literally and it might lead to issues with the prompt - things like $$...$$ or %%...%% are pretty unusual in regular text so we can be pretty safe that they won't be used)
• Whether the tool is an appropriate size for what we need to do
• Ideally don't require regexes to parse the text - but they can be used if they are simple.

Acceptance Criteria:

1. Allow the user to customise the prompt for each tool

2. Keep the current prompts as initial values, and allow the user to reset back to the initial prompts if they break theirs

3. The final part should be fixed on all prompts, as that is designed to ensure GitHub Copilot doesn't do anything unexpected (it either should be fixed based on the tool, or contain substitutions to work for all tools):

   DO NOT create any files. Directly output the test cases in your response here.
   DO NOT include any other text in your response other than the markdown table.
   

   The fixed final part of the prompt should be displayed in the editor screen so the user is aware of it

4. For each prompt, we should display which substitutions are available in the editor, and check if they are used in the user's prompt.

   • Possibly track which substitutions are required or optional, and ensure the required ones are present in the prompt when saving, to ensure the user can't write a faulty prompt
   • Warn the user if they include something that looks like a substitution but we don't recognise it - to catch typos like %%AcceptnceCriteria%%. If AC's aren't required (above point) we will otherwise silently ignore them being missing from the prompt.

[thealternator89]

Actually we will delay this one until #7 is complete, because it will impact how we make this work. Since that will require a very specific prompt it will limit the amount of flexibility we can offer here.

Since we need JSONL format, and need to know what properties will be in the output, we can't leave the names or format to the user.

Instead, we should allow them to customise:

• A general note in the prompt, which is injected between the prompt headline and the context
• The description of what they want in each fixed field
• (For Test Case Writer) the type of format - Cucumber "Given/When/Then", "Preconditions/Steps/Expected Result" etc.

This will allow us to enforce the format required for streaming to work while allowing the user to customise the parts they care about.

We probably should search for keywords that might conflict with our request (like table, markdown, columns, etc.) in the custom general note and alert the user to possible issues. To avoid a conflicting prompt of "Ensure every markdown column has some value [...] Output strictly valid JSONL with a single LF character between output" where the LLM may get confused and produce invalid JSONL.

It's up to them to decide if they want to fix it or risk broken output though.

[thealternator89]

Alternatively, offer a button to check with the Copilot SDK if the prompt conflicts with the built in request - e.g.

Build the whole prompt including the input, but with placeholders for the dynamic ticket/page data, and pass it to the LLM with something like:

The following is a sample prompt requesting JSONL output, is there anything in the prompt which may cause issues conflicting with the JSONL output? If you don't find any issues respond strictly with "No", otherwise respond with markdown bullet points of the potential issues.

> [Prompt]
> [...]

If the LLM responds with "No", just turn the button green with a checkmark (reset the button on change), otherwise display a modal with the response. The user is allowed to save anyway (the "check prompt" button should be optional too), in case either Copilot responds with something slightly different like "No it's fine", or if they are ok with the things it mentions.

Possibly force this check to use a cheap model. While using the selected model is more likely to be accurate in edge cases where one may see a conflict and another may not, however paying 3x quota asking Claude Opus 4.6 whether a prompt is contradictory is probably overkill and will take longer to answer.