feat: Implement AI Wizard for experiment generation and enhance exper…#93
feat: Implement AI Wizard for experiment generation and enhance exper…#93yashpatil641 wants to merge 3 commits intomainfrom
Conversation
sidravi1
left a comment
sidravi1
left a comment
There was a problem hiding this comment.
Thanks @yashpatil641. Left a few comments.
I think this repo is going to become obsolete very soon. It'd be good to have you work on the evidential repos instead.
Also, big PRs are very hard to review. Think about how you can break them up into smaller pieces (e.g. backend only, front end but not connected, integration etc.) to make it easier for the reviewer. You can also use feature branches to make it easier.
backend/app/ai_helpers/routers.py
Outdated
| from typing import Any, Optional | ||
|
|
||
| from fastapi import APIRouter, HTTPException | ||
| from google import genai |
There was a problem hiding this comment.
We use Litellm wherever possible to remain foundation model agnostic
backend/app/ai_helpers/routers.py
Outdated
| router = APIRouter(prefix="/ai_helpers", tags=["AI Helpers"]) | ||
|
|
||
|
|
||
| class ExperimentAIGenerateRequest(BaseModel): |
There was a problem hiding this comment.
You should move these to a schemas.py file. See other routers
There was a problem hiding this comment.
Thanks for all the feedback! I've just pushed a new commit that should address this
backend/app/ai_helpers/routers.py
Outdated
|
|
||
| goal: str | ||
| outcome: str | ||
| num_variants: int |
backend/app/ai_helpers/routers.py
Outdated
|
|
||
| name: str | ||
| description: str | ||
| methodType: str |
There was a problem hiding this comment.
Maybe an enumeration for this?
There was a problem hiding this comment.
Thanks for all the feedback! I've just pushed a new commit that should address this
backend/app/ai_helpers/routers.py
Outdated
|
|
||
| api_key = os.environ.get("GEMINI_API_KEY") | ||
| if not api_key: | ||
| raise HTTPException(status_code=500, detail="Gemini API key not set") |
There was a problem hiding this comment.
This code with run at load time before fastapi app is running so should be a different exception - not HTTPException.
There was a problem hiding this comment.
Thanks for all the feedback! I've just pushed a new commit that should address this
backend/app/ai_helpers/routers.py
Outdated
| ) -> list[dict[str, Any]]: | ||
| """Suggest arms for Bayesian A/B test.""" | ||
| # System instruction for arms suggestion | ||
| system_instruction = ( |
There was a problem hiding this comment.
See https://github.com/IDinsight/ask-a-question for how to move this our to a different file where we can manage prompts. Also, can i suggest using """ for multiline strings and textwrap.dedent?
There was a problem hiding this comment.
Thanks for all the feedback! I've just pushed a new commit that should address this
backend/app/ai_helpers/routers.py
Outdated
| "No explanation, no markdown." | ||
| ) | ||
|
|
||
| user_prompt = ( |
There was a problem hiding this comment.
Why not use JSON? and then use pydantic to dump JSON
There was a problem hiding this comment.
Thanks for all the feedback! I've just pushed a new commit that should address this
backend/app/ai_helpers/routers.py
Outdated
| ), | ||
| ) | ||
| return json.loads(response.text) | ||
| except Exception as e: |
There was a problem hiding this comment.
Don't catch base exceptions! What exception are you expecting here? Let's just catch those
|
|
||
| component: AddNotifications | ||
| }, | ||
| { |
There was a problem hiding this comment.
Do you use prettier to format typescript code? Can i suggest using it?
|
@yashpatil641 - Suggest not doing more work on this since this repo is obsolete. All work should be on the evidential repo from now on |
2 participants
Reviewer: @poornimaramesh @Jayprakash-SE @sidravi1
Estimate:
Ticket
Implement AI Wizard for experiment generation
Description
Goal
This pull request introduces the AI Experiment Wizard, a new feature designed to simplify the experiment creation process. The goal is to empower non-technical platform owners to design and launch robust experiments by answering a few high-level questions, aligning with our core design principle of making the Experiments Engine simple and accessible.
Changes
/experiments/add/ai-wizardwhere users can input their experimentgoal,outcome, and the desirednumber of variants.backend/app/ai_helpers/routers.pythat use the Gemini API to generate a complete experiment configuration, including a suggested name, description, experiment type, arms, and contexts.useExperimentStoreto handle the state of the AI Wizard. The generated experiment details are used to pre-populate the standard experiment creation form for user review./frontend/src/app/(protected)/experiments/add/components/summary.tsx) now displays the AI-generated goal and outcome, providing better context for the experiment's purpose.Future Tasks (optional)
How has this been tested?
To-do before merge (optional)
Checklist
Fill with
xfor completed.Demo Video:
rec2.mp4