Conversation
|
loved reading the blog! short, sweet, and impactful! My couple suggestions are only ensure the formatting looks correct in https://holoviz-dev.github.io/blog-dev/posts/holoviz_llm/ for example: "Problem Formulation The first phase"
Maybe try to explain the reason briefly: |
| Creating the skills was only one part of the challenge. A recurring theme throughout the project has been the need for evidence. It is easy to claim that a prompt, skill, or documentation change improves AI-generated code. Demonstrating that improvement in a repeatable way is much harder. | ||
|
|
||
| To address this, we developed an evaluation framework that allows prompts to be run with and without skills enabled. The framework captures outputs, executes the AI-generated code, gathers metrics, and can produce screenshots for visual comparison. | ||
| To address this, we developed an evaluation framework that allows prompts to be run with and without skills enabled. The framework captures outputs, executes the AI-generated code, gathers metrics such as model used, execution time, total tokens consumed etc., and can also produce screenshots for visual comparison. |
There was a problem hiding this comment.
I'd be curious to see the actual results for the sunset app
There was a problem hiding this comment.
Do you mean posting the json output together with this screenshot?
| ## Recent Progress | ||
| Recent work has focused on expanding the skills repository with more package-specific skills, refining the evaluation workflows, adding more evaluation metrics, and comparing behaviour across models. | ||
|
|
||
| We have also begun to adopt `llms.txt` files within the HoloViz repositories starting from [panel-material-ui](https://panel-material-ui.holoviz.org/llms.txt) and working across all the repositories. This is because we observed that some models have a hard time parsing the html content in our websites directly. |
There was a problem hiding this comment.
Perhaps add an explanation: potentially due to notebook conversion + pyodide?
There was a problem hiding this comment.
The issue I have with this is that we don't know why for sure. It's only going to be speculation. I'd rather not guess.
There was a problem hiding this comment.
I'd argue that speculation is still useful so that other users know what to grab onto in their own case; just disclaim it.
2 participants
This PR adds a mid-journey blog post for the holoviz-skills project.