Help needed to reproduce paper's experiments

[adalrsjr1]

Hello, @MikukuOvO I'm PhD in autonomic computing and runtime adaptation and I got super interested in your work. I read the paper "The Vision of Autonomic Computing: Can LLMs Make It a Reality?" and watched the video. I tried to experiment with the LLM-based multi-agents. However, I couldn't reproduce the experiments, the outcomes are quite different from the paper and video.

I tried to run auto-kube against my local LLM (Ollama -- deepseekr1-1.4 and llama3.2) and it didn't worked, it is missing cloudgpt_aoai.py. Then, I tried the code from paper_artifact_arXiv_2407_14402 and this one seems to work.

I tried a few examples as in the README:

• on working_mechanism_1: report CPU from your component --component catalogue, scale your component to three replicas --component catalogue,
• on working_mechanism_2: Reduce the total P99 latency of catalogue and front-end to under 400 ms --components catalogue,front-end

In all cases, I observed the agents reasoning and trying many different things, but they never converge to what I've asked. For example, when I request for working_mechanism_1 report CPU from your component --component catalogue, the agent diverge after a dozen of iterations and start trying to fetch the response time of the service instead of CPU. Stopping after 50 iterations with no results.

I've tried different models (deepseek-r1:32b, deepseek-r1:14b, llama3.2, etc), and in all cases the outcome diverges from the original task. I'm not sure if it is an issue with the models that are very generic or the setup prompts that need to be fine tuned.

Could you provide some guidance, am I missing anything?

Besides GPT, do you have any suggestion about which other model could perform better? -- I don't have access to GPT model.

In regarding to the prompts, have you tried variations on them? Do you have any hint on how could I customize them to achieve better results?

Thx!

[MikukuOvO]

Hi @adalrsjr1,

Thanks for your interest in our work! Here are some additional details that might be useful for reproducing the results in the paper artifact and video.

AutoKube:
cloudgpt_aoai.py is an API calling file to generate responses with the LLM. Only get_chat_completion is used for AutoKube. You'll need to modify this with the API calling function of the model you deployed. Additionally, you might also need to adjust the line that extracts the response.

Paper Artifact:
We’ve tried deepseek-r1, deepseek-r1-14b (distilled from Qwen), as well as llama-8b. Unfortunately, all of them did not work as expected. This might be due to the reasoning model outputting all the information in a single step, as well as the model’s capabilities. However, we later tried qwen-max provided by Aliyun, and it worked well (it integrates with the OpenAI key, so it's easy to switch). Additionally, reducing the last_n_messages in the service.py file to 1 might help the agents work more stably.
We will continue trying other open-source models, some of which might require prompt modifications. If you encounter any problems reproducing the results or applying them to other microservices and models, feel free to contact us!

Best regards,
ACV team

[adalrsjr1]

Thanks! After following your suggestion, I was able to run AutoKube correctly. I have a few follow-up questions:

1. I implemented my get_chat_completion(engine, messages) by comparing the Ollama API and OpenAI API for chat. However, I'm ignoring the engine argument since AutoKube always uses engine="gpt-4o-20240513". Do you have any comments on this?

2. When parsing the response, I split it into thinking (text within tags) and answer (text after ). I only return the answer part, discarding thinking. Is this correct, or should I return the whole response?

content = message.split('</think>')
answer = ''
if len(content) > 1:
    answer = content[1]
return answer

3. I've tried different open-source models, but none could reproduce the experiments in the video with 100% accuracy (as you pointed out before). Currently, Deepseek-r1-14b (distilled from Qwen) performed best. Is this aligned with your experiments? Should I try other Deepseek models or optimize the prompts for AutoKube? I noticed AutoKube sometimes digresses due to insufficient prompt information for the next step, e.g., it decides to deploy a dummy service on k8s instead of inspect the actual sock-shop services.

[MikukuOvO]

Hi @adalrsjr1,

Thanks for your reply! Here are some comments that might be useful for your questions:

1. AutoKube sometimes uses a fixed engine because, for some low-difficulty tasks, we simply implemented a naive LLM chat completion. Please change it to use the get_chat_completion you implemented.
2. That's correct—we don't have access to the reasoning model when developing AutoKube, so returning only the answer part works for AutoKube.
3. Due to time and effort constraints, we have previously only tried the GPT series model, and it did struggle with some difficult tasks. Based on the circumstances you reported, perhaps you can try a model with larger parameters, or it may work if you optimize the prompt. We are considering adding some task examples into the prompt, which might work with distilled models.

Let us know if you have any further questions!

Best regards,
ACV team