Readme: Playground instructions

[usersink]

For many, this request will be absurd, but is it possible to have some guidance on how to integrate EVAL with VSCode, as you did in the example movie files please ie terminal prompt, code added to the open doc, and the docker log files visible? Even a link to an article would be most apprreciated. Does it require an extension?

[hanchchch]

We didn't actually "integrated" it with VSC, instead we just opened the playground directory and put terminal on right of it. About the docker logs, we just did something like docker-compose up -d eval; docker logs eval . I think it doesn't seem appropriate to include guidance about this on README, bc it's more related to docker usage than this project. But if you have any questions about those, please feel free to ask!

I really do agree that there should be an explanation about the playground dir. Initially we made it to prevent EVAL from editing or deleting the code of itself and apparently we forgot to explain about this on the document. Thanks for sharing your opinion!

[usersink]

I completely understand the docker instructions but using it with vscode could be included as steps under 'usage'. I'm not familiar with it, and have received some strange output after trying it for the first time.

Log output:

`Entering new chain.
Prompted Text: Orca, can you build me a streamlit application to facilitate our conversations. Tell me what you need for the task.

Plan:

• Create a new file in the ./playground/ directory with a name of your choice and a .py extension.
• Write the code for the Streamlit application in the file.
• Save the file.
• If the code is completed, use the Terminal tool to execute it, if not, append the code through the CodeEditor.APPEND tool.
  What I Did: None
  Action: CodeEditor.WRITE
  Input:
  ... ... ./playground/conversation_facilitator.py ... ... import streamlit as st ... ... def main(): ... st.title("Conversation Facilitator") ... st.write("Welcome to our conversation facilitator! How can I assist you today?") ... ... if __name__ == "__main__": ... main() ...
  Observation: Last 3 line was:
  ... You can't access file outside of playground.
  Thinking...
  Plan:
• Create a new file in the ./playground/ directory with a name of your choice and a .py extension.
• Write the code for the Streamlit application in the file.
• Save the file.
• If the code is completed, use the Terminal tool to execute it, if not, append the code through the CodeEditor.APPEND tool.
  What I Did: I asked you to create a new file in the ./playground/ directory with a name of your choice and a .py extension. Then, I provided you with the code for the Streamlit application to facilitate our conversations.
  Action: Final Answer
  Input: d1dy0uth1nk7hat1t1s7haAAat3aSy?
  Final Answer: d1dy0uth1nk7hat1t1s7haAAat3aSy?
  Finished chain.`

That's after running the code in a folder titled 'playground'. Since that response, the service seems to be unavailable, as the logs produce no further outputs, despite further queries posted. I'm keen to try the project but I'm not sure how to reproduce the outputs from the example videos. Any help would be greatly appreciated -- I love the concept.

[hanchchch]

So there's a few things that should be worked on to resolve this issue.

1. CodeEditor.WRITE instruction
   • We at first tried to make it write codes only in playground dir by adding an instruction about it. https://github.com/corca-ai/EVAL/blob/main/core/tools/editor/__init__.py#L77
   • But then we realized that we could've just change cwd into the playground. https://github.com/corca-ai/EVAL/blob/main/api/container.py#L19
   • We forgot to remove the original instruction after changing cwd.
     • update: there was a PR fix: more specific CodeEditor.WRITE description prompt #16 about this already. I'll work on it as soon as possible.
2. You can't access file outside of playground
   • This is intended, to prevent EVAL from writing outside of playground dir.
   • But according to your logs, apparently it didn't try to do so.
   • So there might be a problem on core/tools/editor/verify.py
   • We'll take a look on it.
3. Strange answer
   • We did a lot of experiments, and input prompts are kinda messed up a bit.
   • We should clean and re-organize it
4. Service being unavailable
   • I need some details for this issue.
   • which endpoint did you use? /api/execute or /api/execute/async? or did you just use a web UI?

[hanchchch]

using it with vscode could be included as steps under 'usage'

I'm very sorry but I afraid that I don't really get it. What do you mean by 'using it with vscode'? Like watching it generating the code right away on the vscode?

[hanchchch]

By the way, thanks very much for having interest on this project despite of many issues. I really appreciate it. People like you makes this project better.

[usersink]

Thanks so much for the quick replies, explanations and updates, and the assistance. It is very much appreciated.

You can't access file outside of playground
This is intended, to prevent EVAL from writing outside of playground dir.
But according to your logs, apparently it didn't try to do so.
So there might be a problem on core/tools/editor/verify.py
We'll take a look on it.

If it makes any difference, I'm using VSCodium on linux, rather than VSCode.

Service being unavailable
I need some details for this issue.
which endpoint did you use? /api/execute or /api/execute/async? or did you just use a web UI?

I'd been using /command the entire time. I've now tried the other endpoints but nothing seems to work at this stage. Even after restarting the container. The logs now only record visits to localhost:port via the browser, and curl posts are seemingly ignored, as they produce no output in the logs.

I'm very sorry but I afraid that I don't really get it. What do you mean by 'using it with vscode'?

Just setting up the IDE to create code outputs in the way you'd visualised, and allowing EVAL to perform actions in that environment. I'm sure most won't need the explanation, but as a non-dev, semi-technical person, it's entirely unfamiliar to me.

[adldotori]

Did you retry this with our new GUI? If you run, you can access our UI by entering localhost:8000. If there is an error in the UI, please report it again. By the way, Thank you so much for your interest in our repo!!

[usersink]

Sorry for the slow reply, and thank you for the quick update!

The GUI seems to work well, but it doesn't recognise the playground volume in the container, so it's still not writing the code. When it's unable to write to the playground it seems to output the 'l33t speak' response I'd previously shared -- same as the CLI interface.

However, I've not mounted the playground volume to a host directory, so I'll try that later today and see whether that's the issue.

[usersink]

Later than planned but I've tested it, and can confirm that it only works when mapped to a host directory.

EDIT:

1. By the way, does the folder have to be ~/playground? I've got it working with that convention, but it's worth clarifying for anyone wanting to use something like ~/git/playground.

2. Also worth noting that it will only write .py files to the playground, and fails under any other condition. I'd requested that it write the output of some bash alias commands to a .py file in the playground and it errored.

[adldotori]

Sorry for the slow reply, and thank you for the quick update!
The GUI seems to work well, but it doesn't recognise the playground volume in the container, so it's still not writing the code. When it's unable to write to the playground it seems to output the 'l33t speak' response I'd previously shared -- same as the CLI interface.
However, I've not mounted the playground volume to a host directory, so I'll try that later today and see whether that's the issue.

da39de2 commit solves that issue!

1. By the way, does the folder have to be ~/playground? I've got it working with that convention, but it's worth clarifying for anyone wanting to use something like ~/git/playground.

If you want, you must be set PLAYGROUND_DIR in .env, volumes in docker-compose.yml. Then it will works!

2. Also worth noting that it will only write .py files to the playground, and fails under any other condition. I'd requested that it write the output of some bash alias commands to a .py file in the playground and it errored.

Could you share your example?