Support In-Context-Learning (ICL) in agents

[qingyun-wu]

Scenario: Interactive example selection. More specifically, theAssistantAgent can ask for examples anytime during the interaction with the UserProxyAgent

### Tasks
- [x] Review PR https://github.com/microsoft/FLAML/pull/1056
- [x] Write user code example for the above scenario
- [x] Enable exmaple selection (could start with a vanilla example selection method, e.g., random) in `UserProxyAgent`.

### Tasks: June10-13
- [x] Conduct manual initial experiments to verify whether the LLM can request new examples as planned. The hypothesis is that the LLM can use triggers and prompts to evaluate and request examples on its own. We need to determine whether the LLM possesses this capability.
- [ ] Keep in mind and not urgent: We need to consider whether to add the ICL as a "skill" in the AssistantAgent, aligning it with the UserProxyAgent's skills. One possible solution is to include a prompt in the system message when the UserProxyAgent is detected to be working on ICL.

### Tasks: June 14-20
- [x] Prompt Engineering: find the best prompt which can elicit the LLM to think of which kind of exemplars are most useful
- [x] Find more datasets @tong
- [x] Implement the first algorithm (LLM information+ coverage), and have some initial results

### Tasks: June 20-27
- [x] Study the https://arxiv.org/abs/2304.11406 paper, and figure out whether we can use the personalized datasets
- [x] @littlelittlecloud Check the ICL of Math dataset without Voting
- [ ] @tongwu2020 Think about the methods of improving the Math performance
- [x] Manually check the ICL results of the Math dataset
- [x] Ask for the dataset of API bank

### Tasks: June 27
- [ ] @littlelittlecloud @jtongxin #tongwu2020 Design the experiments for Math
- [ ] @jtongxin Study the performance of GPT-4/3.5 on LaMP dataset

[feiran-jia]

I plan to produce a AssistantAgentICL similar to AssistantAgent, corresponding with a ICLAgent who execute the ICL methods inside. The AssistantAgentICL assistant will take a instance of ICLAgent in its constructor. The user will interact with a AssistantAgentICL.

The rough idea can be demonstrated as follows.

user = UserProxyAgent(
    name="user",
    human_input_mode="ALWAYS", # can be automated
    is_termination_msg=lambda x: x.rstrip().endswith("TERMINATE"),
)

icl_agent = ICLAgent(name="ICLAgent",
                    selector=selector,  # selector is the way to select the demonstrations
                    ...,)

# create an AssistantAgent instance named "LLMAssistant"
ICL_SYSTEM_MESSAGE = """You are a helpful AI assistant.
     can provide you some train instances with correct answer. 
    If you can achieve the answer of the test instance with high confident, you say "<answer>, and stop". 
    If you think you need more instances to infer the correct test answer, you can say "<answer>, but I need more examples"
    """
llm_assistant = AssistantAgentICL(name="LlmIclAssistant", 
        icl_assistant=icl_agent ,
        system_message="",
        request_timeout=600, seed=42, config_list=config_list)

To perform the ICL in agents

llm_assistant.receive(test_case_to_solve, user)

[qingyun-wu]

Is the ICL_SYSTEM_MESSAGE a system message for AssistantAgentICL or ICLAgent?

[feiran-jia]

Here is a new design to integrate the ICL into the ProxyAgent.

We first define a function which takes the message (or a list of message) as the input and return a str.

def additional_info(message): # may need history
    return "the content including the exemplars ..."

Then we feed the function and a trigger to the UserProxyAgent.

user = UserProxyAgent(
    name="user",
    human_input_mode="NEVER", # can be automated
    is_termination_msg=lambda x: x.rstrip().endswith("TERMINATE") or x.rstrip().endswith('"TERMINATE".'),
    work_dir="test",
    max_consecutive_auto_reply=10,
    use_docker=False,  # set to True if you are using docker
    augmented_info_provider = additional_info,
    info_trigger = "more_info",
)

Then we build a AssistantAgent with the System Message including the trigger. For example,

ICL_SYSTEM_MESSAGE = """You are a binary classifier.
User will give you one question and two answers. If sol1 is correct, the label is 0. If sol2 is correct, the label is 1.
    I can provide you some train instances with correct answer. 
    If you think more information is helpful, please return "more_info". 
    If you are sure about the answer, return your label (just 0 or 1, Do not explan) and "stop".
    """

LLMAssistant = AssistantAgent(name="LLMAssistant", 
        system_message=ICL_SYSTEM_MESSAGE,
        request_timeout=600, seed=42, config_list=config_list)

LLMAssistant.receive(
    context_str, 
    user,
)

Through early experiments, there are several issues:

• GPT-4 could follow the instruction of system message, but GPT-3.5 sometimes lost.
• The interactive between agents cannot stop until the rounds end.

Other thoughts:

1. We can consider the ICL as a skill and incorporate the relevant prompt within the AssistantAgent itself.
2. Considering that the LLM may not always follow instructions, an alternative approach could be to determine whether to introduce new content by processing the messages in different ways, eliminating the need for triggers.

[feiran-jia]

New Method
I propose utilizing a trigger, i.e. "more_info," to prompt the Language Model (LLM) to request the desired exemplars. The prompt is

<Instruction for the task> + Besides, I can give you some similar questions and their answers to help your reasoning, called exemplars.
Case 1: If you are not confident about your answer, or think having more information could help your reasoning, please (1) tell me what kind of exemplars or information do you need (2) "EXAMPLES"
Case 2: If you are very sure about your answer, please (1) explain and (2) put the answer in \boxed{}

When the LLM say "more_info", we can feed the exemplars, or ask

What kind of exemplars do you like to know? For example, <add task-specific information>?

However, it is worth noting that for certain tasks such as math problem-solving, piqa, and others, the LLM often provides an answer without requesting additional information, even when it is unable to arrive at the correct solution.

Findings
I explored more datasets, and had the following findings.

1. GPT-4 is more likely to know its confidence and request for more_info than GPT-3.5
2. GPT models are inclined to seek more information or exemplars in situations where the dataset falls into the category of OOD/private and there is a lack of necessary information required for reasoning.

I found a dataset in OpenEval which elicit the ideal behavior https://chat.openai.com/share/195c8a66-0734-4977-988d-0dc22fc03cc2. However, this task can be easily solved by asking for the search engine.

ToDo
Now, we urgently require new datasets that meet the following criteria, preferably before Tuesday:

1. The current prompt should be able to prompt the LLM to request the new exemplars.
2. The datasets should be advantageous in terms of providing exemplars.
3. Utilizing search engines like Bing or Google will not yield satisfactory results.

If we cannot find good use case, we need to change the method then.

I would appreciate if anyone could help to think of the potential datasets these days. @qingyun-wu @LittleLittleCloud @sonichi @levscaut @tongwu2020