Section 1.1: Direct vs. Indirect Reasoning

Large Language Models (LLMs) have displayed remarkable proficiency in tackling complex tasks, including language understanding, logical deduction, and mathematical reasoning. Their success is even more impressive considering they often operate in zero-shot or few-shot contexts, showcasing their ability to learn from the surrounding context.

This capability has led various researchers to explore methods to enhance the performance of these models, resulting in techniques like Chain-of-Thought (CoT) and others.

CoT encourages models to articulate the series of intermediate steps that lead to a final answer. The premise is that revealing each step allows the model to arrive at the correct solution, while hastily jumping to conclusions often leads to errors.

CoT and similar techniques adhere to what is known as the Direct Reasoning (DR) framework, which constructs logical sequences from established facts to a conclusion. However, this approach has limitations, as not all problems can be addressed in such a manner. This raises the question: when faced with an unsolvable problem, can we leverage Indirect Reasoning (IR)?

Section 1.2: Understanding Indirect Reasoning

Indirect Reasoning (IR) serves as a complementary strategy for problem-solving. A widely utilized method is to employ logical techniques to demonstrate the equivalence of two propositions. For instance, one can validate a statement by assuming its negation and arriving at a contradiction: if p leads to q, the contrapositive ¬q leads to ¬p. By showing that ¬q leads to ¬p results in a contradiction, we can infer the truth of the original proposition.

7. 8. Hardy aptly referred to proof by contradiction as "one of a mathematician's finest weapons," noting its superiority over chess gambits: "A chess player may sacrifice a pawn or even a piece, but a mathematician offers the game."

Indirect reasoning has been effectively utilized by humans and could also be harnessed by models to tackle challenges where direct reasoning techniques falter. How can we enable LLMs to capitalize on IR?

Section 2.1: Adapting Indirect Reasoning for LLMs

The authors divide the adaptation process into two key phases:

1. Rule Augmentation: In this phase, the model is guided to expand its set of rules.
2. Indirect Reasoning: Once the model receives facts, rules, and questions, it engages in indirect reasoning.

To facilitate the use of IR with an LLM, the authors define templates for both zero-shot and few-shot prompting.

The assessment of reasoning performance for any method hinges on the accuracy of answer A and the reasoning process P. Three metrics are employed: Answer Accuracy (AA), Reasoning Process Accuracy (AP), and Overall Accuracy (OA).

The authors quantitatively define these metrics based on the count of examples yielding correct answers, correct reasoning processes, and both.

To observe the impact of indirect reasoning, they test their approach with both Chain-of-Thought and self-consistency in models such as GPT-3.5 and Gemini, utilizing datasets that include natural language questions (ProofWriter) and mathematical problems (ProofMath) where proof by contradiction is essential. The findings indicate that IR enhances the model's capacity to solve these types of challenges.

Additionally, rule augmentation proves beneficial even when employing DR alone, streamlining the path to conclusions, thus accelerating the reasoning process.

This video features a discussion with Yann LeCun and Lex Fridman on whether LLMs can reason effectively, shedding light on the capabilities and limitations of these models.

In this video, explore five straightforward methods to enhance LLM reasoning, providing practical insights for leveraging indirect reasoning techniques.