Backward Reasoning in LLMs: A Strategy for Identifying Irrelevant Context in Mathematical Word Problems

dc.contributor.authorKumaran, Aishwaryaen
dc.contributor.committeechairRamakrishnan, Narendranen
dc.contributor.committeememberWang, Xuanen
dc.contributor.committeememberHuang, Lifuen
dc.contributor.departmentComputer Science and#38; Applicationsen
dc.date.accessioned2025-06-04T08:01:45Zen
dc.date.available2025-06-04T08:01:45Zen
dc.date.issued2025-06-03en
dc.description.abstractWe investigate backward reasoning in LLMs for identifying irrelevant context in mathematical word problems. Our evaluation across models from 1B to 70B parameters expose a sizeable performance gap between forward and backward reasoning: accuracy on the latter lags by 3-47%, with parameter-constrained models showing the largest drops. To mitigate this gap we propose five structured prompting methods: semantic-role abstraction, timeline reasoning, tree representations, contrastive prompting, and a unified process-supervision prompt. The unified scheme yields the most reliable gains, boosting backward-reasoning accuracy by almost 20%. In addition, simply rephrasing problems in plain language lifts performance by roughly 8 % across the board, underscoring the importance of task formulation. Strikingly, models can identify irrelevant facts with about 75 % accuracy when asked directly, yet fail to exploit this latent knowledge during problem solving, with accuracy plum- meting to 20% once distractors are embedded in the input. Leveraging this insight, we introduce a consistency-based variable-verification framework that activates hidden knowledge through backward reasoning, filters spurious context, and markedly strengthens robustness in realistic settings where misleading details are common.en
dc.description.abstractgeneralWe explore how AI language models solve math word problems, especially when the question includes extra, irrelevant details. We found that while these models are generally good at solving problems when given all the facts in the right order, they struggle much more when they have to work backwards from the final answer to an unknown starting value or when some information is unnecessary. This struggle is worse for smaller models, but even the large models show a clear gap. To help with this, we tested five different ways of guiding the models through the problem more clearly such as using structured language, breaking down the steps, or showing both wrong and right ways to solve a problem. The most effective method gave nearly a 20% improvement in accuracy. Surprisingly, just rewriting the questions in in a better format improved results by 8%, showing how much the wording matters. We also discovered that these models can often tell which facts are unimportant if asked directly. However, when those same facts are mixed into a real problem, the models tend to get confused. To fix this, we designed a method that helps the models double check their reasoning and ignore misleading information. This made them much more reliable at solving real world style problems where not everything in the question is useful.en
dc.description.degreeMaster of Scienceen
dc.format.mediumETDen
dc.identifier.othervt_gsexam:44237en
dc.identifier.urihttps://hdl.handle.net/10919/135022en
dc.language.isoenen
dc.publisherVirginia Techen
dc.rightsIn Copyrighten
dc.rights.urihttp://rightsstatements.org/vocab/InC/1.0/en
dc.subjectLarge Language Modelen
dc.subjectNatural Language Processingen
dc.subjectStructured Promptingen
dc.subjectMathematical Reasoningen
dc.titleBackward Reasoning in LLMs: A Strategy for Identifying Irrelevant Context in Mathematical Word Problemsen
dc.typeThesisen
thesis.degree.disciplineComputer Science & Applicationsen
thesis.degree.grantorVirginia Polytechnic Institute and State Universityen
thesis.degree.levelmastersen
thesis.degree.nameMaster of Scienceen

Files

Original bundle
Now showing 1 - 1 of 1
Name:
Kumaran_A_T_2025.pdf
Size:
1.52 MB
Format:
Adobe Portable Document Format

Collections