A Regression Framework for Understanding Prompt Component Impact on LLM Performance

Prompt components significantly influence LLM performance: Regression models analyzing individual prompt features explained 72-77% of the variation in how well two open-source models (Mistral-7B and GPT-OSS-20B) performed on arithmetic tasks, demonstrating that prompt design is a critical factor in model outcomes.

Incorrect examples actively harm model performance: The study found that including false example query-answer pairs in prompts substantially impeded both models from solving arithmetic problems, suggesting that data quality in prompts matters as much as quantity.

Positive instructions show inconsistent effects: Interestingly, positive instructions did not produce consistent improvements across models, and contradictory instructions had unpredictable effects on performance—highlighting that prompt engineering is not straightforward and requires careful testing.

Statistical analysis can reveal how LLMs use prompts: This regression framework provides a practical tool for organizations to understand exactly which components of a prompt influence model behavior, enabling more informed decisions about prompt design in high-stakes applications.

Transparency into model decision-making is achievable: By breaking down prompts into measurable components and analyzing their individual impact, this approach contributes to making LLM behavior more interpretable and auditable—important for responsible deployment in critical scenarios.