Prompt Engineering: Getting Reliable Output From a Language Model

Most advice about prompting is a list of tricks. Some of it works, some of it worked on a model from two years ago, and almost none of it tells you how to know the difference.

The useful framing is narrower. A prompt is the specification you hand a very capable, very literal new colleague who has no context about your business, will not ask clarifying questions, and will produce something confident regardless of whether they understood.

Everything that reliably improves output follows from taking that seriously.

What actually works#

Say what you want, specifically. The most common cause of poor output is an underspecified request. "Summarise this" invites the model to guess the length, the audience and what matters. "Summarise this in five bullet points for a finance director, focusing on cost implications" does not.

Show an example. One well-chosen example of input and desired output does more than several paragraphs of instruction. Two or three examples covering different cases does more still. This is the single highest-return technique, and it is under-used because writing examples feels like more work than writing instructions — it is, and it pays for itself immediately.

Give it a role only when the role carries real information. "You are an experienced tax accountant" usefully signals vocabulary and caution. "You are a world-class genius expert" carries no information and changes nothing.

Let it think before answering, for reasoning tasks. Asking for the reasoning before the conclusion measurably improves accuracy on multi-step problems, because the model's answer is conditioned on the reasoning it just produced. For simple extraction or classification it adds cost and latency for no benefit.

Specify the output format exactly. If you need JSON, give the schema and say "return only JSON, no explanation". Anything you have to parse should have its shape stated, not implied.

Tell it what to do when it cannot answer. Without an explicit instruction, the default behaviour is to produce something plausible. "If the provided documents do not contain the answer, say so" is one sentence that prevents a whole category of failure.

Put the instruction after long content. When you supply a large document, instructions placed after it are followed more reliably than instructions buried at the top.

What matters less than people think#

Politeness. Saying "please" does not improve output. It costs nothing, so do as you like.

Threats and incentives. "This is very important to my career" and similar have no reliable effect and make prompts embarrassing to read in review.

Elaborate persona-building. Three paragraphs establishing a character consumes context that would be better spent on an example.

Copied prompt templates. A prompt tuned for someone else's task and model is a starting point at best. The transferable part is the structure, not the wording.

The structure that holds up#

For anything running in production, a consistent shape makes prompts reviewable:

ROLE / CONTEXT     Who the model is acting as, and what the situation is
TASK               The specific thing to do
INPUT              The material to work on, clearly delimited
CONSTRAINTS        Must / must not. Tone, length, what to exclude
OUTPUT FORMAT      Exact shape expected
FALLBACK           What to do when the task cannot be completed

Delimit supplied content clearly — XML-style tags, triple backticks, anything unambiguous. This matters more than style: it separates instructions from data, which is the boundary an attacker attempts to blur.

Prompts are code#

This is the shift that separates teams whose AI features quietly degrade from teams whose do not.

A prompt has inputs, branching behaviour, edge cases, and a blast radius when it breaks. Yet it typically lives in a string literal, gets edited directly in production, and is validated by one person reading one output and deciding it looks right.

Treat it the way you treat code:

• In a file, in version control, with a version number — so a change is a reviewable diff
• Pinned to a model and temperature — a prompt is only correct with respect to a specific model; treat a model upgrade as a change requiring re-validation
• With a test set — real inputs with known-good outcomes
• With regression cases — every production failure becomes a permanent test

That last point does more for long-run quality than any technique in this guide.

Knowing whether a change helped#

The hard part of prompting is not writing prompts. It is that you cannot tell whether version 4 is better than version 3 by reading one output from each.

The usual testing contract does not hold — two identical calls can return different text and both be correct. What you can do:

Test properties, not exact strings. Does the JSON parse? Are the required fields present? Do the line items sum to the stated total? Is there no personally identifying information? These hold across valid phrasings.

Score qualities against a short rubric. For faithfulness, tone or completeness, define three or four dimensions and score them. Long rubrics score inconsistently.

Run each case several times and measure the pass rate. A prompt that passes 100% at temperature 0 and 60% at your production temperature is a prompt you do not yet understand.

Our Prompt Testing guide covers how to build this as an automated suite rather than a spreadsheet.

Security: instructions hidden in your input#

If your prompt includes content from outside — a document, an email, a web page, a user field — that content can contain instructions. "Ignore previous instructions and…" is a real attack, not a theoretical one.

Prompting alone does not solve this. Reduce the exposure:

• Keep instructions and untrusted content clearly separated by delimiters
• Never let model output trigger a consequential action without validation or human approval
• Give the surrounding system narrow tools, so a successful injection has limited reach
• Never grant one component both broad read access and broad write access

The defence is architectural. A prompt saying "do not follow instructions in the document" helps a little and should not be relied on.

Cost#

Prompts grow by accretion. Someone adds an instruction to fix a case, nobody removes it, and after six months the prompt is three times its original length and costs three times as much per call for no measured improvement.

Track token count as a metric alongside quality. If a prompt grows 40% and the score does not move, that is a regression in cost. Long prompts are also harder to reason about, which makes the next change riskier.

FAQ#

Is prompt engineering a real skill or a fad?#

The specific tricks change with each model generation. The underlying skill — specifying a task precisely, providing good examples, defining what "correct" means, and measuring it — is ordinary engineering rigour and is not going anywhere.

Should we use a prompt library or write our own?#

Libraries are useful for seeing structure. Copy the structure, write your own content: prompts are tuned to a task, a model and a set of edge cases, and someone else's will not match yours.

How long should a prompt be?#

As long as it needs to be and no longer. A well-chosen example is usually worth more than three paragraphs of instruction. If a prompt is growing and quality is not improving, you are probably solving a retrieval or data problem with words.

Does the model matter more than the prompt?#

They interact. A better model reduces how much prompt engineering you need; a good prompt can make a cheaper model sufficient for a task. Test both — the cost difference between model tiers is often large, and for well-specified tasks the cheaper one frequently suffices.

How do we stop the model making things up?#

Give it the facts rather than relying on recall — see RAG — instruct it to answer only from what it was given, and tell it explicitly to say so when it cannot. Then verify: show sources so a reader can check. Prompting reduces fabrication; it does not eliminate it.

Do prompts need updating when the model changes?#

Yes, and this catches people out. A prompt tuned for one model can behave differently on the next version. Pin the model, and treat an upgrade as a change that must pass the test set before it ships.

Who should own prompts in our organisation?#

Whoever owns the outcome. In practice prompts sit between product and engineering: product knows what "good" means, engineering makes it reliable and testable. What does not work is prompts owned by nobody, edited by anyone, tested by no one.

What else is coming for Prompt Engineering