What Are Large Language Models (LLMs)? Use Cases in Software Development and Example Projects

Over the past three years, large language models have found their way into almost every client project I touch. Sometimes it is a support bot. In other cases, it is an internal tool that drafts product copy. Sometimes it is an assistant that searches thousands of PDFs. I have worked on web and digital marketing projects since 2012, and in this guide I explain LLMs at the level a developer or project owner actually needs. Then I move on to API integration, RAG, function calling and concrete project ideas.

What are large language models (LLMs)?

Large language models are AI models trained on very large text datasets to predict, by probability, which piece of text should come next. By repeating that prediction step after step, they answer questions, summarise documents, write code and produce output that follows your instructions.

The key word in that definition is "predict". A model does not look up facts like a database. Instead, it generates the most likely continuation based on patterns it learned during training. That is why it can write a fluent sentence that is simply wrong. Most current models build on the transformer architecture from the 2017 paper Attention Is All You Need. For day to day development, however, you mainly need to understand how the model behaves at the input and output level.

How does an LLM actually generate text?

In practice, the model does not write word by word. It writes piece by piece. First, it converts your input into numeric pieces. Then, at every step, it calculates a probability distribution for the next piece and picks one. Finally, it appends that piece to the input and repeats the loop.

This loop also explains why output settings matter. With a low temperature, the model stays close to the most likely choice. With a higher temperature, it makes more varied choices. For example, a tool that extracts fields from invoices needs a low temperature. On the other hand, a slogan generator may benefit from a little more randomness.

• Input: the system prompt, the user message and any attached documents.
• Processing: tokenisation and a probability calculation at each step.
• Output: text or structured data, up to the length limit you set.

What is a token, and why does it drive cost?

Put simply, a token is the smallest unit a model works with. Sometimes a token is a whole word. Sometimes it is part of a word or a punctuation mark. Languages with long compound words or heavy suffixes, such as German or Turkish, often need more tokens for the same meaning than English does.

Tokens sound like a technical detail, yet they land directly on your invoice. Most API providers price input and output tokens separately. So a long system prompt that you send with every request can turn into a serious line item by the end of the month. To get a rough feel for text length, a word counter helps. For exact token counts, use the provider's own counting tool.

Token limits apply to the output as well. If you set the maximum output length too low, the model may stop mid sentence. Therefore, check the stop reason field in the API response and offer a "continue" option when needed.

What does the context window mean for large language models?

For large language models, the context window is the total number of tokens they can see in one request. The system prompt, conversation history, attached documents and the model's own answer all have to fit inside it. Once it fills up, you need to trim or summarise the oldest messages.

A huge window looks attractive, but it does not solve everything. First, every token you send costs money and time. Second, research suggests that models can miss information buried in the middle of very long inputs. That is why I prefer "send only what is needed" over "send everything". The RAG approach I describe below grew out of exactly this need.

As a practical rule, summarise the conversation history once it passes a certain length and continue with the short version. That way you keep both cost and latency under control.

Why do hallucinations happen, and how can you reduce them?

Put simply, a hallucination is when the model states something false with full confidence. A made up source, a function that does not exist and a wrong date are typical examples. The root cause is simple: the model optimises for probability, not truth. So when its training data has little to say about a question, it still produces a fluent answer.

You cannot eliminate hallucinations. Still, these methods reduced them noticeably in my projects:

1. Give the model the source text and ask it to answer only from that text.
2. Explicitly allow it to say "I don't know".
3. Request output in a fixed schema and validate it in code.
4. Add a human approval step for critical fields such as prices, dates and legal wording.
5. Ask it to return the ID of the source passage, so you can check the answer.

For developers, the sneakiest kind of hallucination is a library method that does not exist. The model hands you convincing code, but the method it calls is missing in your version. Always compile it, test it and compare it with the official documentation.

How do large language models differ from a search engine?

In short, a search engine finds existing documents and ranks them as links. Large language models, in contrast, generate new text. That difference decides which job belongs to which tool. For volatile facts such as prices, stock levels or regulations, a model on its own is not a reliable source. However, it is excellent at summarising scattered information, drafting and classifying text.

This distinction matters for marketing too, because AI search experiences now blend both approaches. If you want to know how your brand appears in those answers, read my article on how your brand shows up in ChatGPT and Gemini. For the content side, my guide on writing content for AI Overviews is a good next step.

How do you use an LLM through an API in a software project?

For most projects, you do not need to train your own model. Instead, you send requests to hosted models through the APIs of providers such as OpenAI, Anthropic or Google. The basic flow looks similar everywhere. You get an API key, send a list of messages and receive a JSON response.

In practice, a typical request contains these parts:

• Model name: which model and version you call.
• System prompt: the model's role, tone and rules.
• Messages: the user and assistant turns in order.
• Parameters: maximum output length, temperature and stop sequences.
• Tool definitions: the schema of functions the model may call.

One security note matters above all: never put your API key in browser JavaScript. Route requests through your own server, add per user quotas and log errors. Otherwise someone will find the key, and you will pay the bill.

How do you write a good system prompt?

Specifically, the system prompt is the model's job description. If you write something short and vague, the model fills the gaps with its own guesses. So treat it like a brief for a new employee on day one.

My structure has four parts. First, I describe the role and the audience. Next, I list what the model should and should not do. Then I show the output format, ideally with a sample JSON. Finally, I add a rule for ambiguous cases, for example "if you are unsure, ask a question".

Do not write the prompt once and forget it. Build a small test set from real user messages and rerun it after every change. As a result, you notice early when a fix for one issue breaks another behaviour.

Examples deserve a mention too. Showing two or three realistic samples of the output you want often works better than a long list of rules. Keep the samples varied, though. If they all follow one pattern, the model will cling to it.

What is RAG, and when do you need it?

RAG (Retrieval-Augmented Generation) means retrieving relevant documents from your own data and adding them to the context before the model answers. The idea spread with the 2020 paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Lewis and colleagues. In other words, the model answers from the current text you supply, not from memory.

You need RAG when the knowledge is private to your company, changes often or is too large for the context window. A 400 page product manual, internal procedures or a support history all fit that description. On the other hand, building RAG for a short, static FAQ adds needless complexity. In that case, putting the text straight into the prompt is enough.

RAG reduces hallucinations, but it guarantees nothing. If the retrieval step returns the wrong passage, the model still gives a wrong answer. Consequently, quality in RAG projects depends more on retrieval than on the model itself.

How do you build a RAG pipeline step by step?

A basic RAG pipeline has five steps. Keep each one simple at first, then improve it by measuring.

1. Chunking: split documents into meaningful sections. Splitting by headings often beats a fixed character count.
2. Embedding: turn each chunk into a numeric vector with an embedding model.
3. Storage: keep vectors and source metadata in a vector database or a PostgreSQL setup with vector support.
4. Retrieval: embed the user question and find the closest chunks. Combining this with keyword search usually improves accuracy.
5. Generation: pass the retrieved chunks with their IDs to the model and ask it to answer only from that context.

The most neglected part of this pipeline is updating. When a source document changes, you must refresh its vectors too. Otherwise the system keeps quoting an old price or an outdated procedure with full confidence.

Chunk size also shapes results, so plan it. Tiny chunks lose context, while huge chunks drag irrelevant text along. My starting point is a few paragraphs per chunk, stored together with its heading. After that, I tune it against the evaluation set.

How do you get structured output (JSON) from a model?

Once an LLM connects to real software, free text becomes a problem. Your code wants to read a field. The model, meanwhile, sometimes adds a friendly sentence or renames a key. That is why I recommend requesting output against a fixed JSON schema in every software project.

Most providers now offer structured output or a JSON mode. With it enabled, the model has to return an object that matches your schema. Even so, keep a validation layer in your code. Check the schema with Pydantic, Zod or a similar library. If validation fails, call the model once more or show a safe error.

Specifically, a few rules help with schema design:

• Choose clear, unambiguous field names, such as "order_status" instead of "status".
• Prefer fixed enums over free text wherever you can.
• State that the model may leave a field empty when unsure.
• Write a description for every field inside the schema, because the model reads them.

With that in place, the model's output becomes a clean record you can write straight into your database or CRM.

What is the difference between an embedding model and a chat model?

An embedding model does not write text. It converts text into a sequence of numbers, a vector, that represents its meaning. A chat model, by contrast, generates new text. The two serve different purposes and work together in systems like RAG.

Above all, the power of embeddings lies in similarity. Sentences with similar meaning produce vectors that sit close together. For example, "my parcel is late" and "my order still hasn't arrived" share almost no words, yet their vectors are close. As a result, you catch matches that classic keyword search misses.

Embedding models usually cost far less than chat models. So embedding thousands of documents once is a small budget item in most projects. However, if you switch embedding models, you must regenerate every vector, since vectors from different models are not comparable. I also use embeddings to group similar content, spot duplicate support tickets and power simple recommendation features.

What is function calling (tool use)?

Function calling means the model tells you, in a structured format, which of your functions it wants to run and with which arguments. The model does not run the function itself. Your code receives the request, executes the function and sends the result back. Then the model writes its reply to the user based on that result.

For example, an e-commerce assistant might have a function called "get order status". When a customer asks where their parcel is, the model signals that it wants to call this function with the order number. That way, the model reads live data instead of inventing it. Provider documentation covers this flow in detail; the Anthropic tool use documentation, for instance, walks through the schema step by step.

Also, write the description field of each tool with care. The model relies heavily on that text when deciding which tool to pick and when.

How do you keep function calling secure?

The moment you give a model tools, you also give it authority. So security design belongs at the start, not at the end. The OWASP Top 10 for LLM Applications ranks prompt injection first and excessive agency sixth.

In practice, I apply these rules:

• Separate read tools from write tools, and ask for user confirmation before any write.
• Limit every tool to the permissions the user already has.
• Validate model generated arguments in code, and never execute raw SQL or shell commands.
• Do not trust instructions inside external content like documents or emails, because they may carry hidden commands.
• Log every tool call and set upper limits for amounts and quantities.

How do you choose the right model?

Put simply, choosing a model is not a one time decision. Providers release new versions and change prices often. Therefore, avoid tying your code tightly to a single provider. Put the model call behind a thin abstraction layer instead.

In practice, a mixed setup has served me best. Routing simple tasks to a small model and hard questions to a large one cuts cost without hurting quality.

Use your own evaluation set when choosing. Public benchmarks give a general idea. Still, they do not show how a model behaves on your data, in your language and for your task. Running the same twenty or thirty questions across several models and reading the answers side by side is the most reliable method I know.

What example projects can you build with large language models?

When you pick your first project with large language models, ask two questions. Can you measure the result? And is the damage limited when the model gets something wrong? The ideas below meet both conditions, so they make good starting points.

• Support knowledge assistant: a help bot that uses RAG over FAQs and manuals and cites its sources.
• Product description drafter: a tool that turns a spec sheet into on brand copy for an editor to approve.
• Form and email classifier: a service that tags incoming requests by topic and urgency, then writes them to your CRM.
• Meeting notes summariser: an internal tool that pulls decisions and action items from transcripts.
• Code review helper: a team tool that summarises changes and flags risky sections.
• Document field extractor: a module that reads dates, amounts and parties from invoices and contracts as JSON.

These projects become even more valuable once they connect to your website. In my e-commerce consulting work, for example, I use LLM based helpers to clean up product data.

How do you integrate an LLM feature into a website?

Getting the architecture right at the start saves a lot of time later. The front end handles only the interface. The model call, quotas, logging and RAG steps stay on the server. Streaming the response also helps, because the user sees text appear instead of staring at a blank box.

On large corporate sites, building the feature as a separate module makes maintenance easier. I explained the general idea in my article on micro frontends. If you need help with the interface and conversion side, I plan these integrations as part of my web design service.

Keep in mind that a chat box is not the right interface for every site. Sometimes a single "summarise this product" button gets far more use than an open ended chat.

How do you keep cost and performance under control?

LLM bills tend to grow quietly until one month they suddenly stand out. That is why I recommend measuring cost from day one. Log the input and output tokens, latency and originating feature of every request.

These methods work well in practice. First, cache answers to frequently repeated questions. Second, use the prompt caching features some providers offer for static system prompts. Third, set a sensible cap on output length. Finally, route simple tasks to a smaller model.

Example calculation: picture an assistant that handles 1,000 requests a day, each carrying about 2,000 input tokens. If you cut the system prompt from 1,500 tokens to 500, daily input volume drops by roughly half. The numbers are hypothetical, but the ratio logic holds for any price list.

How do you measure the quality of LLM output?

"It seems to work" is not a measurement. Instead, collect at least a few dozen real user questions and write the expected answer for each. This evaluation set becomes the test you run after every prompt or model change.

Specifically, I look at three dimensions: correctness, faithfulness to the source and format compliance. You can check correctness with human review and format compliance automatically in code. You can also use a second model as a judge; still, compare its verdicts with human decisions from time to time.

After launch, collect user feedback. Even a simple "was this helpful" button shows you which question types give the system trouble.

What should you watch for with personal data and GDPR?

First, remember that every piece of text you send to an API travels to the provider's servers. So before sending text that contains personal data, ask whether you really need to. Masking names, phone numbers and ID numbers rarely hurts answer quality in most scenarios.

When choosing a provider, check the contract for data retention periods, whether your data trains their models and where the servers are located. If you operate in the EU, review cross border transfers under GDPR with your legal adviser. This article is not legal advice; it only lists the questions to ask on the technical side.

Also remember your logs. When you store model requests for debugging, you may store personal data without noticing. Apply masking in logs and set a retention period.

Which mistakes do I see most often in LLM projects?

The mistakes I meet in client projects look remarkably alike. In short, the problem is usually the project setup, not the model.

• Starting with "let's add AI" without a measurable goal.
• Showing model answers to customers without validation.
• Leaving the API key on the client side.
• Forgetting to update RAG sources.
• Skipping cost logging and getting a surprise at month end.
• Locking into one provider so tightly that switching models becomes impossible.

That said, none of these requires advanced AI knowledge. They are simply good software discipline applied to LLM work.

Expectation management matters as well. Stakeholders see the first demo and assume the product will be right every time. Real users, however, ask questions nobody thought of during the demo. So talk openly about error rates and fallback behaviour before launch.

How does LLM generated content affect SEO?

Non technical project owners ask me this all the time, so here is my view. Google's position is clear: it cares whether content helps users, not how it was produced. So drafting with an LLM is fine. Publishing unchecked, low value pages at scale, however, creates risk.

My advice is to treat the model as a co writer. Let it produce the draft, while expertise, examples and fact checking come from you. On the infrastructure side, how AI crawlers read your site matters more and more; I cover that in technical SEO after AI. For strategic support at company scale, see my SEO consulting page.

Where should you start with large language models?

The best way to start with large language models is to pick a small, measurable, low risk task. In week one, try a provider's API and build a simple prototype for one feature. Next, in week two, prepare your evaluation set and add cost logging.

In week three, compare results with the business goal. Did support response time drop? Did the editor's correction effort shrink? If the numbers look good, expand the scope. If not, revisit the prompt and the data.

After that, run a limited test with real users. Add layers such as RAG or function calling only when a real need appears. That way, complexity stays proportional to the job.

One last thought: this field moves fast, but the core concepts stay put. Once tokens, context, hallucinations, RAG and tool use make sense to you, every new model becomes just a configuration change. If you would like to discuss your project, reach me through my contact page.