How Large Language Models Actually Work

An LLM is a Large Language Model — a type of artificial intelligence trained to predict the next word in a sequence, given all the words before it. It does this by learning statistical patterns from billions of examples of human text (books, websites, code, legal documents). The key insight: it is not programmed with rules; it learns patterns. This is radically different from the Master of Laws degree, also called an LLM, that some attorneys pursue. In AI, LLM is the abbreviation; in education, it's your credential. Both are real in legal contexts, so we clarify: here, LLM = Large Language Model, the AI system.

When we say "large," we mean two things. First, the model itself is huge — measured in billions of parameters (internal knobs the network adjusts during training). GPT-3 has 175 billion parameters; GPT-4's size is not publicly disclosed but is estimated to be larger. Second, the training dataset is enormous — GPT-3 trained on ~300 billion tokens of text, a scale that took years to assemble. Scaling up both the model and the data has proven to be one of the most reliable ways to improve performance across a vast range of tasks, a trend codified in scaling laws (Kaplan et al., 2020).

The core mechanism is surprisingly simple: predict the next token (a chunk of text, usually a few characters or a word). During training, the model sees millions of examples of text, and for each one, it learns to adjust its internal weights so that, given the first N words, it can guess the (N+1)th word accurately. This process, called next-token prediction or causal language modeling, is the entire training objective. Once trained, the model can generate new text by applying this same prediction logic iteratively — predict the next word, then the one after that, and so on — a process called autoregressive generation.

Why does predicting the next word teach a model anything useful? Because to predict text accurately, a model must learn to understand grammar, facts, reasoning, and even some aspects of how humans think. It must learn that "attorney" and "lawyer" are semantically related; that certain practices tend to involve litigation; that there is something like legal jurisdiction and how it works. None of this is explicitly programmed in. The model infers it by finding patterns in the data.

Words are not the fundamental unit for LLMs. Instead, text is split into tokens — usually sub-word units that represent a few characters or a whole word. The algorithm that does this is called Byte-Pair Encoding, or BPE (Sennrich et al., 2016). During tokenization, the model first breaks text into individual characters, then iteratively merges the most common adjacent pairs until it reaches a target vocabulary size (typically 50,000 to 100,000 tokens). For example, the word "playing" might be tokenized as ["play", "ing"] or ["p", "lay", "ing"], depending on the frequency of these chunks in the training data.

Why not just use words? Three reasons. First, there are millions of possible words across all languages (including technical jargon, proper nouns, slang), but far fewer common sub-word units; a fixed vocabulary is tractable. Second, rare words can be represented as combinations of more common tokens, so the model can process any text without unknown-word problems. Third, breaking text into smaller units helps the model learn morphological patterns — the fact that "playing," "played," and "play" share a root. Tokenization is reversible: tokens can be converted back to text.

One consequence: token count is not the same as word count. A phrase like "it's a beautiful day" might tokenize to 5–8 tokens depending on the tokenizer, even though it is 5 words. This matters for context windows: when we say a model has a 4,096-token context window, that is not 4,096 words — it is roughly 3,000–4,000 words, depending on the language and domain (legal documents often tokenize to more tokens per word because of specialized terminology).

Different LLM providers use different tokenizers. OpenAI's GPT models use a custom BPE tokenizer; Google's models use SentencePiece, another subword algorithm. These differences are implementation details, but they have real effects: the same text might produce different token counts across models, which affects cost and context-window management. When building systems that interact with multiple LLMs, token counting becomes a practical concern.

Once text is tokenized, each token is converted into a vector — a list of numbers, typically 500–4,000 numbers long, depending on the model. This is called an embedding. The embedding captures semantic meaning: tokens that mean similar things have embeddings that are close together in vector space, measured by cosine similarity (the angle between two vectors). For example, "lawyer" and "attorney" have embeddings that point in similar directions; "plaintiff" and "defendant" are opposite. This geometric interpretation of meaning is the foundation of how LLMs understand text.

Embeddings are learned during training. The model starts with random vectors for each token, then adjusts them so that tokens that appear in similar contexts end up with similar embeddings. This process was first formalized in Word2Vec (Mikolov et al., 2013), which showed that you can learn meaningful word vectors just by predicting a word given its neighbors. LLMs use a similar idea: during training on next-token prediction, tokens that often appear together learn to have related embeddings.

The power of this approach: once a token has an embedding, the model can reason about it geometrically. It can measure that "Paris" minus "France" plus "Germany" is close to "Berlin" — a famous example of vector arithmetic encoding geographic and semantic relationships. For legal AI, this means the model can learn that "statute of limitations," "filing deadline," and "prescriptive period" are semantically related even if they never appear in the same document, because they all embed into a similar region of vector space.

Vector embeddings also bridge to modern AI search: search engines like Perplexity, Gemini, and ChatGPT use embeddings to find relevant passages in a vast corpus. When you ask a question, your question is converted to an embedding, then the system searches a database of document embeddings for vectors that are close (high cosine similarity). Those semantically similar passages are then fed to the LLM as context. This is the retrieval-augmented generation framework, discussed later. The practical implication: to be found by AI search, your content's embedding must land in the right semantic neighborhood — which depends on word choice, entity consistency, and structural signals.

Every modern LLM is built on the transformer architecture, introduced in "Attention Is All You Need" (Vaswani et al., 2017). Before transformers, language models used recurrent neural networks (RNNs) — networks that process text sequentially, one token at a time, maintaining a hidden state that gets updated as new tokens arrive. RNNs were the state of the art for years, but they had a fatal flaw: they process sequentially, one token per step, which means you cannot parallelize training across many tokens at once. For billion-token datasets, sequential processing is prohibitively slow.

The transformer replaces recurrence with attention. The core idea: every token in a sequence can directly attend to (look at, weighted by relevance) every other token, all at once. This is self-attention. Given a sequence of token embeddings, the attention mechanism computes a score for how much each token should attend to every other token, then uses these scores to create a weighted average of the embeddings. The result: each token gets a new representation that incorporates information from all other tokens in the sequence, in a single parallel step.

Here's how self-attention works, step by step. Each token embedding is transformed into three vectors: a Query (Q), a Key (K), and a Value (V). The Query represents "what am I looking for?" The Key represents "what information do I have?" The Value is "what data should I pass along if someone attends to me?" For each token, the attention mechanism computes a similarity score between that token's Query and every other token's Key (using dot product, a measure of vector alignment). These scores are normalized into probabilities (so they sum to 1), then used to create a weighted sum of all the Values. The result is a new embedding for that token that incorporates context from the entire sequence.

Why does this work for understanding meaning? Because the model can learn to assign high attention weights to tokens that matter for the current prediction task. For example, when predicting the next word after "The defendant argued the statute of limitations had...," the model might attend heavily to "statute of limitations" and "defendant" because those are semantically critical. The model learns what to attend to during training.

The transformer has one more ingredient: multi-head attention. Instead of one attention mechanism per layer, there are many (often 8, 16, or more) running in parallel. Each head learns to attend to different aspects of meaning. One head might focus on grammatical relationships; another on semantic similarity; another on entity references. This redundancy and specialization makes the model more robust and expressive.

Finally, since the attention mechanism treats all positions equally, the model has no built-in notion of word order. A token at position 1 looks identical to a token at position 100, except for the embeddings themselves. To preserve position information, the architecture adds positional encodings — vectors that encode position and are added to each embedding. These are usually sinusoidal (Vaswani et al. used sin and cos functions), but the key point is that they ensure the model can distinguish "The lawyer sued the company" from "The company sued the lawyer."

Transformers are stacked in layers (typically 12–96 layers, depending on model size), where each layer applies self-attention and a feed-forward network. Larger models have more layers and higher dimensionality, which increases their capacity to learn and represent complex patterns. This is why GPT-4 is more powerful than GPT-3: more parameters means more capacity.

Training an LLM is a two-stage process: pretraining and alignment. Pretraining is unsupervised learning on massive text corpora. The model receives sentences from the internet, books, code repositories, and legal documents (anything it was licensed to use), and for each one, it learns to predict the next token given all prior tokens. Billions of examples of this task teach the model to model language structure and encode facts.

During pretraining, the model never sees human labels or instructions like "summarize this" or "is this correct?". It only sees raw text and the implicit signal of whether its prediction was right or wrong. Yet from this signal alone, models learn remarkable abilities. GPT-3 (Brown et al., 2020), trained on ~300 billion tokens, showed that even without explicit instruction tuning, the model could solve downstream tasks like question-answering, translation, and arithmetic with just a few examples of the task (few-shot learning). This generalization is one of the deepest mysteries in deep learning: how does predicting the next word encode so much knowledge?

Pretraining is expensive. A state-of-the-art model can take weeks to train on thousands of GPUs or TPUs. Once trained, the model is frozen and released as a base model. But base models are not ready for user-facing applications. They are excellent at next-token prediction but not at following instructions or avoiding harmful outputs. This is where alignment comes in.

Alignment is supervised fine-tuning (SFT) followed by reinforcement learning from human feedback (RLHF). In SFT, humans write high-quality examples of responses to instructions, and the model is fine-tuned to predict those responses. For example, a human writes "User asks: What is statute of limitations? Assistant: A statute of limitations is a law that limits the time..." and the model learns to match that response pattern. SFT alone produces a more instruction-following model.

RLHF adds a second stage: a different model (the reward model) learns to predict human preferences. Humans rank multiple responses (good vs. bad, more helpful vs. less helpful), and the reward model learns to assign scores to candidate outputs. Then the original LLM is fine-tuned using reinforcement learning to maximize its reward-model score — essentially, to produce outputs that humans prefer. This is why ChatGPT and Claude feel more aligned to user intent than raw GPT-3: they are trained with human preferences baked in (Ouyang et al., 2022).

Scaling laws are an empirical discovery: model loss (prediction error) decreases predictably as you increase model size, dataset size, and compute budget. Specifically, loss follows a power law: doubling the number of parameters reduces loss by a constant factor (roughly 1–2%), and the same holds for data and compute (Kaplan et al., 2020; Hoffmann et al., 2022). This means you can estimate in advance how much improvement you get for a fixed budget. For large organizations, this has driven enormous investment in computing infrastructure and data collection, because the gains are predictable.

One implication: there is no obvious efficiency ceiling. The scaling laws suggest that if you keep scaling, performance keeps improving, at least up to human-level reasoning on many tasks. This is why companies are investing in ever-larger models.

Inference is the process of using a trained model to generate text. Given a prompt (e.g., "Explain venue in civil procedure:"), the model predicts the most likely next token, adds it to the sequence, and repeats. This is autoregressive generation: each token is conditioned on all prior tokens.

At each step, the model does not output a single token. Instead, it outputs a probability distribution over all possible next tokens (the vocabulary is 50,000 to 100,000+ tokens). For example, after "The attorney argued that...," the probabilities might be: "the" (20%), "venue" (8%), "damages" (5%), etc. To actually select a token, the model must sample from this distribution. There are two main strategies:

Greedy decoding picks the single most likely token at each step. This is deterministic — the same prompt always produces the same output. But it often produces repetitive, bland text because the model just takes the safest bet at each step. Diversity in language requires some randomness.

Sampling picks tokens from the probability distribution using randomness. Uniform sampling (every token equally likely) produces nonsense. Instead, temperature scaling is used: the logits (raw scores before probability conversion) are divided by a temperature parameter. Low temperature (e.g., 0.1) sharpens the distribution so high-probability tokens are even more likely; high temperature (e.g., 1.5) flattens it so low-probability tokens have a better chance. Temperature 1.0 is neutral. This is why the same prompt can produce different outputs even from the same model: sampling introduces randomness.

Top-k and nucleus sampling add additional constraints: only the top k most likely tokens are considered, or only tokens that collectively account for the top p fraction of probability mass. This keeps generation focused on plausible continuations while preserving diversity.

One consequence: LLM outputs are not fully reproducible. If you prompt ChatGPT twice with the same question, you might get slightly different answers (unless temperature is 0). For mission-critical applications like legal writing, this is important: the output must be reviewed and may need to be regenerated or edited for consistency.

A hallucination is when an LLM produces text that is fluent and plausible but factually false or fabricated. For example, it might cite a made-up law, invent a court case, or claim a fact with high confidence that is simply not true. For law firms, hallucinations are a serious concern.

The root cause is fundamental: LLMs model the distribution of text, not the distribution of truth. An LLM trained to predict the next token learns patterns in language and facts that co-occur in training data. But it does not have an internal representation of "is this true?" or access to a knowledge base of verified facts. When it generates text, it is synthesizing the most likely continuation given the prompt, using patterns it learned. If the pattern for generating text about a topic is similar to the pattern for generating false claims about the topic (because falsehoods appear in the training data), the model will generate falsehoods.

A related issue: models are overconfident. They do not usually express uncertainty. When a model does not know something, it does not say "I don't know"; it generates a plausible-sounding guess. This is because the training objective (next-token prediction) does not reward the model for expressing uncertainty — it rewards accurate prediction, whether or not the model actually knows the answer.

Hallucinations are worse for rare or recently changed facts. A model trained on data up to April 2024 will hallucinate about May 2024 events because it has never seen them. Similarly, a specific local statute or recent court ruling might not be in the training data, so the model will invent a plausible-sounding rule. For legal applications, this is a critical limitation.

Mitigation strategies exist but are not perfect. Retrieval-augmented generation (discussed next) grounds the model in external sources, dramatically reducing hallucinations. Chain-of-thought prompting (asking the model to show its reasoning) sometimes catches hallucinations because the reasoning reveals the model's uncertainty. But none of these approaches eliminate hallucinations entirely. Any application that relies on factual accuracy must have a verification step — a human reviewing the output against a trusted source.

Retrieval-Augmented Generation, or RAG, is the mechanism that modern AI search engines use to ground LLM responses in external sources (Lewis et al., 2020). The idea: instead of relying only on facts in the model's training data, the system retrieves relevant passages from the web or a document corpus, feeds those passages to the LLM as context, and asks the LLM to answer the user's question based on the retrieved passages.

Here is the typical RAG pipeline: (1) User asks a question, e.g., "What is spoliation in legal discovery?". (2) The system converts the question into an embedding — a vector representing the semantic content of the question. (3) The system searches a database of embeddings for passages that have high cosine similarity to the question embedding. These are semantically relevant passages. (4) The top passages (typically 3–20, ranked by similarity) are retrieved. (5) These passages are concatenated and fed to the LLM as context. (6) The LLM generates an answer based on the context. (7) The answer is post-processed: sources are extracted and cited.

Why RAG works so well: the LLM is forced to ground its answer in retrieved text. If the retrieved passages contain the answer, the LLM will usually find it and cite it accurately. RAG dramatically reduces hallucinations because the model has explicit evidence to draw from. Studies show that RAG systems have ~20–50% fewer hallucinations compared to LLMs without retrieval, depending on the domain.

For law firms, RAG is how AI search engines like Perplexity, ChatGPT (with web browsing), Gemini, and Claude (with web search) find and cite your content. The pipeline is: (1) The user types a query. (2) The system embeds the query. (3) It searches the web for pages whose content embeddings are similar to the query embedding. (4) It fetches those pages (or retrieves cached embeddings if they were previously indexed). (5) It extracts relevant passages from the pages (usually using passage-level embeddings, since a full page might be too long; a page is split into overlapping passages, each ~100–200 words). (6) It feeds the top passages to the LLM. (7) The LLM cites the passages and the originating URLs.

Critically: to be found and cited by an AI search engine, your page must be (a) crawlable by the AI engine's bot (requires allowing bots like GPTBot, ClaudeBot, PerplexityBot in robots.txt), (b) server-rendered (content must be in the initial HTML, not loaded by JavaScript, so the bot can index it), (c) have embeddings that land in the semantic neighborhood of the query, and (d) contain citable passages — structured, extractable text that the LLM can quote verbatim. This is why SSR (server-side rendering), passage-level structure (H2/H3, tables, lists), and semantic consistency matter.

AI search engines prioritize different signals, but all value: relevance (embedding similarity), authority (inbound links, domain age), and freshness (recent content gets a boost). Google AI Overviews additionally favor passages from top-ranking organic results, creating some correlation between traditional SEO and AI citability. But the correlation is imperfect — a page can rank organically but be poorly structured for AI extraction and thus not be cited. Conversely, a well-structured, semantically clear page can be cited by AI engines even if it does not rank #1 organically.

Hallucinations are not a bug; they are a fundamental feature of how LLMs work. The model's training objective — predict the next token given all prior tokens — does not include a truth signal. The model learns to assign high probability to continuations that are similar to things in the training data. If the training data contains falsehoods (and it does, from the open web, disputed historical claims, and errors), the model learns to reproduce them.

Additionally, LLMs are stateless. They do not query a fact-checking system or a knowledge base. They do not "check" whether their generated text is consistent with known facts. They generate text by sampling from a learned probability distribution. Once a hallucination begins (e.g., "The statute of limitations for medical malpractice in California is 5 years"), the model will continue to hallucinate confidently, because the early false claim shapes the conditional distribution of subsequent tokens.

A striking example of this is when an LLM confidently cites a paper that doesn't exist. The model has learned that academic papers are cited a certain way (author, year, title, journal). When asked to cite its sources, it generates plausible-looking citations in that format. But the paper never existed. Humans have called this "citation fabrication" or "invented citations," and it is disturbingly common in GPT-3 and GPT-4 without retrieval augmentation.

For lawyers, this is a nightmare scenario. A lawyer using an LLM to draft a motion might unknowingly cite a case that does not exist. This has happened — there are documented cases of lawyers citing fabricated case law in court filings, discovered only after opposing counsel questioned the cite. The consequence is embarrassment, sanctions, and loss of credibility.

The solution is enforced retrieval and human review. Any LLM application handling facts must (a) use RAG to ground answers in retrieved sources, (b) require humans to verify critical facts before they are used (especially legal facts, dates, names, numbers), and (c) be transparent about the model's limitations. If an LLM generates a response about a recent ruling or a specialized area of law outside its training data, that response should be treated as a draft for human review, not as authoritative.

Understanding LLM mechanics changes how law firms should think about online visibility and content strategy. Traditional SEO optimizes for Google's crawler and ranking algorithms. LLM-centric visibility — what we call GEO (Generative Engine Optimization) — optimizes for a different set of constraints: embeddings, retrieval, and extractability.

First: retrievability. For an AI search engine to cite your content, it must be able to crawl it. This means (a) allowing AI bots in robots.txt (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot), (b) server-rendering all critical content (no JavaScript-only content), and (c) ensuring the page is not behind a login, paywall, or IP restriction. Many law firm sites fail on (b) and (c): practice-area pages are sometimes only accessible after authentication, or are rendered as SPA (single-page application) JavaScript, both of which make them invisible to AI crawlers.

Second: semantic clarity and passage structure. Embeddings are dense, high-dimensional representations of meaning. For your content's embedding to land in the right semantic neighborhood — close to user queries about your expertise — the content must be semantically clear and consistent. This means (a) using consistent terminology (if you alternate between "statute of limitations," "filing deadline," and "prescriptive period" randomly, the embeddings will be noisier), (b) structuring content into clear sections (H2/H3 headings) so the passage-level chunking algorithm can isolate citable passages, and (c) defining key entities (your practice areas, jurisdictions, case types) consistently so the model's entity embeddings reinforce each other.

Third: entity linking and semantic neighborhoods. LLMs learn that certain entities co-occur: "medical malpractice" with "statute of limitations," "California" with "state bar," "plaintiff" with "damages." If your page mentions all of these consistently, the model learns to associate your page with this semantic cluster. Conversely, if your page is generic and mentions everything under the sun ("we handle all types of law," no specificity), the model's embedding for your page will be diffuse and match many queries poorly — your page will not stand out in any semantic neighborhood.

Fourth: fact verification and transparency. Any time an LLM would cite your content, that content must be truthful and verifiable. The model cannot fact-check; it will reproduce whatever is on your page. If your page claims you won a case and got a certain verdict, and the claim is false, the model will cite it confidently. An attorney who read the LLM's output might include the false claim in their own work, not knowing it is fabricated. For law firms, every fact must be real: real case outcomes, real results, real credentials, real locations. If you exaggerate or invent, an AI search engine will propagate the false claim.

Fifth: the bridge between traditional SEO and GEO. A page that ranks #1 on Google but is poorly structured for passage extraction will likely be cited less by AI engines. Conversely, a page that is beautifully structured for AI but has no inbound links might not be crawled at all. The best strategy optimizes for both: clear, fact-dense, semantically consistent content that is both organic-rankable (good links, domain authority, topical authority) and AI-citable (SSR, passage structure, entity clarity).

Finally: the role of your own website as a source. AI search engines have started to downrank pages from websites they don't trust and uprank pages from high-authority domains. If your law firm website has been around for years, has accumulated links and citations, and has consistently published accurate, detailed content, the model will treat your pages as more reliable sources. If your site is new, generic, or has low authority, your pages will rank lower in semantic search despite having identical content. This is where off-site authority (press, legal directories, reviews, citations from reputable sources) becomes critical.

The field is evolving rapidly. As of 2026, several trends are shaping the next generation of LLMs:

Longer context windows: Models are getting access to longer and longer sequences of text. GPT-3 had a 2,048-token context window; GPT-4 has 8,000–128,000 tokens depending on the variant. Claude 3 supports 100,000 tokens. Longer context enables models to reason over entire documents, legal briefs, and case files without summarization. This is a profound shift for law: an attorney can feed a 50-page contract into an LLM and ask it to identify risks, and the model can maintain coherent reasoning across the entire document.

Multimodal reasoning: Modern LLMs are incorporating images, videos, and audio. This matters for legal discovery (analyzing images of documents), patent law (diagrams and schematics), and evidence presentation (videos). Multimodal models can read charts, tables in images, and diagrams, extracting structured information from formats that text-only models struggle with.

Mixture of Experts (MoE): Instead of one monolithic model, some new architectures route different inputs to different specialized sub-models. This can improve efficiency and allow selective scaling of model capacity. The tradeoff is increased complexity.

Fine-tuning and customization: While base models are frozen during inference, users are increasingly fine-tuning models on domain-specific data. A law firm could fine-tune a model on their case wins, client intake documents, and playbooks, creating a specialized model that better understands their practice. This is becoming more accessible and cost-effective.

Reasoning and Chain-of-Thought: Models are being trained to show their reasoning explicitly ("Let me break this down step-by-step..."), improving transparency and reducing errors. This is especially valuable for legal reasoning, where the logic needs to be defensible.

Uncertainty quantification: Newer models are beginning to express confidence levels and uncertainty explicitly. Instead of always sounding certain, a model might say "I am 70% confident this applies based on..." This is still early, but would be transformative for law.

Note: These are rapid developments and the field's priorities shift quarterly. This summary reflects the state as of mid-2026, but LLM research is moving faster than any documentation can keep up with.