How can I create an LLM from scratch?

The Real Cost of Building Your Own LLM (and Why 76% of Enterprises Don't)

Before you write a single line of training code, you need to know what the data says: most organizations that could build their own LLM choose not to. According to Menlo Ventures' 2025 enterprise AI survey, 76% of enterprise AI solutions are now purchased rather than built internally — a decisive flip from 2024's near-even 47%/53% split. Enterprise generative AI spend reached $37B in 2025, up 3.2x year-over-year, and the overwhelming majority of it went to external platforms, not internal training runs.

The numbers behind building are sobering. Practitioner analysis puts the realistic timeline from decision to production at 9–18 months, with a team of 6–10 people needed just to serve a few business units — 15–20 for larger deployments (https://llmcapsule.ai/resources/learn/hidden-cost-of-building-your-own-llm/). And the biggest line item isn't GPUs: over three years, people costs exceed infrastructure costs by 2–3x.

Here's the counterintuitive part: the model itself is the cheapest component. As one analysis bluntly puts it, "the model itself (the actual weights, the open-source release the team is going to deploy) is the cheapest part of the deployment" (https://llmcapsule.ai/resources/learn/hidden-cost-of-building-your-own-llm/). The real budget goes to the surrounding system:

• Data pipelines, evaluation harnesses, and safety testing that keep the model useful and trustworthy
• Serving infrastructure, monitoring, and the engineers who run it at scale
• Ongoing maintenance — retraining, upgrades, and drift management long after launch

A common failure pattern is worth naming: the project succeeds technically and fails strategically. The internal model reaches production just as the workflow has already been re-architected around external endpoints. That's why most successful enterprises land on a hybrid architecture — self-hosted models for absolute constraints like data sovereignty, external endpoints for everything else (https://llmcapsule.ai/resources/learn/hidden-cost-of-building-your-own-llm/).

So what does "from scratch" actually mean for you? For most readers, it doesn't mean pretraining a model on trillions of tokens. It means starting from open weights — permissively licensed models like Qwen3, Phi-4-mini, and Gemma 3 — and adapting them to your needs, then wiring them into real workflows with agent skills. (If you go that route, SkillzBench is building a free directory and MCP connector to help you discover and vet those skills safely, since research has found roughly 26% of published skills carry at least one vulnerability.)

That leaves three realistic paths, and the rest of this article walks through each:

• Buy — use an external API; fastest, cheapest to start, least control
• Fine-tune — adapt an open-weight model to your domain; the sweet spot for most teams
• Build — train from scratch; justified only at millions of requests per day at sustained scale

The Practical Path: Start from Open Weights, Not a Blank Slate

The fastest way to "create" an LLM isn't to train one from nothing — it's to start from open weights and make them yours. That's not a shortcut; it's what most of the market already does: according to Menlo Ventures' 2025 enterprise survey, 76% of enterprise AI solutions are purchased or adopted rather than built internally.

The open-weight ecosystem is now rich enough that you can pick a model matched to your hardware. The practical guidance from the local-model community is simple: don't start with the biggest model — start with the best model your hardware can comfortably run:

• 8 GB RAM: Phi-4-mini (3.8B, MIT-licensed, CPU-runnable), Gemma 3 1B, or Qwen3 1.7B
• 16 GB laptop: Gemma 3 4B, Qwen3 4B/8B, or gpt-oss 20b
• RTX 3090/4090: Gemma 3 27B, Qwen3 30B, or Devstral

One distinction matters before you ship anything commercial: "open source" and "open weight" are not the same. Open-weight models like Llama 4 and Gemma 3 publish downloadable parameters under conditional licenses, while permissively licensed models (Apache 2.0/MIT) include Qwen3, Mistral Small 3.1, Devstral, Phi-4-mini, and gpt-oss. If you're building a product, read the license before shipping — Apache 2.0 and MIT are the safest choices.

What makes this path compelling is that small models keep getting better without getting bigger. Red Hat's 2025 state-of-open-models analysis notes that small language models now run on almost any consumer device, and their capabilities are increasing even with flat parameter counts — thanks to improved attention kernels, efficient block layouts, and synthetic data generation. Qwen became the most-used local model family by the end of 2025, evidence that open weights are credible, not just cheap.

Deployment tooling has matured to match. You can pull models from Hugging Face and run them with a single command via Ollama or RamaLama, or serve them in production with vLLM — which was the top open-source project by contributors on GitHub in 2025.

Once your model is running locally, the next question is what it actually does for you. Agent skills — SKILL.md instruction packages that teach coding agents and chat-app agents how to handle specific workflows — are how you turn a working model into a useful one. SkillzBench is building a free directory and MCP connector to discover, verify, and benchmark those skills safely, since published research found roughly 26% of skills carry at least one vulnerability. You can join the waitlist at skillzbench.com while the platform is being built.

If You Must Train: Scaling Laws for Smart Budgeting

Training your own LLM without a budget plan is like renovating a house without a quote — the bill arrives either way. The good news is that peer-reviewed research from the MIT-IBM Watson AI Lab offers a practical recipe for spending compute dollars wisely instead of guessing.

The core insight comes from a meta-analysis of 485 pre-trained models across 40 model families, drawing on roughly 1.9 million performance metrics and more than 1,000 fitted scaling laws. As MIT's Jacob Andreas notes, past scaling laws were typically built from "one model, or one model family, and one dataset, and one developer" — this study changed that. The result is guidance small teams can actually apply.

The method starts with two decisions made before any training run: fix your compute budget and define your target accuracy. Then, rather than betting everything on one configuration, you gather evidence cheaply.

• Train about five models across a spread of sizes — enough data points to fit a reliable scaling law for your setup.
• Discard training data before 10 billion tokens, which the researchers found too noisy to inform predictions.
• Partially train your target model to roughly 30% of its dataset, then extrapolate its final performance from that curve.

The payoff is striking: the best achievable prediction accuracy is around 4% absolute relative error, and even predictions up to 20% off remain useful for planning. Leshem Choshen of the MIT-IBM Watson AI Lab points out that intermediate checkpoints of a fully trained model can be reused "as if they were individual models — you don't pay anything in the training, because you already trained the full model." In other words, scaling laws democratize the field by enabling researchers without vast resources to plan like the big labs do.

This matters because the real costs of building are people, not GPUs. Practitioner analysis suggests 9–18 months to production and people costs exceeding infrastructure costs by 2–3x over three years, so every training run you avoid wasting has outsized value. A cost breakdown of self-hosting makes clear why disciplined budgeting is the difference between a learning project and a money pit.

Once your model is trained and running, the next challenge is making it genuinely useful in workflows — and that is where agent skills come in. SkillzBench is building a free directory and MCP connector for discovering, verifying, and benchmarking skills, which is worth a look given that published research found roughly 26% of skills carry at least one vulnerability.

Step-by-Step: Running and Fine-Tuning a Small LLM Locally

You do not need a data center to run a capable language model anymore. Small language models now run on almost any consumer device, and their capabilities keep improving even without parameter growth, thanks to better attention kernels and synthetic data techniques, according to Red Hat's analysis of the open-source landscape.

Start by downloading permissively licensed weights from Hugging Face. The safest licenses for commercial work are Apache 2.0 and MIT, covering models like Qwen3, Mistral Small 3.1, Phi-4-mini, and gpt-oss, per a Hugging Face community guide. Match the model to your hardware rather than chasing size: Phi-4-mini (3.8B parameters) runs on a CPU, gpt-oss 20b fits within 16 GB of memory, and an RTX-class GPU comfortably handles 27B–30B models.

Running the model is often a single command. The standard local tooling ecosystem includes Ollama, LM Studio, llama.cpp, and vLLM — Ollama for quick single-command deployments, vLLM for production inference serving. Quantization shrinks models for consumer hardware: Kimi K2, with roughly 1T total parameters, runs on a 48 GB A6000 at 4-bit quantization.

• 8 GB RAM: Phi-4-mini, Gemma 3 1B, Qwen3 1.7B
• 16 GB laptop: Gemma 3 4B, Qwen3 4B/8B, gpt-oss 20b
• RTX 3090/4090: Gemma 3 27B, Qwen3 30B, Devstral

Once the base model runs, fine-tune it with domain-specific data using parameter-efficient techniques such as LoRA, which reduce compute requirements without significantly compromising performance — a practice recommended by practitioners for making models relevant to your industry. Before shipping, benchmark the exact checkpoint and configuration you intend to deploy, because a model that generates impressive text but picks the wrong tool makes a poor production agent.

For architecture, most successful teams land on a hybrid pattern: self-hosted models where hard constraints demand them (data sovereignty, air-gapped deployment, millions of requests per day at sustained scale), and external endpoints for everything else. As one practitioner analysis puts it, the hybrid pattern "isn't a compromise — it's the architecture that matches the cost profile of each workflow category to its actual constraint."

Finally, a local model becomes genuinely useful when paired with agent skills — SKILL.md instruction packages that teach it your workflows. Since published research found roughly 26% of skills carry at least one vulnerability, SkillzBench is building a free directory and MCP connector that security-scans, license-checks, and benchmarks every skill so you can equip your agent with skills, proven — join the waitlist at skillzbench.com to get early access.

From Model to Workflow: Making Your LLM Actually Useful (and Safe)

So you've got a model running locally — a Phi-4-mini answering questions on your laptop, or a Qwen3 variant humming along via Ollama. Now comes the part that actually matters for your work: turning a chat window into a workflow. A model that generates impressive text but picks the wrong tool is, as one evaluation guide puts it, a poor production agent.

That's where agent skills come in. Skills — SKILL.md instruction packages used by coding agents like Claude Code, Codex, and Cursor, plus chat-app agents — are how you teach an LLM to do a specific job: triage support tickets, draft reports in your format, or run a data pipeline. Instead of fine-tuning weights, you package instructions, and the model follows them. It's the pragmatic path the market has already endorsed: 76% of enterprise AI solutions are purchased rather than built, per Menlo Ventures' 2025 enterprise survey — and skills are the lightweight equivalent for individuals and small teams.

There's a catch, though. Published research found that roughly 26% of skills carry at least one vulnerability. Installing a random skill from GitHub means trusting an unverified instruction package with access to your agent, your files, and sometimes your credentials. As with models themselves, the license and provenance questions are easy to skip and painful to discover later.

A safe skills workflow looks like this:

• Discover skills from sources you can audit, not random pasted snippets.
• Verify each one — check the license, scan for security issues, and pin it to a specific commit so you know exactly what you installed.
• Benchmark it: test the same task with and without the skill, on the same model, before trusting it in production.
• Stay current — track upstream changes and re-scan when a skill updates.

That last point deserves emphasis. The same reasoning that applies to models applies to skills: benchmark the exact configuration you intend to deploy, because a skill that helps one model on one task can be neutral or harmful on another.

This is the problem SkillzBench is being built to solve. The platform — currently in development, with a waitlist open — will offer a free directory that continuously crawls and dedupes skills from GitHub, pins every version to a commit SHA, runs license checks and security scans, and benchmarks each skill in a sandbox to publish its measured Lift. A free MCP connector will let you attach verified skills to any agent, with per-agent loadouts so each agent only sees what you've chosen for it. Everything on the platform — directory, connector, benchmarks, alerts — will be free, and money never touches scores.

You built or borrowed a model. Skills are how it earns its keep — and how it does so safely. Join the SkillzBench waitlist to be first in line when the directory and MCP connector go live, and start putting your local model to real work with skills you can actually trust.