Can ChatGPT build an AI agent?

The Short Answer: ChatGPT as Your Agent's Co-Builder

Yes — ChatGPT can genuinely help you build an AI agent, though with an honest caveat: it's a drafting partner, not a deployment pipeline. The reason it works at all comes down to what an agent actually is. OpenAI's own practical guide defines agents as combinations of Model, Tools, and Instructions — and two of those three ingredients are fundamentally writing tasks.

Instructions are the clearest example. Modern agent skills are often just a markdown file — a SKILL.md instruction package plus optional scripts — sitting on a developer's machine. That's exactly the kind of artifact an LLM can generate. As one practitioner guide puts it, skills function like "onboarding documentation written specifically for your AI assistant," turning an agent that makes educated guesses into one that behaves like a developer already onboarded to your project.

Where ChatGPT fits in a configuration workflow:

• Drafting the instruction file itself — scope, conventions, tool-use rules, and failure-handling steps.
• Refining prompts iteratively, since agent quality often hinges on instruction clarity more than tool count.
• Describing tool connections in plain language before you wire them up via a protocol like MCP, the standardized "USB-C for AI tools" interface.
• Reviewing an existing skill's instructions for ambiguity or scope creep.

Set expectations honestly, though. ChatGPT can write the instructions; it cannot verify them, pin them to a version, or test what happens when they run. That matters because Cisco research found roughly 26% of 31,000 analyzed agent skills contained at least one vulnerability — and skills execute with the user's own permissions, not in a sandbox. A beautifully drafted instruction file is still untrusted input until someone checks it.

There's also a scope question. OpenAI's guide is explicit that customers "typically achieve greater success with an incremental approach," recommending you maximize a single agent's capabilities first before attempting complex multi-agent architectures. So the realistic first project isn't an autonomous swarm — it's one well-scoped agent with clear instructions, a handful of distinct tools, and a human reviewing the output.

This is where verification tooling earns its place. SkillzBench is building a free directory and benchmarking platform that pins every skill version to a commit SHA, runs security scans and license checks, and tests each skill in a sandbox to measure its actual lift — plus a free MCP connector so your agent only sees the skills you explicitly chose. Think of the division of labor this way: ChatGPT drafts, you decide, and independent verification closes the trust gap that naive LLM-generated skills leave wide open.

Why Naive LLM-Generated Skills Are a Security Gamble

If a skill can run code on your machine with your own login credentials, you'd want to know exactly who wrote it — and yet most people install agent skills with far less scrutiny than they'd give a random app. The numbers back up that instinct. When Cisco researchers analyzed 31,000 agent skills, roughly 26% contained at least one vulnerability — and that's just what static analysis caught.

The scale of the ecosystem makes this worse, not better. NSFOCUS Tianyuan Lab's research found the number of skills has exceeded 100,000, with 105,000+ related projects growing exponentially. But that growth has outpaced any safety infrastructure: the architecture "lacks unified, standardized distribution channels with security verification." There's no app-store review process here. Anyone can publish a skill, and nothing stops it from shipping alongside a script that runs shell=True commands or reads your environment files.

That last part matters more than most people realize. As Backslash Security puts it, "Agent Skills execute with the employee's own permissions" — there's no sandbox. A malicious skill doesn't need an exploit chain; it inherits whatever you can access, including credential stores, Git config, and env files. The review question isn't just "does this skill look malicious?" but "what can it reach?"

And here's where it gets uncomfortable for anyone relying on automated scanners. Security researchers at AIR built a deliberately fake skill called "brand-landingpage" that pointed to an external domain hosting genuine documentation at install time — then swapped that content post-install to serve malicious instructions. It passed every major scanner tested, including Cisco's, NVIDIA's, and those wired into skills.sh. Trail of Bits independently demonstrated the same bypass weeks earlier against ClawHub's detector and all three skills.sh scanners. The structural flaw is simple: a scanner checks a fixed package, while an attacker can keep tweaking the payload until it passes.

• ~26% of 31,000 analyzed skills contained at least one vulnerability
• The skills ecosystem has blown past 100,000 projects with no standardized security verification
• Skills execute with the user's own permissions — no sandbox isolation
• A fake skill passed every major scanner by swapping external content after install

This is the gap SkillzBench is being built to close: pinning every skill version to a commit SHA, re-scanning on upstream changes, and security-scanning before anything reaches your agent. Because when a skill can read your credentials, "probably fine" isn't a security posture — it's a gamble.

SkillzBench is currently in pre-launch. Join the waitlist to get early access to verified, benchmarked agent skills.

The Verification Workflow: Draft with ChatGPT, Deploy with Proof

So ChatGPT can draft your agent's instructions — but can you trust what it writes? The research says probably not on its own: Cisco's analysis of 31,000 skills found roughly 26% carried at least one vulnerability.

The answer is a two-stage pipeline: draft with the LLM, deploy only with proof. ChatGPT is genuinely good at the first stage. Agent Skills are, at their core, markdown instruction packages — onboarding documentation for your AI, written for tools like Claude Code, Codex, Cursor, and Gemini CLI. An LLM that understands your workflow can produce a competent SKILL.md draft in minutes.

The second stage is where most teams skip steps — and where the risk lives. Every artifact ChatGPT produces should be pinned to a specific commit SHA, security-scanned, license-checked, and benchmarked with and without the skill to measure whether it actually helps. A skill that adds no measurable Lift is dead weight even if it's harmless.

Why pin to a version, not a name? Because, as Backslash Security's guidance puts it, approval must attach to a specific version of the files — if the source refreshes itself, "the artifact that runs tomorrow may share nothing with the one that was examined except its label." A skill named "pdf-helper" that passed review last month is not the same artifact today.

External links demand even more vigilance. Security researchers demonstrated a fake skill that passed every major scanner — including Cisco's and NVIDIA's — by pointing to an external domain that hosted genuine docs at install time, then was swapped post-install to serve malicious instructions. The structural flaw: a scanner checks a fixed package, while an attacker keeps tweaking the payload until it passes.

A working verification workflow covers four gates:

• Pin each skill version to a commit SHA so approval refers to an exact artifact, not a mutable label
• Security-scan and license-check every draft before it reaches an agent
• Benchmark with and without the skill in a sandbox to quantify real Lift
• Continuously monitor any external URLs the skill references, since content can change after vetting

This is precisely the gap SkillzBench is being built to close. Its Verify pillar pins every skill version to a commit SHA, validates, license-checks, security-scans, and quality-judges each artifact, while its Benchmark pillar tests skills in a sandbox — same model, with and without the skill — and publishes the Lift in percentage points. And because upstream skills change, its monitoring tracks those changes, re-scans, and re-benchmarks automatically.

The takeaway for anyone configuring agents with ChatGPT's help: use the LLM as your drafting engine, never as your deployment gate. Skills, proven — that's the standard worth holding AI-generated configuration to.

Putting It Together: Your First ChatGPT-Assisted Agent in Five Steps

So you've decided ChatGPT can help you build an agent — but how do you actually do it without shipping a security risk? Here's a five-step walkthrough that treats ChatGPT as your drafting assistant, not your deployment pipeline.

Step 1: Give your agent one job. OpenAI's practical guide to building agents is blunt: customers succeed with an incremental approach, and you should maximize a single agent's capabilities before splitting into multi-agent systems. Pick one workflow — triaging tickets, summarizing documents — and resist the urge to build a fully autonomous system on day one.

Step 2: Have ChatGPT draft the stack. Every agent is Model + Tools + Instructions. Paste your task description into ChatGPT and ask it to draft the instructions file and tool list. Because agent skills are essentially markdown-plus-script packages, an LLM can write them — but treat the output as a draft, not a finished product.

Step 3: Source proven skills, not raw GitHub repos. Ecosystem research shows the number of skills has exceeded 100,000 with no standardized security verification, and Cisco's analysis found roughly 26% of 31,000 sampled skills carried at least one vulnerability. That's why SkillzBench is building a verified directory that pins every skill version to a commit SHA, runs security scans, license checks, and quality judging — rather than trusting whatever a random repository ships today.

Step 4: Wire tools via MCP, with a per-agent loadout. MCP acts as "USB-C for AI tools" — one standard interface connecting any agent to any tool server. The key discipline is scope: your agent should see only the tools you chose for it, nothing more. SkillzBench's free MCP connector and per-agent loadouts are being built around exactly this principle, so a summarization agent never sees, say, your credential store.

Step 5: Set up change alerts. Security researchers demonstrated that a skill passing every scanner at install time can be swapped post-vetting, and security reviewers argue approval must attach to a specific version, not a name. If an upstream skill edits itself, your agent is running code nobody re-verified. Continuous tracking and re-scanning closes that gap.

One more planning note: OpenAI is sunsetting Agent Builder and Evals on November 30, 2026, steering builders toward code-level workflows. That's a strong argument for portable, text-based configurations — markdown skills and MCP wiring you can move between Claude Code, Codex, Cursor, or Gemini CLI without rebuilding from scratch.

Skills, proven — browse SkillzBench's free directory and MCP connector to equip your first agent with verified, benchmarked skills.