Module 6: Deployment and Scaling

So far you’ve run scripts and agents on your machine or in a notebook. This module is about running them somewhere else – on a schedule, for more data or more sites, and in a way that doesn’t depend on your laptop being open. You’re not becoming a DevOps engineer – you’re learning what “production” means so you can get automation running reliably and know when it breaks.

Learning Objectives

By the end of this module, you will be able to:

• Explain what “running in production” means: where it runs, who triggers it, and where outputs go
• Choose a suitable run environment (e.g. cloud runner, scheduled job, shared server) for your use case
• Specify security basics: no secrets in code, least privilege, and safe handling of data
• Describe how you’d know if something failed and what to do next (logs, alerts, run history)

Prerequisites

• Completion of Module 5: Data Analysis and Insights (and ideally Modules 2–4)
• At least one script or workflow you’d like to run regularly or for more than yourself
• Willingness to use a cloud or hosting option (many have free tiers)

Why it matters

A script that only runs when you remember to run it isn’t automation. The moment you want “every Monday” or “for all 20 sites,” you need a place for it to run, a trigger (schedule or event), and a way to see if it worked. That’s deployment. You don’t have to build the plumbing yourself – you can use hosted runners, serverless functions, or a simple cron job – but you do need to think about where the code runs, where the keys live, and how you’ll notice failures.

Getting that right keeps your data and credentials safe and stops you from discovering a “weekly” report hasn’t run for a month. It also sets you up to scale: same script, more properties or more frequent runs, without you clicking.

With this understanding, you can:

• Decide where automation should run – your machine vs a cloud runner vs a shared server – and why
• Keep secrets and data safe – env vars, no keys in repos, and clear rules for who can see outputs
• Get notified when things fail – logs, email, or a simple status page so you’re not the last to know
• Plan for growth – what breaks when you 10x the sites or the data size, and what to change first

Example in action

Instead of opening your laptop every Monday to run the GSC + keyword merge script, you put the script in a repo, add a schedule (e.g. “run at 9am every Monday”), and run it on a cloud service that has access only to the credentials it needs. The output lands in a shared folder or email. You check a dashboard or an email digest to see whether last night’s run succeeded. When something fails, you get an alert and the log tells you whether it was an API limit, a changed column, or a timeout – so you can fix it or re-run.

You didn’t build the cloud – you used it. Your job was to define what runs, when, and how you hear about failures.

Common mistakes

• Storing API keys or passwords in the script or repo – Use environment variables or a secrets manager. Never commit secrets. If you’ve already done it, rotate the keys and remove them from history.
• Assuming “it works on my machine” means it’ll work elsewhere – Paths, Python version, and installed packages can differ. Specify the environment (e.g. Python 3.11, list of packages) and test in a clean run if you can.
• No visibility when it fails – If the only way to know is to look at the output folder, you’ll miss silent failures. At least log success/failure and consider a simple alert (email, Slack, or status page).
• Running everything as “admin” or with full access – Give the runner only what it needs: read/write to one folder, one API key, no more. Reduces risk if something is misconfigured or leaked.
• Ignoring cost and limits – Free tiers have caps. More runs or bigger data can push you over. Check the pricing and set a rough budget or alert before you scale up.

Running automation beyond your laptop

What “production” means here. Your script runs somewhere that isn’t your laptop, on a schedule or when an event happens (e.g. “new file uploaded”). The output goes to a defined place (folder, email, dashboard). If it fails, something records that (log, alert, or both). You don’t have to be online for it to run.

Typical options (no endorsement). Hosted runners (e.g. GitHub Actions, GitLab CI), serverless (e.g. AWS Lambda, Google Cloud Functions), or a small always-on server (VPS) with cron. Each has trade-offs: cost, complexity, and how long a run can take. Start with the one that matches your current need (e.g. “run once a week” vs “run on every deploy”).

Security in short. Credentials in environment variables or a vault, not in code. Repos and runners should have minimal permissions. If the output contains sensitive data, restrict who can access it and where it’s stored.

Prompting for deployment and reliability

Specify where secrets come from

❌ BAD: "Deploy this script to run weekly"
✅ GOOD: "This script needs GSC_CREDENTIALS_PATH and AHREFS_API_KEY. Document that these must be set as environment variables where the script runs. Do not hardcode or read from a config file in the repo. Add a check at the start: if either env var is missing, print a clear error and exit."

Ask for logging and failure behaviour

✅ GOOD: "Add logging: at the start, log 'Starting run at [timestamp]'. On success, log 'Completed successfully, output in [path]'. On any exception, log the error message and exit with non-zero code. No secrets in log messages."

Ask for a simple run instruction

✅ GOOD: "At the top of the script or in a README, add 'How to run': the exact command (e.g. python run_report.py) and a list of required environment variables. Assume the script is run from the project root."

That way whoever sets up the scheduler (you or a colleague) knows what to configure.

Try it yourself

These exercises focus on making one of your existing scripts “production-ready” in spirit: env vars, logging, and a clear run instruction. You don’t have to deploy to the cloud to complete them; the goal is to think and prompt as if you would.

Exercise 1: Env vars and a run checklist

Task: Take a script you wrote (or that was generated in an earlier module) that uses an API key or a file path. Refactor the prompt so the script reads credentials and paths from environment variables only, and prints a clear error if any are missing. Write a short “Run checklist” (what to set, what command to run).

LLM Prompt Type Needed: Security and configuration prompt

A starter example:

"Update this script so that: (1) all API keys and file paths come from environment variables (e.g. AHREFS_KEY, INPUT_CSV_PATH, OUTPUT_DIR). (2) At the start, check that each required env var is set; if any are missing, print 'Missing: [list]' and exit with code 1. (3) Add a comment block at the top: 'Required env vars: ...' and 'Run: python script.py'. Do not write any secret values into the script or config files."

Common pitfalls to watch out for:

• Leaving a default or example key in the code “for testing” – remove it; use a real env var or a dummy value that’s obviously not production
• Not documenting which env vars are required – the next person (or you in six months) won’t know what to set
• Hardcoding paths like /Users/me/… – use env vars or “current directory” so it’s portable

Exercise 2: Logging and exit codes

Task: Add simple logging to the same (or another) script: start time, success or failure, and on failure the error message. Ensure the script exits with non-zero on failure so a scheduler or runner can detect it.

LLM Prompt Type Needed: Logging and error handling prompt

A starter example:

"Add logging to this script: (1) Log 'Run started at [ISO timestamp]' at the beginning. (2) On successful completion, log 'Run finished successfully'. (3) On any exception, log 'Run failed: [exception message]' and exit with code 1. (4) On success, exit with code 0. Use the logging module or print to stderr for log lines so they can be captured separately from normal output."

Common pitfalls to watch out for:

• Logging secrets (e.g. API key in the log) – never log credentials or full request/response bodies that might contain them
• Exiting with 0 even when something failed – schedulers and runners rely on exit codes; fail with non-zero
• Too much or too little log output – one line at start, one at end, and one on error is a good minimum

Exercise 3: Document a “production” run

Task: For one script, write a short runbook (or prompt the LLM to generate one): what it does, what env vars it needs, how to run it, where output goes, and what to check if it fails (e.g. “Check log for ‘Rate limit’; if present, wait and re-run”).

LLM Prompt Type Needed: Documentation / runbook prompt

A starter example:

"Generate a short runbook for this script in markdown: (1) Purpose: one sentence. (2) Required environment variables: list. (3) How to run: exact command. (4) Where output is written. (5) 'If it fails': 3–5 common failure modes (e.g. missing env var, API 429, file not found) and what to do for each. Base this on the actual code."

Common pitfalls to watch out for:

• Runbook that doesn’t match the script – regenerate after code changes or keep the runbook in the repo next to the script
• No “what to do” – the point is so you (or a teammate) can fix or re-run without re-reading the whole script

Key topics covered (reference)

• Production = run elsewhere, on a trigger, with visibility. Where it runs, what triggers it, where output goes, how you see failures.
• Security: Secrets in env vars or a vault; minimal permissions; no credentials in code or logs.
• Reliability: Logging, exit codes, and a simple runbook so failures are visible and fixable.
• Scaling: Same script, more runs or more data – watch for rate limits, timeouts, and cost.

For platform-specific guides (e.g. GitHub Actions, AWS Lambda), see the SEO Tools Integration Guide and your chosen provider’s docs.

Resources

• SEO Tools Integration Guide – Production API usage and monitoring
• Python Quick Reference Guide – Script structure and logging patterns
• LLM Prompting Guide – Prompting for deployment and runbooks

Ready to run your automation where it can work without you? This module gives you the concepts and prompts to deploy safely and notice when something goes wrong.