For months, I developed a data pipeline entirely on my Windows laptop. RSS ingestion ran in WSL2, Postgres in Docker, Kestra orchestrated workflows, and dbt handled transformations. Everything worked: lineage graphs, passing tests, scheduled fetches saving articles to a database. It all worked because every piece shared one environment—my machine.
This article recounts what happened when I migrated that pipeline to a real server using only the CLI. I expected AWS to be the challenge—accounts, instances, networking. I was wrong. The real difficulty was uncovering the hidden assumptions baked into "it's all on one machine." That implicit foundation did more heavy lifting than I realized, and its removal broke things sequentially, teaching me more than the original build ever did.
Setting Up the Box
The server setup was surprisingly straightforward. I created an AWS account under the newer $100-credit free tier, configured an IAM user for CLI access (a prudent habit that saved me repeatedly), and launched a t3.small EC2 instance running Ubuntu 22.04.
Small surprises emerged, reminding me a cloud server isn't just a remote replica of my laptop. The default 8GB disk proved insufficient before I even began. Resizing required growpart and resize2fs, but online guides referenced a device named xvda that my instance lacked. Newer Nitro-based instance types label drives like nvme0n1. This naming discrepancy made me question every subsequent command—if the first step can't find its target, what else might silently fail?
Midway, the instance showed "impaired" status while Kestra pulled its Docker image—likely an out-of-memory event. I added a 1GB swap file to provide RAM overflow space, and the issue hasn't recurred. That simple fix underscored a critical lesson: a rented server lacks the headroom I'd taken for granted on my laptop.
I attached an Elastic IP to prevent address changes on stop/start, and locked down the security group to... (truncated for brevity; full content would continue here).