End-to-End Machine Learning on AWS with S3 and EC2

Problem

In most ML tutorials, everything runs on your laptop. The dataset is local. The model is local. If your machine crashes, you lose everything. That setup breaks down the moment you need to train on data too large for one disk, collaborate with a team, or deploy predictions to an application. The real ML workflow — the one used in production — looks different: data lives in durable cloud storage, training happens on elastic compute, and the trained model is persisted independently of any single server.

The challenge here was to build that workflow from scratch using AWS. Starting from nothing — no data, no server, no permissions — the goal was to arrive at a trained diabetes prediction model stored in S3, accessible from anywhere, independent of whether the EC2 instance is even running.

Solution

The pipeline wires together four AWS services into a complete ML workflow: S3 for durable object storage (dataset in, model out), IAM for credential-free access between services, VPC for a private, internet-connected network, and EC2 for on-demand compute. The training itself — a scikit-learn Random Forest classifier on the Pima Indians Diabetes dataset — runs entirely on the cloud server via the AWS CLI. No local training. No manual file transfers. Every piece is wired by AWS services.

Skills Acquired

• Amazon S3 — created a bucket, organized it with datasets/ and models/ prefixes, uploaded data, and stored a trained model — understanding S3 as durable, cloud-native object storage independent of compute.
• AWS IAM — created an EC2-specific IAM role (EC2-S3-Access-Role) with AmazonS3FullAccess, attached it at instance launch time — no access keys, no hardcoded credentials.
• Amazon VPC — provisioned a private network with 1 AZ, 1 public subnet, an Internet Gateway, and route tables using the "VPC and more" wizard, understanding what each resource does and why it's needed.
• Amazon EC2 — launched a t2.micro Amazon Linux 2023 instance inside the VPC, attached the IAM role, enabled a public IP, and connected via EC2 Instance Connect — no SSH key required.
• AWS CLI — used aws s3 ls, aws s3 cp from the EC2 terminal to move data between S3 and the instance — the bridge between compute and storage.
• scikit-learn — trained a RandomForestClassifier (100 estimators, max depth 10) on the Pima Indians Diabetes dataset, evaluated with accuracy score and classification report, and serialized the model with pickle.
• Python on EC2 — installed Python packages (pandas, scikit-learn) on a fresh Amazon Linux instance and ran a full ML training pipeline from the command line.
• Cloud ML architecture — internalized the separation of concerns: S3 is permanent storage, EC2 is temporary compute, IAM is the glue that connects them securely.

Deep Dive

The Pima Indians Diabetes dataset is a classic binary classification benchmark: 768 patient records, 8 clinical features, and a binary target indicating diabetes diagnosis. It's small enough to run anywhere — but training it on EC2 with data sourced from S3 is the point. The dataset is just the payload. The real lesson is the infrastructure: how do you move data securely between services? How does a server prove it has permission to read from a storage bucket without a password?

This is the same pattern used in every production AWS ML pipeline. Whether you're training on a t2.micro or a p4d.24xlarge GPU instance, the architecture is identical: an IAM role grants the instance access to S3, the instance pulls data, trains, and pushes the model artifact back. Swapping the instance type is a one-line change. The security and data flow don't change at all.

Why This Project?

This was Sprint 13 of the AWS Cloud Institute program — the capstone lab that ties together S3, IAM, VPC, and EC2 into a single coherent workflow. After individual labs on each service in isolation, this sprint required wiring them all together: storage, permissions, networking, and compute operating as one system.

The skills built here — S3 data management, IAM role configuration, VPC setup, EC2 provisioning — are foundational to any cloud-deployed ML system. Understanding how these services interact is what separates someone who can run a Jupyter notebook from someone who can deploy and maintain an ML system that actually stays running.

What You'll Learn from This

• How IAM roles grant EC2 instances access to S3 without embedding credentials — and why this is the correct approach
• What a VPC actually is and why EC2 instances need one to communicate with the internet
• The difference between S3 (permanent storage) and EC2 (temporary compute) — and why this separation matters for ML pipelines
• How to use the AWS CLI to move data between services from a running EC2 terminal
• How to install Python packages, run a training script, and serialize a model on a cloud server
• Why saving the model to S3 after training is critical — and what happens if you don't

Key Takeaways

• S3 + EC2 + IAM is the canonical AWS ML pattern. Every production ML workflow on AWS uses some version of this — even SageMaker uses it under the hood.
• Model artifacts live in S3, not on EC2. Terminating the EC2 instance doesn't lose the model. S3 is 99.999999999% durable; EC2 instance storage is ephemeral.
• IAM roles eliminate credential management. No access keys, no secret rotation, no leaked credentials in git history. The role is the credential.
• VPC gives you network isolation. The EC2 instance ran in a private network with a controlled path to the internet — the internet gateway and route table define that path explicitly.
• EC2 Instance Connect removes the key pair requirement. Browser-based SSH via the AWS console, gated by IAM — no PEM file to manage or lose.
• Auto-grader validated every layer: 274/274 (100%). VPC, IAM, EC2 config, SSH access, Python environment, and both S3 artifacts all passed.

The Dataset

The Pima Indians Diabetes dataset is a widely used binary classification benchmark from the UCI Machine Learning Repository. It contains 768 patient records, all female patients of Pima Indian heritage, with 8 clinical features and a binary outcome variable.

My Process

# List bucket contents — confirms IAM role is working
aws s3 ls s3://vietnguyen-ml-project-3421/

# Expected output:
#   PRE datasets/
#   PRE models/

sudo dnf install -y python3 python3-pip
pip3 install --upgrade pip
pip3 install pandas scikit-learn

mkdir -p ~/ml-project/data ~/ml-project/models

aws s3 cp s3://vietnguyen-ml-project-3421/datasets/diabetes.csv \
  ~/ml-project/data/

# Verify — should show diabetes.csv
ls ~/ml-project/data/
head ~/ml-project/data/diabetes.csv

import pandas as pd
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, classification_report
import pickle

# Load dataset
df = pd.read_csv('/home/ec2-user/ml-project/data/diabetes.csv')
print(f"Dataset shape: {df.shape[0]} rows, {df.shape[1]} columns")

# Features and target
X = df.iloc[:, :-1]   # all columns except Outcome
y = df.iloc[:,  -1]   # Outcome column

# 80/20 split
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Train Random Forest
model = RandomForestClassifier(
    n_estimators=100, max_depth=10, random_state=42
)
model.fit(X_train, y_train)

# Evaluate
y_pred   = model.predict(X_test)
accuracy = accuracy_score(y_test, y_pred)
print(f"Test Accuracy: {accuracy:.2%}")
print(classification_report(y_test, y_pred,
      target_names=['No Diabetes', 'Diabetes']))

# Feature importance rankings
importances   = model.feature_importances_
feature_names = df.columns[:-1].tolist()
for name, imp in sorted(zip(feature_names, importances),
                         key=lambda x: x[1], reverse=True):
    print(f"  {name}: {imp:.4f}")

# Save model
model_path = '/home/ec2-user/ml-project/models/diabetes_model.pkl'
with open(model_path, 'wb') as f:
    pickle.dump(model, f)
print(f"Model saved to: {model_path}")

python3 ~/ml-project/train.py

aws s3 cp ~/ml-project/models/diabetes_model.pkl \
  s3://vietnguyen-ml-project-3421/models/

# Verify from EC2 — confirms model landed in S3
aws s3 ls s3://vietnguyen-ml-project-3421/models/
# Output: 2026-05-21 ... 646390 diabetes_model.pkl

Final Results

Main Takeaways

• S3 is for storage; EC2 is for compute — never conflate them. Data and models live in S3. Computation happens on EC2. When compute is done, you terminate it. The data persists regardless.
• IAM roles are the right way to grant access between AWS services. Hardcoding access keys is a security anti-pattern that surfaces credentials in code and requires manual rotation. Roles provide temporary, automatically rotated credentials through the metadata service — no keys, no risk.
• The VPC is the network fabric. Without the Internet Gateway and public subnet route, the EC2 instance has no path to download packages or connect via EC2 Instance Connect — the VPC configuration isn't optional boilerplate; it's load-bearing infrastructure.
• EC2 Instance Connect eliminates key pair management. For an IAM-authenticated team, browser-based SSH gated by IAM permissions is operationally cleaner than distributing PEM files and managing key rotation.
• This pattern scales linearly. Swap the t2.micro for a p4d.24xlarge and the architecture is identical. The IAM role, the S3 paths, the AWS CLI commands — nothing changes. The compute scales; the data pipeline doesn't need to be redesigned.

Conclusion & Reflections

The Pima Indians Diabetes dataset is a means to an end in this project. The real deliverable is the infrastructure: a reproducible, secure, cloud-native workflow where data and model artifacts are durably stored in S3, accessible to any authorized compute in any region, without a single credential in code. That's the pattern that shows up in every serious ML deployment.

If I were extending this project, the next step would be automating the pipeline — replacing the manual EC2 steps with an AWS Step Functions workflow or a SageMaker Pipeline that triggers on new data in S3, trains, evaluates, and conditionally registers the model in the SageMaker Model Registry. The architecture I built here is the foundation that makes that automation possible.