Scaling and monitoring

Understanding Scaling in the Cloud

In cloud environments, scaling is about matching resources to demand, automatically and economically. With Linux-based systems, this usually means controlling how many instances (VMs or containers) you run and how much work each can handle.

Key dimensions:

• Vertical scaling (scale up/down)
  Changing the size of a single instance:
• More CPU, RAM, network, or disk IOPS on the same VM/container host.
• Example: Moving from t3.micro to t3.large on AWS.
• Pros: Simple, no app changes in many cases.
• Cons: Hardware limits; typically requires downtime or restart; less fault-tolerant.
• Horizontal scaling (scale out/in)
  Changing the number of instances:
• Add more Linux servers (VMs or containers) behind a load balancer.
• Pros: Better redundancy, can handle large growth, easier to do gradually.
• Cons: App must be designed for multiple instances (stateless or shared state).
• Reactive vs. proactive scaling
• Reactive: Scale when metrics cross a threshold (e.g., CPU > 70% for 5 minutes).
• Proactive: Scale based on predictable patterns (cron jobs, campaign launches, business hours) or forecasts.
• Stateful vs. stateless workloads
• Stateless services are much easier to scale horizontally:
• Session data stored in cookies, Redis, or a database, not in local files.
• No reliance on local /var/www/uploads for user data, for example.
• Stateful services:
• Databases, queues, and file stores need specialized mechanisms (replication, sharding, shared storage).

Scaling Patterns with Linux Systems

Load Balancers and Linux Nodes

In cloud setups, a load balancer distributes traffic across your Linux instances:

• Layer 4 vs Layer 7:
• L4: Operates on TCP/UDP level; faster, simpler (e.g., AWS NLB).
• L7: Understands HTTP/HTTPS; can route based on URL/headers (e.g., AWS ALB, Nginx).

Typical pattern:

1. Client → Load balancer (HTTPS).
2. Load balancer → Pool of Linux web/app servers.
3. Auto scaling group controls how many servers are in the pool.

On each Linux node, you typically run:

• A web server (nginx, apache2) or reverse proxy.
• An application server (Gunicorn, Node.js runtime, etc.).
• A monitoring agent (for metrics/logs).

Auto Scaling Groups (ASG-like Concepts)

Different cloud providers name them differently, but the idea is similar: tie a group of Linux instances to scaling rules.

Common components:

• Launch template / instance template / image:
• Reference a base Linux image (AMI, image, snapshot).
• Specify OS, instance type, security groups (firewall rules), startup script.
• Scaling policies:
• CPU-based (average CPU across instances).
• Request rate or queue length.
• Custom metrics (requests in progress, error rate).
• Health checks:
• Cloud health check: is the VM reachable?
• App health check: HTTP /health endpoint returns 200.
• Unhealthy instances are terminated and replaced.

Linux-specific considerations:

• Ensure idempotent startup scripts (cloud-init, user-data, systemd units) so instances join the cluster correctly on boot.
• Use configuration management tools or baked images to install and configure software.

Container-Based Scaling

If you run containers on Linux (e.g., Kubernetes, Docker Swarm, ECS, AKS, GKE):

• You scale pods/containers, not bare VMs (though node scaling still exists).
• Autoscaling often occurs at two levels:
• Pod-level: e.g., Kubernetes Horizontal Pod Autoscaler (HPA) scales number of pods for a deployment.
• Node-level: Cluster Autoscaler adds/removes Linux worker nodes based on pod scheduling needs.

For container scaling, Linux hosts must be:

• Properly sized (CPU/memory) and labeled/tainted if you separate workloads.
• Monitored for node pressure (memory, disk, inodes, PIDs) to prevent scheduling failures.

Scaling Databases and Storage

Although databases may be managed services, they still rely on Linux underneath.

Patterns:

• Read replicas for scale-out reads.
• Sharding for write scaling.
• Shared storage (EFS/NFS) or object storage (S3/GCS/Azure Blob) for files.

Linux perspective:

• Mount network file systems (nfs, cifs, efs) carefully:
• Use appropriate mount options (noatime, hard/soft, timeouts).
• Monitor iostat, df, and mount health.
• Avoid local-only storage for data that must survive instance termination.

Monitoring Fundamentals for Scalable Systems

Scaling decisions should be driven by reliable monitoring. Monitoring on Linux in the cloud has three primary categories:

1. Metrics (numerical time series)
2. Logs (discrete event records)
3. Traces (end-to-end request paths)

For this chapter we’ll focus mainly on metrics and logs, plus basic alerting.

Key Metrics for Scaling Decisions

System-Level Metrics (Per Linux Instance)

These come from Linux itself (kernel and /proc):

• CPU
• Overall utilization (% user, % system, % iowait).
• Load average from uptime or top (1m, 5m, 15m).
• Context switches, interrupts (for advanced tuning).
• Memory
• Used/free memory, buffers, cache (free -m, /proc/meminfo).
• Swap usage and page faults.
• OOM killer events (check system logs).
• Disk and I/O
• Disk utilization, read/write IOPS, throughput (iostat, dstat, /sys/block).
• Filesystem usage (df -h).
• Inodes usage (df -i).
• Network
• Throughput (bytes in/out), packets in/out.
• Errors, drops, retransmissions.

On cloud platforms, many of these are exposed through the provider’s monitoring service (e.g., CloudWatch, Azure Monitor, Cloud Monitoring) as instance metrics.

Application-Level Metrics

These derive from your Linux-based application:

• Request rate: requests per second ($RPS$).
• Latency:
• Common quantiles: $P50$, $P90$, $P95$, $P99$.
• Error rate:
• 4xx, 5xx HTTP responses, timeouts.
• Throughput:
• Items processed/sec, jobs/sec, queries/sec.
• Queue lengths:
• If using message queues or background jobs.

These metrics are often instrumented with tools like Prometheus client libraries or application-specific exporters and scraped/collected from processes running on Linux.

Capacity and Saturation Metrics

Useful for scaling:

• CPU saturation:
• Average CPU > threshold (e.g., 70%) for a period.
• Request queue backlog:
• Current queue length / desired queue size.
• Number of active connections:
• From ss, netstat, or app server metrics.
• Worker utilization:
• For process managers like gunicorn, php-fpm, systemd services.

These form the basis for autoscaling policies (e.g., scale out when average CPU > 65% for 10 minutes).

Monitoring Tools and Agents on Linux

Although each cloud provider has native monitoring, on Linux you typically use an agent or exporter:

• Cloud vendor agents
• AWS CloudWatch Agent (Linux rpm/deb).
• Azure Monitor Agent.
• Google Cloud Ops Agent.
• Installed via package manager, configured with a config file, run as a systemd service.
• Open-source monitoring stacks
• Prometheus Node Exporter: exposes Linux metrics via HTTP.
• Prometheus server: scrapes metrics from exporters.
• Grafana: dashboards.
• Alertmanager: handles alerts.
• Collectd, Telegraf: collect metrics and send to backends (InfluxDB, Graphite, etc.).

Linux specifics:

• Run exporters/agents as system services and monitor their own health.
• Use OS packages (apt, dnf, pacman) or container runtimes to manage them.

Logs and Centralized Logging

As you scale to many Linux instances, logs must be aggregated centrally; reading /var/log on each instance becomes impractical.

Local Logs on Linux Servers

Typical sources:

• System logs:
• journalctl (systemd journal).
• Files under /var/log (e.g., /var/log/syslog, /var/log/messages, /var/log/auth.log).
• Application logs:
• Web server logs: access.log, error.log.
• Custom app logs: usually under /var/log/<app> or /var/www/<app>/logs.

Standard practices:

• Use structured logging (JSON where possible).
• Include:
• Timestamp (UTC).
• Request ID / correlation ID.
• Severity level (INFO, WARN, ERROR).
• Service name, instance ID/hostname.

Centralizing Logs from Multiple Linux Hosts

Common approaches:

• Cloud-native log services:
• AWS CloudWatch Logs, Azure Log Analytics, Google Cloud Logging.
• Agents tail log files or read from the systemd journal and ship entries to the service.
• Self-managed log stacks (ELK/EFK)
• Elasticsearch + Logstash + Kibana (ELK).
• Elasticsearch + Fluentd/Fluent Bit + Kibana (EFK).
• Agents (filebeat, fluent-bit, fluentd) run on each Linux node and send logs to a central store.

Benefits of centralized logging:

• Can search logs across all instances.
• Correlate logs with metrics and traces.
• Build alerts based on log patterns (e.g., spike in 5xx errors).

Linux considerations:

• Make sure log rotation is working (logrotate or systemd journal retention) to prevent /var from filling.
• Tune journal settings in /etc/systemd/journald.conf when using persistent logs.

From Monitoring to Alerting

Monitoring is useful only if it leads to meaningful actions. Alerts connect metrics/logs to people or automated systems.

Designing Effective Alerts

Good alerts:

• Are tied to user impact or system health relevant to SLOs/SLAs.
• Are actionable: someone can fix or mitigate the problem.
• Avoid constant false positives (alarm fatigue).

Categories:

• Critical alerts: require immediate response.
• Service totally unavailable.
• Latency or error rate breaches SLO for several minutes.
• Disk almost full on a core database node.
• Warning alerts: non-urgent, but need attention.
• CPU/Memory trending upwards.
• Error rate above baseline but still within limits.

Common examples for Linux cloud setups:

• HighCPUUtilization:
• Condition: CPU > 80% for 10 minutes across majority of instances.
• Action: Trigger autoscaling and/or notify on-call.
• HTTP5xxErrorRateHigh:
• Condition: 5xx error rate > 2% of all requests for 5 minutes.
• Action: Notify on-call, open incident.
• DiskSpaceLow:
• Condition: / or /var filesystem usage > 85% for 15 minutes.
• Action: Notify and/or trigger automated cleanup job.

Alert Targets and Integrations

Alerts can integrate with:

• Email, SMS.
• Chat tools (Slack, Microsoft Teams).
• Incident management tools (PagerDuty, Opsgenie).

For automated scaling actions, alerts may instead trigger:

• Cloud-native scaling actions (built-in).
• Webhooks to orchestration tools.
• Lambda/Functions that adjust capacity.

Autoscaling Strategies Based on Monitoring

Scaling rules are expressed in terms of monitored metrics.

CPU-Based Scaling

Common and simple:

• Rule example:
• If average CPU across group > 70% for 5 minutes → add instance.
• If average CPU < 30% for 15 minutes → remove instance.

Pros:

• Easy to understand and implement.
• Works well for CPU-bound workloads.

Cons:

• Not ideal for I/O-bound or latency-sensitive workloads.
• Might react slowly if average CPU hides hotspots.

Request/Latency-Based Scaling

Ties scaling to user experience:

• Request rate:
• Target: maintain $X$ requests per instance, e.g., 100 RPS/instance.
• When load increases beyond this, add instances.
• Latency:
• Target: $P95$ latency < 200 ms.
• If $P95$ latency exceeds target for a period, scale out.

This often requires application-level metrics and possibly custom scaling metrics.

Queue-Based Scaling

For background jobs:

• Monitor queue depth (e.g., number of messages in SQS, Pub/Sub, RabbitMQ).
• Scale number of worker instances/pods based on:
• Queue depth.
• Oldest message age.

Basic formula:

$$
N_{\text{workers}} = \left\lceil \frac{\text{DesiredProcessingRate}}{\text{ProcessingRatePerWorker}} \right\rceil
$$

Where:

• $\text{DesiredProcessingRate}$ is jobs/sec you need to handle.
• $\text{ProcessingRatePerWorker}$ is jobs/sec per Linux worker process.

Queue-based autoscaling ensures backlog doesn’t grow unbounded.

Scaling Cooldowns and Stability

To avoid oscillation (scale out, then in, then out again):

• Use cooldown periods:
• Wait $N$ minutes after a scale event before another event in the same direction.
• Use moving averages of metrics or percentiles rather than instantaneous values.
• Define minimum and maximum instance counts:
• Ensure at least one or a few instances always run.
• Cap at a cost or capacity limit.

Designing Linux Systems for Scalable Monitoring

As you deploy more Linux instances, the monitoring system itself must scale.

Agent Management at Scale

For hundreds or thousands of Linux hosts:

• Use:
• Configuration management (Ansible, Puppet, Chef) to deploy and configure agents.
• Base images (golden images, AMIs) with agents pre-installed.
• Ensure:
• Agents auto-start on boot (systemd units).
• Configuration is environment-aware (dev/stage/prod).

Metric Storage and Retention

Time-series data can grow quickly:

• Choose retention per environment:
• High resolution (10–30s) for a few days.
• Aggregated rollups (5–15 min intervals) for weeks or months.
• Partition by:
• Environment, service, region.
• For self-managed Prometheus:
• Use federation or remote storage when scaling beyond a single instance’s capacity.

Dashboards and Visualization

Dashboards help you reason about scaling:

• Build:
• Service overview: error rate, latency, request rate, instance count.
• Infrastructure overview: CPU/memory/disk across nodes.
• Per-node dashboards for deep investigations.
• Dashboards should be:
• Template-based (filtered by environment, service).
• Checked regularly, not only during incidents.

Capacity Planning with Metrics

Autoscaling handles short-term variability, but you still need medium/long-term capacity planning.

Estimating Capacity Per Linux Instance

Using load testing and metrics:

1. Run load tests on a single instance.
2. Measure:
• Max sustained RPS at acceptable latency and error rate.
• Corresponding CPU, memory, and I/O usage.
3. Derive:
• Capacity per instance, e.g., 150 RPS at 60% CPU and $P95$ latency of 200 ms.

Then, for a target $RPS_{\text{total}}$:

$$
N_{\text{instances}} = \left\lceil \frac{RPS_{\text{total}}}{RPS_{\text{per instance}}} \right\rceil \times S
$$

Where $S$ is a safety factor (e.g., 1.2) to account for spikes and node failures.

Budgeting and Cost Awareness

Monitoring can show:

• Average and peak instance counts.
• Resource utilization vs. overprovisioning.

Use these data to:

• Adjust instance types (right-size VMs).
• Decide between fewer large instances vs many small ones.
• Optimize scaling thresholds to reduce cost while maintaining performance.

Putting It All Together: A Typical Workflow

1. Instrument your Linux services:
• System metrics via agents or exporters.
• Application metrics (request rate, latency, errors).
• Structured logs with relevant context.
2. Centralize:
• Send metrics to a time-series database or cloud monitor.
• Aggregate logs into a centralized logging system.
3. Visualize:
• Build dashboards for infrastructure and per-service views.
4. Define SLOs:
• Availability, latency targets.
• Error rate thresholds.
5. Configure alerts:
• Tie alerts to SLOs and core system health.
• Integrate with on-call tools and chat.
6. Implement autoscaling:
• Use monitoring metrics to drive scaling policies.
• Validate with load tests.
7. Review and iterate:
• After incidents or large traffic events, review metrics and logs.
• Refine thresholds, capacity estimates, and architectural assumptions.

By combining robust monitoring with thoughtful scaling policies, your Linux-based cloud systems can stay reliable, performant, and cost-effective as demand grows and changes.