ToTo
TT

Google Cloud: Network, Data, and Container Management

· 16 min read
google cloudnetworkingbigquerycloud sqlkubernetesmonitoringiamdevelopment
Google Cloud: Network, Data, and Container Management

Google Cloud Development Essentials: Network, Data, and Container Management

This walkthrough explores essential Google Cloud Platform development skills through a series of hands-on labs. The progression moves from foundational networking and identity management through data processing and container orchestration, establishing the development expertise required for building scalable cloud applications.

Each lab demonstrates practical development patterns while building toward comprehensive cloud architectures that address real-world requirements for performance, scalability, and operational excellence.

Lab 1: Develop your Google Cloud Network

Network architecture forms the backbone of cloud applications. This foundational lab establishes advanced networking concepts and implementation patterns that support modern application development requirements.

Implementation Strategy

Advanced VPC Architecture Design:

# Create custom VPC with strategic IP planning
gcloud compute networks create development-vpc \
  --subnet-mode=custom \
  --bgp-routing-mode=global

# Create multi-tier subnet architecture
gcloud compute networks subnets create web-tier-subnet \
  --network=development-vpc \
  --range=10.1.1.0/24 \
  --region=us-central1 \
  --enable-private-ip-google-access

gcloud compute networks subnets create app-tier-subnet \
  --network=development-vpc \
  --range=10.1.2.0/24 \
  --region=us-central1 \
  --enable-private-ip-google-access

gcloud compute networks subnets create data-tier-subnet \
  --network=development-vpc \
  --range=10.1.3.0/24 \
  --region=us-central1 \
  --enable-private-ip-google-access

Strategic Firewall Rule Implementation:

# Create hierarchical firewall rules for application tiers
gcloud compute firewall-rules create allow-web-tier \
  --network=development-vpc \
  --allow=tcp:80,tcp:443 \
  --source-ranges=0.0.0.0/0 \
  --target-tags=web-servers \
  --description="Allow public HTTP/HTTPS to web tier"

gcloud compute firewall-rules create allow-app-tier \
  --network=development-vpc \
  --allow=tcp:8080,tcp:9000 \
  --source-tags=web-servers \
  --target-tags=app-servers \
  --description="Allow web tier to app tier communication"

gcloud compute firewall-rules create allow-data-tier \
  --network=development-vpc \
  --allow=tcp:3306,tcp:5432 \
  --source-tags=app-servers \
  --target-tags=database-servers \
  --description="Allow app tier to database communication"

Cloud Router and NAT Configuration:

# Configure Cloud Router for advanced routing capabilities
gcloud compute routers create development-router \
  --network=development-vpc \
  --region=us-central1 \
  --bgp-asn=65001

# Create Cloud NAT for secure outbound connectivity
gcloud compute routers nats create development-nat \
  --router=development-router \
  --region=us-central1 \
  --nat-all-subnet-ip-ranges \
  --auto-allocate-nat-external-ips \
  --min-ports-per-vm=64

Development Architecture Principles

Scalable Network Design: Multi-tier subnet architecture enables independent scaling of application components while maintaining security boundaries and performance optimisation.

Security-First Networking: Hierarchical firewall rules implement defense-in-depth principles with explicit trust relationships between application tiers.

Global Connectivity Planning: BGP routing configuration supports multi-region deployments and hybrid cloud architectures for enterprise-scale applications.

Lab 2: Cloud IAM Quickstart

Identity and access management provides the security foundation for all cloud development activities. This lab establishes practical IAM patterns that scale from development through production environments.

Implementation Strategy

Development Team IAM Structure:

# Create development project with appropriate IAM policies
gcloud projects create dev-project-$(date +%s) \
  --name="Development Environment" \
  --organization=$ORG_ID

# Assign development team roles with appropriate permissions
gcloud projects add-iam-policy-binding $DEV_PROJECT_ID \
  --member="group:developers@company.com" \
  --role="roles/editor"

gcloud projects add-iam-policy-binding $DEV_PROJECT_ID \
  --member="group:devops@company.com" \
  --role="roles/compute.admin"

Service Account Management for Applications:

# Create application-specific service accounts
gcloud iam service-accounts create web-app-service \
  --display-name="Web Application Service Account" \
  --description="Service account for web application components"

gcloud iam service-accounts create data-processor-service \
  --display-name="Data Processing Service Account" \
  --description="Service account for data processing workloads"

# Assign minimal required permissions
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:web-app-service@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/storage.objectViewer"

gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:data-processor-service@$PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/bigquery.dataEditor"

IAM Conditional Access Implementation:

# Create conditional IAM policy for time-based access
cat > conditional-policy.json << EOF
{
  "bindings": [
    {
      "role": "roles/compute.instanceAdmin",
      "members": ["user:admin@company.com"],
      "condition": {
        "title": "Business Hours Access",
        "description": "Access limited to business hours",
        "expression": "request.time.getHours() >= 9 && request.time.getHours() <= 17"
      }
    }
  ]
}
EOF

gcloud projects set-iam-policy $PROJECT_ID conditional-policy.json

Development Architecture Principles

Role-Based Development Access: Structured IAM policies enable secure collaboration while maintaining appropriate separation of concerns between development, operations, and security teams.

Service Account Security: Application-specific service accounts with minimal permissions support secure automation and deployment patterns.

Conditional Access Control: Advanced IAM features enable dynamic access policies that adapt to operational requirements and security constraints.

Lab 3: Introduction to SQL for BigQuery and Cloud SQL

Data management forms the core of modern applications. This lab demonstrates practical SQL implementations across Google Cloud’s managed database services for both analytical and transactional workloads.

Implementation Strategy

BigQuery Data Warehouse Setup:

# Create BigQuery dataset for analytics workloads
bq mk --dataset \
  --description="Production analytics dataset" \
  --location=US \
  $PROJECT_ID:analytics_warehouse

# Create table with optimised schema design
bq mk --table \
  $PROJECT_ID:analytics_warehouse.user_events \
  user_id:STRING,event_type:STRING,timestamp:TIMESTAMP,properties:JSON

Advanced BigQuery SQL Operations:

-- Create materialised view for performance optimisation
CREATE MATERIALIZED VIEW `analytics_warehouse.daily_user_metrics`
PARTITION BY DATE(event_date)
CLUSTER BY user_segment
AS
SELECT
  DATE(timestamp) as event_date,
  user_segment,
  COUNT(DISTINCT user_id) as active_users,
  COUNT(*) as total_events,
  APPROX_QUANTILES(session_duration, 100)[OFFSET(50)] as median_session_duration
FROM `analytics_warehouse.user_events`
WHERE timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 90 DAY)
GROUP BY event_date, user_segment;

-- Implement data quality checks
CREATE OR REPLACE TABLE `analytics_warehouse.data_quality_metrics` AS
SELECT
  CURRENT_TIMESTAMP() as check_timestamp,
  'user_events' as table_name,
  COUNT(*) as total_records,
  COUNT(DISTINCT user_id) as unique_users,
  COUNTIF(user_id IS NULL) as null_user_ids,
  COUNTIF(timestamp > CURRENT_TIMESTAMP()) as future_timestamps
FROM `analytics_warehouse.user_events`
WHERE DATE(timestamp) = CURRENT_DATE();

Cloud SQL Production Setup:

# Create high-availability Cloud SQL instance
gcloud sql instances create production-db \
  --database-version=POSTGRES_14 \
  --tier=db-custom-4-16384 \
  --region=us-central1 \
  --availability-type=REGIONAL \
  --storage-type=SSD \
  --storage-size=500GB \
  --storage-auto-increase \
  --backup-start-time=03:00 \
  --maintenance-window-day=SUN \
  --maintenance-window-hour=04

# Configure SSL and authorised networks
gcloud sql ssl-certs create client-cert \
  --instance=production-db

gcloud sql instances patch production-db \
  --authorized-networks=10.1.0.0/16 \
  --require-ssl

Operational SQL Patterns:

-- Implement connection pooling and optimisation
CREATE EXTENSION IF NOT EXISTS pg_stat_statements;

-- Create indexes for application query patterns
CREATE INDEX CONCURRENTLY idx_users_email ON users(email);
CREATE INDEX CONCURRENTLY idx_orders_user_created ON orders(user_id, created_at);

-- Implement table partitioning for large datasets
CREATE TABLE user_events_partitioned (
    id SERIAL,
    user_id INTEGER NOT NULL,
    event_type VARCHAR(50) NOT NULL,
    created_at TIMESTAMP NOT NULL,
    data JSONB
) PARTITION BY RANGE (created_at);

CREATE TABLE user_events_2024_q1 PARTITION OF user_events_partitioned
    FOR VALUES FROM ('2024-01-01') TO ('2024-04-01');

Development Architecture Principles

Polyglot Data Strategy: Combining BigQuery for analytics with Cloud SQL for transactional workloads provides optimal performance for different data access patterns.

Performance Optimisation: Strategic use of partitioning, clustering, and materialised views ensures scalable query performance as data volumes grow.

Data Quality Engineering: Automated data quality checks and monitoring support reliable analytics and operational decision-making.

Lab 4: Multiple VPC Networks

Complex applications often require network isolation and controlled connectivity between different environments and services. This lab demonstrates advanced multi-VPC architectures with strategic peering and security controls.

Implementation Strategy

Multi-Environment VPC Architecture:

# Create production VPC with strict security controls
gcloud compute networks create production-vpc \
  --subnet-mode=custom \
  --bgp-routing-mode=regional

gcloud compute networks subnets create prod-web-subnet \
  --network=production-vpc \
  --range=10.10.1.0/24 \
  --region=us-central1 \
  --enable-private-ip-google-access

# Create staging VPC for testing and integration
gcloud compute networks create staging-vpc \
  --subnet-mode=custom \
  --bgp-routing-mode=regional

gcloud compute networks subnets create staging-subnet \
  --network=staging-vpc \
  --range=10.20.1.0/24 \
  --region=us-central1 \
  --enable-private-ip-google-access

# Create shared services VPC for common infrastructure
gcloud compute networks create shared-services-vpc \
  --subnet-mode=custom \
  --bgp-routing-mode=global

gcloud compute networks subnets create monitoring-subnet \
  --network=shared-services-vpc \
  --range=10.30.1.0/24 \
  --region=us-central1

Strategic VPC Peering Implementation:

# Create hub-and-spoke peering topology
gcloud compute networks peerings create prod-to-shared \
  --network=production-vpc \
  --peer-network=shared-services-vpc \
  --auto-create-routes

gcloud compute networks peerings create shared-to-prod \
  --network=shared-services-vpc \
  --peer-network=production-vpc \
  --auto-create-routes

gcloud compute networks peerings create staging-to-shared \
  --network=staging-vpc \
  --peer-network=shared-services-vpc \
  --auto-create-routes

gcloud compute networks peerings create shared-to-staging \
  --network=shared-services-vpc \
  --peer-network=staging-vpc \
  --auto-create-routes

Cross-Network Security Policies:

# Implement environment-specific firewall rules
gcloud compute firewall-rules create prod-web-access \
  --network=production-vpc \
  --allow=tcp:443 \
  --source-ranges=0.0.0.0/0 \
  --target-tags=prod-web-servers \
  --description="Production HTTPS access"

gcloud compute firewall-rules create staging-dev-access \
  --network=staging-vpc \
  --allow=tcp:80,tcp:443,tcp:8080 \
  --source-ranges=203.0.113.0/24 \
  --target-tags=staging-servers \
  --description="Development team access to staging"

# Allow monitoring from shared services
gcloud compute firewall-rules create allow-monitoring \
  --network=production-vpc \
  --allow=tcp:9090,tcp:3000 \
  --source-ranges=10.30.1.0/24 \
  --target-tags=all-servers \
  --description="Monitoring access from shared services"

Development Architecture Principles

Environment Isolation: Separate VPCs for production, staging, and shared services provide clear boundaries while enabling controlled connectivity for operational requirements.

Hub-and-Spoke Topology: Centralised shared services reduce complexity and operational overhead while maintaining security boundaries between environments.

Security zones: Network-level security policies enforce access controls that complement application-level security measures.

Lab 5: Cloud Monitoring Quickstart

Observability is essential for maintaining reliable cloud applications. This lab establishes comprehensive monitoring, alerting, and logging capabilities that support both development and production operations.

Implementation Strategy

Comprehensive Monitoring Setup:

# Create custom dashboard for application metrics
gcloud monitoring dashboards create --config-from-file=dashboard-config.json

# Dashboard configuration for key application metrics
cat > dashboard-config.json << EOF
{
  "displayName": "Application Performance Dashboard",
  "mosaicLayout": {
    "tiles": [
      {
        "width": 6,
        "height": 4,
        "widget": {
          "title": "Request Rate",
          "xyChart": {
            "dataSets": [
              {
                "timeSeriesQuery": {
                  "timeSeriesFilter": {
                    "filter": "resource.type=\"gce_instance\"",
                    "aggregation": {
                      "alignmentPeriod": "60s",
                      "perSeriesAligner": "ALIGN_RATE"
                    }
                  }
                }
              }
            ]
          }
        }
      }
    ]
  }
}
EOF

Alerting Policy Configuration:

# Create alerting policies for critical metrics
gcloud alpha monitoring policies create --policy-from-file=alert-policy.yaml

cat > alert-policy.yaml << EOF
displayName: "High Error Rate Alert"
conditions:
  - displayName: "Error rate above threshold"
    conditionThreshold:
      filter: 'resource.type="gce_instance" AND metric.type="compute.googleapis.com/instance/up"'
      comparison: COMPARISON_LESS_THAN
      thresholdValue: 0.95
      duration: 300s
      aggregations:
        - alignmentPeriod: 60s
          perSeriesAligner: ALIGN_MEAN
          crossSeriesReducer: REDUCE_MEAN
notificationChannels:
  - projects/$PROJECT_ID/notificationChannels/$NOTIFICATION_CHANNEL_ID
alertStrategy:
  autoClose: 86400s
EOF

Custom Metrics Implementation:

# Application code for custom metrics
from google.cloud import monitoring_v3

def record_custom_metric(project_id, metric_value):
    client = monitoring_v3.MetricServiceClient()
    project_name = f"projects/{project_id}"
    
    # Create time series data
    series = monitoring_v3.TimeSeries()
    series.metric.type = "custom.googleapis.com/application/response_time"
    series.resource.type = "gce_instance"
    series.resource.labels["instance_id"] = "1234567890123456789"
    series.resource.labels["zone"] = "us-central1-a"
    
    # Add data point
    point = series.points.add()
    point.value.double_value = metric_value
    point.interval.end_time.seconds = int(time.time())
    
    # Write time series data
    client.create_time_series(name=project_name, time_series=[series])

Log-Based Metrics and Analysis:

# Create log-based metrics for application insights
gcloud logging metrics create error_rate_metric \
  --description="Application error rate from logs" \
  --log-filter='severity>=ERROR AND resource.type="gce_instance"' \
  --value-extractor='EXTRACT(jsonPayload.response_time)'

# Set up log exports for long-term storage and analysis
gcloud logging sinks create bigquery-export \
  --log-filter='resource.type="gce_instance"' \
  --destination='bigquery.googleapis.com/projects/$PROJECT_ID/datasets/application_logs'

Development Architecture Principles

Proactive Monitoring: Comprehensive metrics collection and alerting enable early detection of performance issues and system anomalies.

Custom Metrics Integration: Application-specific metrics provide business-relevant insights that complement infrastructure monitoring.

Observability as Code: Programmatic configuration of monitoring and alerting supports consistent observability across environments and applications.

Lab 6: Managing Deployments Using Kubernetes Engine

Container orchestration with Kubernetes provides scalable, reliable deployment patterns for modern applications. This lab demonstrates production-ready Kubernetes configurations with advanced deployment strategies.

Implementation Strategy

Production GKE Cluster Configuration:

# Create production-ready GKE cluster
gcloud container clusters create production-cluster \
  --zone=us-central1-a \
  --machine-type=e2-standard-4 \
  --num-nodes=3 \
  --enable-autoscaling \
  --min-nodes=3 \
  --max-nodes=10 \
  --enable-autorepair \
  --enable-autoupgrade \
  --enable-network-policy \
  --enable-workload-identity \
  --enable-shielded-nodes \
  --disk-size=50GB \
  --disk-type=pd-ssd

Advanced Deployment Configurations:

# Blue-green deployment configuration
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-app-blue
  labels:
    app: web-app
    version: blue
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web-app
      version: blue
  template:
    metadata:
      labels:
        app: web-app
        version: blue
    spec:
      containers:
      - name: web-app
        image: gcr.io/$PROJECT_ID/web-app:v1.0
        ports:
        - containerPort: 8080
        resources:
          requests:
            cpu: 100m
            memory: 128Mi
          limits:
            cpu: 500m
            memory: 512Mi
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
---
apiVersion: v1
kind: Service
metadata:
  name: web-app-service
spec:
  selector:
    app: web-app
    version: blue
  ports:
  - port: 80
    targetPort: 8080
  type: LoadBalancer

Horisontal Pod Autoscaler Configuration:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: web-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web-app-blue
  minReplicas: 3
  maxReplicas: 50
  metrics:
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 70
  - type: Resource
    resource:
      name: memory
      target:
        type: Utilization
        averageUtilization: 80
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Percent
        value: 10
        periodSeconds: 60
    scaleUp:
      stabiliationWindowSeconds: 60
      policies:
      - type: Percent
        value: 50
        periodSeconds: 60

GitOps Deployment Pipeline:

# Configure Cloud Build for automated deployments
cat > cloudbuild.yaml << EOF
steps:
# Build container image
- name: 'gcr.io/cloud-builders/docker'
  args: ['build', '-t', 'gcr.io/\$PROJECT_ID/web-app:\$COMMIT_SHA', '.']

# Push to Container Registry
- name: 'gcr.io/cloud-builders/docker'
  args: ['push', 'gcr.io/\$PROJECT_ID/web-app:\$COMMIT_SHA']

# Deploy to GKE
- name: 'gcr.io/cloud-builders/gke-deploy'
  args:
  - run
  - --filename=k8s/
  - --image=gcr.io/\$PROJECT_ID/web-app:\$COMMIT_SHA
  - --location=us-central1-a
  - --cluster=production-cluster

# Run integration tests
- name: 'gcr.io/cloud-builders/gcloud'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    kubectl wait --for=condition=available --timeout=300s deployment/web-app-blue
    kubectl run test-pod --image=appropriate/curl --rm -i --restart=Never -- \
      curl -f http://web-app-service/health
EOF

# Create Cloud Build trigger
gcloud builds triggers create github \
  --repo-name=web-app-repo \
  --repo-owner=company \
  --branch-pattern=main \
  --build-config=cloudbuild.yaml

Development Architecture Principles

Container-Native Development: Kubernetes provides consistent deployment patterns that scale from development through production environments.

Declarative Configuration: Infrastructure and application configuration as code enables version control, testing, and automated deployment processes.

Scalability by Design: Horizontal pod autoscaling and cluster autoscaling provide automatic capacity management based on demand patterns.

Lab 7: Develop your Google Cloud Network - Challenge Lab

The challenge lab synthesizes all previous concepts into a comprehensive network architecture that demonstrates real-world implementation skills and best practices.

Challenge Architecture Requirements

Multi-Tier Application Network:

  • Public-facing load balancer with SSL termination
  • Private application tier with auto-scaling capabilities
  • Isolated database tier with backup and replication
  • Monitoring and logging infrastructure across all tiers
  • Development environment with controlled production access

Advanced Implementation Example:

# Complete challenge solution demonstrating integration
#!/bin/bash

PROJECT_ID="challenge-project-$(date +%s)"
REGION="us-central1"
ZONE="us-central1-a"

# Create comprehensive VPC architecture
gcloud compute networks create challenge-vpc \
  --subnet-mode=custom \
  --bgp-routing-mode=global

# Multi-tier subnet creation
gcloud compute networks subnets create web-tier \
  --network=challenge-vpc \
  --range=10.1.1.0/24 \
  --region=$REGION

gcloud compute networks subnets create app-tier \
  --network=challenge-vpc \
  --range=10.1.2.0/24 \
  --region=$REGION \
  --enable-private-ip-google-access

gcloud compute networks subnets create data-tier \
  --network=challenge-vpc \
  --range=10.1.3.0/24 \
  --region=$REGION \
  --enable-private-ip-google-access

# GKE cluster for application tier
gcloud container clusters create app-cluster \
  --zone=$ZONE \
  --network=challenge-vpc \
  --subnetwork=app-tier \
  --enable-private-nodes \
  --master-ipv4-cidr-block=172.16.0.0/28 \
  --enable-ip-alias \
  --enable-autoscaling \
  --min-nodes=2 \
  --max-nodes=10

# Cloud SQL for data tier
gcloud sql instances create challenge-db \
  --database-version=POSTGRES_14 \
  --tier=db-custom-2-8192 \
  --region=$REGION \
  --network=challenge-vpc \
  --no-assign-ip \
  --availability-type=REGIONAL

# BigQuery dataset for analytics
bq mk --dataset --location=US ${PROJECT_ID}:challenge_analytics

Success Criteria Validation

Network Connectivity Testing:

# Verify multi-tier connectivity
kubectl run connectivity-test --image=busybox --rm -i --restart=Never -- \
  nslookup challenge-db.internal

# Test load balancer functionality
curl -H "Host: challenge-app.example.com" http://$LOAD_BALANCER_IP/health

# Validate monitoring and alerting
gcloud logging read "resource.type=gce_instance AND severity>=WARNING" \
  --limit=10 --format="table(timestamp,severity,textPayload)"

Performance and Scalability Verification:

  • Application responds to load with appropriate scaling behavior
  • Database connections and queries perform within acceptable thresholds
  • Monitoring dashboards show comprehensive system visibility
  • Network policies properly restrict cross-tier communication

Development Architecture Principles

Integration Excellence: Successful challenge completion demonstrates ability to integrate multiple Google Cloud services into cohesive application architectures.

Production Readiness: Implementation includes all necessary components for production deployment including monitoring, security, and scalability features.

Operational Excellence: Architecture supports ongoing operations with appropriate logging, monitoring, and maintenance capabilities.

Enterprise Development Patterns and Best Practices

This comprehensive lab series establishes several critical enterprise development patterns:

Infrastructure as Code: All infrastructure components are defined through code and configuration management, enabling version control, testing, and automated deployment processes.

Microservices Architecture: Container orchestration and service mesh patterns support scalable, maintainable application architectures with clear service boundaries.

Data-Driven Development: Integration of analytics and operational data provides insights that drive application optimization and business decision-making.

DevOps Integration: Continuous integration and deployment patterns enable rapid, reliable software delivery while maintaining quality and security standards.

Production Implementation Considerations

Scalability Planning: Architecture designs must anticipate growth patterns and implement scaling strategies that maintain performance characteristics as demand increases.

Cost Optimisation: Development practices should include cost awareness and optimization strategies that balance performance requirements with operational expenses.

Security Integration: Security controls must be integrated throughout the development lifecycle rather than added as an afterthought to production systems.

Operational Excellence: Development practices should support operational requirements including monitoring, debugging, and incident response capabilities.

This comprehensive development essentials series provides the foundation for building production-ready cloud applications that leverage Google Cloud Platform’s managed services and development tools effectively.

Previous Article

Optimising Web Performance for Better UX

Share: