Learning & ADRs

Mastered Go generics syntax and type constraints, enabling reusable algorithms while maintaining compile-time type safety.

Key Learnings:

• Type Parameters: Written in square brackets (e.g., [T any]) before parameter lists.
• Constraints: Define what types are allowed. Standard constraints are in the golang.org/x/exp/constraints package (like Ordered for < and > comparisons).
• The any constraint: An alias for interface{}.
• The comparable constraint: A built-in interface that allows operations like == and !=.
• Underlying types: Using the ~ symbol (e.g., ~int matches any custom type whose underlying type is int).

Real-World Code Example

package generics

// MapKeys returns a slice of all keys in a map.
// Keys must be comparable, and values can be any type.
func MapKeys[K comparable, V any](m map[K]V) []K {
    keys := make([]K, 0, len(m))
    for k := range m {
        keys = append(keys, k)
    }
    return keys
}

This utility function works on maps of strings to ints, ints to floats, or any comparable key, eliminating duplicate helper functions!

Benchmarking and tuning gRPC services for high-throughput, low-latency microservices communication.

Key Learnings:

• HTTP/2 Multiplexing: Enables sending multiple requests and responses concurrently over a single TCP connection, reducing handshake overhead by 80%+.
• Protocol Buffers: Binary encoding is 3-10x faster to serialize/deserialize than JSON and has a much smaller payload footprint.
• Connection Pooling: In Go, client connections should be shared or pooled across goroutines to prevent active TCP connection thrashing.
• Keepalive Pings: Setting up client-side and server-side keepalive parameters keeps connections warm and quickly catches dead TCP sockets.
• Message Compression: Gzip/Snappy compression reduces network bandwidth but trades CPU cycles. Only compress larger payloads (>100KB).

Deep dive into consumer groups and rebalancing strategies to optimize messaging performance and throughput in large-scale event-driven systems.

Key Learnings:

• Group Coordinator: A broker responsible for managing consumer group state and assigning partitions.
• Stop-The-World Rebalancing: The traditional eager rebalance protocol revokes all partitions and pauses message processing during rebalance, which can cause latency spikes.
• Cooperative Sticky Assignor: Introduced to allow consumers to keep their partitions while other partitions are reassigned, reducing downtime during scaling.
• Tuning timeouts: Keep session.timeout.ms and max.poll.interval.ms properly adjusted to prevent false consumer failures.
• Rebalance Listeners: Implement custom logic on partitions revoked and assigned to flush database changes or commit offsets before partition ownership shifts.

Understanding scheduler placement behaviors, resource limits enforcement, cgroups, and pod eviction mechanisms under load.

Key Learnings:

• Requests: Used by the scheduler to place Pods on Nodes. If a Node has enough allocatable CPU/Memory to satisfy requests, the Pod is scheduled.
• Limits: Enforced at runtime by the container runtime via Linux kernel cgroups.
• CPU Throttling: If a container exceeds its CPU limit, it is throttled by the kernel using CPU shares. It does not crash.
• OOM Killing: If a container exceeds its Memory limit, it is terminated with exit code 137 (Out Of Memory).
• QoS Classes:
  • Guaranteed: Requests equal limits for both CPU and memory. Least likely to be evicted.
  • Burstable: Requests are set and are less than limits. Medium eviction priority.
  • BestEffort: No requests or limits set. First to be evicted under resource pressure.
• Overcommit: Kubernetes allows overcommitting nodes (sum of limits > node size). This is cost-efficient but requires careful monitoring to prevent cascading failures.

Context

In a microservices architecture, services make frequent RPC calls to fetch lookup values, verify auth states, and trigger downstream processes. Using standard REST/JSON creates performance bottlenecks due to string serialization, lacks strict typing interfaces, and does not support streaming naturally.

Decision

We will use gRPC with Protocol Buffers for all synchronous, internal service-to-service communication. External traffic from users will still enter through HTTP/JSON or GraphQL via our API Gateway, which will translate incoming payloads to internal gRPC requests.

Consequences

• Pros:
  • Highly optimized serialization and deserialization speeds.
  • Multi-language support with automatic code generation from .proto definition files.
  • Strongly typed communication contracts checked at compile-time.
  • Out-of-the-box support for client, server, and bidirectional streaming.
• Cons:
  • Human readability is reduced compared to JSON (requires tools like grpcurl or UI debuggers like Postman/BloomRPC).
  • Requires maintaining a schema definition workflow (e.g. shared repo or registry) to prevent breaking API contracts.

Context

Our core platform needs to scale horizontally to support millions of concurrent connections, isolate high-load components (like ML inference serving and real-time streaming), and enable multiple teams to deploy their services independently without coordination blockages.

Decision

We will adopt a Microservices Architecture instead of a unified monolith. Each service will be responsible for a single domain (e.g., Shipment, Routing, Authentication, Billing), have its own database to ensure loose coupling, and run in isolated containers.

Consequences

• Pros:
  • Independent scaling of resource-heavy services (e.g. ML inference requires GPUs, auth requires memory-focused DB caching).
  • Isolated failure domains; a crash in route calculation does not bring down billing or user login.
  • Team autonomy and faster development cycles.
• Cons:
  • Increased network hop latency.
  • Higher operational overhead (monitoring, distributed tracing, Kubernetes configuration).
  • Complex transaction management (requires patterns like Saga or Outbox instead of local ACID transactions).