This week’s system design refresher:
Top 5 Kafka use cases
What is OSI Model?
Logging, Tracing, Metrics
Top 4 Kubernetes Service Types in one diagram
Kafka was originally built for massive log processing. It retains messages until expiration and lets consumers pull messages at their own pace.
Let’s review the popular Kafka use cases.
Log processing and analysis
Data streaming in recommendations
System monitoring and alerting
CDC (Change data capture)
System migration
Over to you: Do you have any other Kafka use cases to share?
How is data sent over the internet? What does that have to do with the OSI model? How does TCP/IP fit into this?
7 Layers in the OSI model are:
Physical Layer
Data Link Layer
Network Layer
Transport Layer
Session Layer
Presentation Layer
Application Layer
Logging, tracing and metrics are 3 pillars of system observability. The diagram below shows their definitions and typical architectures.
Logging
Logging records discrete events in the system. For example, we can record an incoming request or a visit to databases as events. It has the highest volume. ELK (Elastic-Logstash-Kibana) stack is often used to build a log analysis platform. We often define a standardized logging format for different teams to implement, so that we can leverage keywords when searching among massive amounts of logs.
Tracing
Tracing is usually request-scoped. For example, a user request goes through the API gateway, load balancer, service A, service B, and database, which can be visualized in the tracing systems. This is useful when we are trying to identify the bottlenecks in the system. We use OpenTelemetry to showcase the typical architecture, which unifies the 3 pillars in a single framework.
Metrics
Metrics are usually aggregatable information from the system. For example, service QPS, API responsiveness, service latency, etc. The raw data is recorded in time-series databases like InfluxDB. Prometheus pulls the data and transforms the data based on pre-defined alerting rules. Then the data is sent to Grafana for display or to the alert manager which then sends out email, SMS, or Slack notifications or alerts.
Over to you: Which tools have you used for system monitoring?
The diagram below shows 4 ways to expose a Service.
In Kubernetes, a Service is a method for exposing a network application in the cluster. We use a Service to make that set of Pods available on the network so that users can interact with it.
There are 4 types of Kubernetes services: ClusterIP, NodePort, LoadBalancer and ExternalName. The “type” property in the Service's specification determines how the service is exposed to the network.
ClusterIP
ClusterIP is the default and most common service type. Kubernetes will assign a cluster-internal IP address to ClusterIP service. This makes the service only reachable within the cluster.
NodePort
This exposes the service outside of the cluster by adding a cluster-wide port on top of ClusterIP. We can request the service by NodeIP:NodePort.
LoadBalancer
This exposes the Service externally using a cloud provider’s load balancer.
ExternalName
This maps a Service to a domain name. This is commonly used to create a service within Kubernetes to represent an external database.