This week’s system design refresher:
Roadmap for Learning SQL (Youtube video)
Can Kafka lose messages?
9 Best Practices for building microsercvices
Roadmap for Learning Cyber Security
How does Javascript Work?
SPONSOR US
New Relic Interactive Application Security Testing (IAST) allows security and engineering teams to save time by focusing on real application security problems with zero false positives, as validated by the OWASP benchmark result of 100% accuracy.
Error handling is one of the most important aspects of building reliable systems.
Today, we will discuss an important topic: Can Kafka lose messages?
A common belief among many developers is that Kafka, by its very design, guarantees no message loss. However, understanding the nuances of Kafka's architecture and configuration is essential to truly grasp how and when it might lose messages, and more importantly, how to prevent such scenarios.
The diagram below shows how a message can be lost during its lifecycle in Kafka.
Producer
When we call producer.send() to send a message, it doesn't get sent to the broker directly. There are two threads and a queue involved in the message-sending process:
1. Application thread
2. Record accumulator
3. Sender thread (I/O thread)
We need to configure proper ‘acks’ and ‘retries’ for the producer to make sure messages are sent to the broker.
Broker
A broker cluster should not lose messages when it is functioning normally. However, we need to understand which extreme situations might lead to message loss:
1. The messages are usually flushed to the disk asynchronously for higher I/O throughput, so if the instance is down before the flush happens, the messages are lost.
2. The replicas in the Kafka cluster need to be properly configured to hold a valid copy of the data. The determinism in data synchronization is important.
Consumer
Kafka offers different ways to commit messages. Auto-committing might acknowledge the processing of records before they are actually processed. When the consumer is down in the middle of processing, some records may never be processed.
A good practice is to combine both synchronous and asynchronous commits, where we use asynchronous commits in the processing loop for higher throughput and synchronous commits in exception handling to make sure the the last offset is always committed.
If you’re not a paid subscriber, here’s what you missed.
To receive all the full articles and support ByteByteGo, consider subscribing:
Creating a system using microservices is extremely difficult unless you follow some strong principles.
9 best practices that you must know before building microservices:
Design For Failure
A distributed system with microservices is going to fail.
You must design the system to tolerate failure at multiple levels such as infrastructure, database, and individual services. Use circuit breakers, bulkheads, or graceful degradation methods to deal with failures.
Build Small Services
A microservice should not do multiple things at once.
A good microservice is designed to do one thing well.
Use lightweight protocols for communication
Communication is the core of a distributed system.
Microservices must talk to each other using lightweight protocols. Options include REST, gRPC, or message brokers.
Implement service discovery
To communicate with each other, microservices need to discover each other over the network.
Implement service discovery using tools such as Consul, Eureka, or Kubernetes Services
Data Ownership
In microservices, data should be owned and managed by the individual services.
The goal should be to reduce coupling between services so that they can evolve independently.
Use resiliency patterns
Implement specific resiliency patterns to improve the availability of the services.
Examples: retry policies, caching, and rate limiting.
Security at all levels
In a microservices-based system, the attack surface is quite large. You must implement security at every level of the service communication path.
Centralized logging
Logs are important to finding issues in a system. With multiple services, they become critical.
Use containerization techniques
To deploy microservices in an isolated manner, use containerization techniques.
Tools like Docker and Kubernetes can help with this as they are meant to simplify the scaling and deployment of a microservice.
Over to you: what other best practice would you recommend?
By Henry Jiang. Redrawn by ByteByteGo.
Cybersecurity is crucial for protecting information and systems from theft, damage, and unauthorized access. Whether you're a beginner or looking to advance your technical skills, there are numerous resources and paths you can take to learn more about cybersecurity. Here are some structured suggestions to help you get started or deepen your knowledge:
Security Architecture
Frameworks & Standards
Application Security
Risk Assessment
Enterprise Risk Management
Threat Intelligence
Security Operation
The cheat sheet below shows most important characteristics of Javascript.
Interpreted Language
JavaScript code is executed by the browser or JavaScript engine rather than being compiled into machine language beforehand. This makes it highly portable across different platforms. Modern engines such as V8 utilize Just-In-Time (JIT) technology to compile code into directly executable machine code.
Function is First-Class Citizen
In JavaScript, functions are treated as first-class citizens, meaning they can be stored in variables, passed as arguments to other functions, and returned from functions.
Dynamic Typing
JavaScript is a loosely typed or dynamic language, meaning we don't have to declare a variable's type ahead of time, and the type can change at runtime.
Client-Side Execution
JavaScript supports asynchronous programming, allowing operations like reading files, making HTTP requests, or querying databases to run in the background and trigger callbacks or promises when complete. This is particularly useful in web development for improving performance and user experience.
Prototype-Based OOP
Unlike class-based object-oriented languages, JavaScript uses prototypes for inheritance. This means that objects can inherit properties and methods from other objects.
Automatic Garbage Collection
Garbage collection in JavaScript is a form of automatic memory management. The primary goal of garbage collection is to reclaim memory occupied by objects that are no longer in use by the program, which helps prevent memory leaks and optimizes the performance of the application.
Compared with Other Languages
JavaScript is special compared to programming languages like Python or Java because of its position as a major language for web development.
While Python is known to provide good code readability and versatility, and Java is known for its structure and robustness, JavaScript is an interpreted language that runs directly on the browser without compilation, emphasizing flexibility and dynamism.
Relationship with Typescript
TypeScript is a superset of JavaScript, which means that it extends JavaScript by adding features to the language, most notably type annotations. This relationship allows any valid JavaScript code to also be considered valid TypeScript code.
Popular Javascript Frameworks
React is known for its flexibility and large number of community-driven plugins, while Vue is clean and intuitive with highly integrated and responsive features. Angular, on the other hand, offers a strict set of development specifications for enterprise-level JS development.
Get your product in front of more than 500,000 tech professionals.
Our newsletter puts your products and services directly in front of an audience that matters - hundreds of thousands of engineering leaders and senior engineers - who have influence over significant tech decisions and big purchases.
Space Fills Up Fast - Reserve Today
Ad spots typically sell out about 4 weeks in advance. To ensure your ad reaches this influential audience, reserve your space now by emailing [email protected].