Understanding how data flows across networks is fundamental to building reliable distributed systems. Master these protocols and patterns.
Distributed system are spread out over a network. Without communication between nodes, distributed system are simply disconnected standalone system. The critical decision made in communication protocol in system design influences the distributed system's performance, availability and scalability.
Pro tip: Network protocols are build on stacks. Each layer depends on the abstraction of the layer below it.
Every time you make an API call, send a message, or load a webpage, data travels through the stack from the top layer to the bottom layer and vice versa on the other end of the network as shown in Figure 1.
when a distributed system is not working, it is important to understand the network protocols used by the system. The troubleshooting engineer might ask for the following information:
TCP is a protocol to ensure reliable data exchange between two process on top of IP. It guarantees that data is delivered in the correct order, without any gaps, duplicates and loss.
The process of sending data over TCP is as follows:
TCP's reliability and stability come at the cost of lower bandwidth and higher latencies.
If there is no need for reliability and stability in data exchange, UDP can be used to send data over the network. UDP is perfect for applications where speed is more important than reliability, such as live video streaming, online gaming, VoIP, and DNS lookups.
Streaming and gaming services rely on UDP because real-time data has a short shelf life. If a packet is lost during transmission, retransmitting it serves no purpose—the game or stream has already moved on. By the time the retransmitted data arrives, it's outdated. This is where UDP excels: unlike TCP, which would insist on redelivering lost packets and introduce latency, UDP prioritizes speed over reliability, keeping the user experience smooth.
TLS (Transport Layer Security) runs on top of TCP and provides secure communication channel so that application layer protocols like HTTP can be securely transmitted over the network. TLS provide encryption, authentication and integrity to the data transmitted over the network.
Client and server exchange public and private keys to encrypt and decrypt data. The shared secret key is never transmitted over the network. Although asymmetric encryption is slow and expensive, it's only used to create the shared encryption key.
Client and server priodically renegotiated to minimize amount of the data that can be deciphered if shared secret key is broken. The shared secret key is transmitted over the network.
TLS implements authentication using digital signatures based on asymmetric cryptography. The client has no idea whether the public key shared by the server is authentic, so we have certificates to prove the ownership of a public key for a specific entity.
A certificate authority (CA) is a trusted third party that issues digital certificates to verify the identity of a server. we have certificates to prove the ownership of a public key.
When it rides on top of TLS, it’s also referred to as HTTPS. You should use HTTPS by default.
Modern applications often use both protocols. For example, a web-based video conferencing app might use TCP/HTTP for signaling and authentication but UDP/WebRTC for the actual audio/video streams.