UDP is just about as reliable or unreliable as IP. It's a shim on top of IP to let unprivileged users send IP datagrams that can be multiplexed back to the right user. (Hence the name: "user" datagram protocol.)
Lots of people talk about a "TCP or UDP" design decision, but that's usually not the relevant question for an application developer. Most UDP applications wouldn't just blat their data directly into UDP datagrams, just like almost nobody blats their data directly into IP datagrams.
Typical applications generally want some sort of transport protocol on top of the datagrams (whether UDP-in-IP or just plain IP) that cares about reliability and often about fairness.
That could be TCP-over-IP, but some (like MixApp, which wanted TCP but not the kernel's default implementation) use TCP-over-UDP-over-IP, BitTorrent uses LEDBAT-over-UDP, Mosh uses SSP-over-UDP, Chrome uses QUIC-over-UDP and falls back to SPDY-over-TCP and HTTP-over-TCP, etc.
The idea that using UDP somewhere in the stack means "designing and implementing a reliable transport protocol from scratch" is silly. It doesn't mean that any more than using IP does.
The question of SOCK_STREAM vs. SOCK_DGRAM is more typically, "Does the application want to use the one transport protocol that the kernel implements, namely a particular flavor of TCP, or does it want to use some other transport protocol implemented in a userspace library?"
> UDP is just about as reliable or unreliable as IP
That's in theory.
In practice ISPs routinely classify, prioritize and shape application traffic (based on either TCP/UDP ports or results of deep packet inspection), so running SSL over IP will yield different IP loss profile than when running SSL over UDP.
Fun how this is more or less the contrary of what I think, so I'll add my point of view as a counter argument. IMHO if you need a transport layer, you are 99.9% of cases well served by TCP and should not consider UDP at all if not as a last resort to implement your transport protocol. Instead UDP is the way to go when you don't need a transport layer at all, and you can do with trivial ack / resend strategy (DNS is the obvious example), or just best-effort transfers where packet loss is not an issue at all (think a thermometer sending readings every second). In those cases the TCP three way handshake would be an overkill, and the benefits of TCP are minimal because of the use case.
> Instead UDP is the way to go when you don't need a transport layer at all, and you can do with trivial ack / resend strategy (DNS is the obvious example), or just best-effort transfers where packet loss is not an issue at all (think a thermometer sending readings every second)
That depends I guess. As somebody who works mostly with cloud telephony, SIP (signaling) and RTP (audio) are almost always carried over UDP. Funny thing is that you'll see more occurrences of one-way audio or call drops because of people implementing the application level protocol like SIP wrong rather than the transport layer letting you down.
That said, I would any day prefer a private MPLS network rather than the public internet.
Though I can understand why somebody who writes a fast nosql database prefers TCP. Also, thanks for redis, we use it a lot and love it! :)
SIP signalling over UDP was a bad idea, especially with the mandatory TCP failover. SIP's actually a spectacular clusterfuck of a design overall anyways. It's actually impossible to implement SIP in an unambiguous manner on today's networks because even the parsing rules are so bad, that multiple, popular, stacks have incompatible ways of dealing with things.
UDP for RTP makes sense because transmitting packets is pointless. In case of packetloss, the end user has to interpolate the result either via software or in their head.
This private MPLS network... it wouldn't happen to run over the exact same equipment that'd be handling your IP traffic, would it?
At the same time, beware of "helpful" lower layer protocols.
Mobile wireless stuff (EDGE/3G) is going to great lengths to avoid dropping packets, so if you get conditions right (moving train in the areas with poor coverage is one place where you can easily see this), you can get the packets "reliably delivered" in 20 seconds and more.
TCP has been designed in such a way that it interprets packet drops as a sign of "congestion" (which was typically true in ye olden days of purely wired networking), and it will start sending less data in response.
Whereas in wireless networking, occasional packet drops are just a fact of life and are not indicative of competing flows trying to share the channel. So it actually makes [some] sense that wireless protocols try to compensate for the behaviour of the transport protocol used by 90% of all data: TCP.
The one case where you don't really want TCP semantics, is when you want SCTP/RTP semantics: basically, user datagrams decomposed into frames and then priority-multiplexed over a reliable circuit, then buffered back into in-order messages. This is, it turns out, what web browsers, game clients, VoIP services, and a lot of other things want—to the point that if we're going to have one kernel-implemented protocol and everything else has to live on top of UDP, I wish the one kernel protocol was SCTP. (Because one-channel SCTP really looks a lot like TCP.)
Cheaper SCTP isn't quite enough—one of the major problems with lots of little TCP connections is that they all have their own backpressure, so you get things like worker processes that share a remote flapping their window sizes up and down in response to one-anothers' requests. Ideally, backpressure would occur at the IP layer (host-to-host rather than port-to-port) but SCTP gives you the ability to have a set of streams that share a backpressure "group" on both the local and remote ends.
> UDP is just about as reliable or unreliable as IP.
Ok.
> It's a shim on top of IP to let unprivileged users send IP datagrams that can be multiplexed back to the right user. (Hence the name: "user" datagram protocol.)
It's mostly a way to send data in such a manner that you don't need to do the full 'call-setup-data-transmission-and-terminate' that would be required for other virtual circuit based protocols such as TCP when you don't need all that luxury. So for protocols that carry small amounts of data in a manner where retrying is not a problem and where loss of a packet is not an immediate disaster. It's also more suitable for real-time applications because of this than TCP (especially true for the first packet). Because of the fact that there is no virtual circuit a single listener can handle data from multiple senders.
The 'USER' does not refer to unprivileged users but simply to users as opposed to system packets (such as for instance ICMP and other datagram like packets that are not usually sent out directly by applications). So it's not a privilege matter but a matter of user-space vs system modules elsewhere in the stack.
> Lots of people talk about a "TCP or UDP" design decision, but that's usually not the relevant question for an application developer.
It absolutely is.
> Most UDP applications wouldn't just blat their data directly into UDP datagrams,
They usually do exactly that.
> just like almost nobody blats their data directly into IP datagrams.
You're comparing apples with oranges, IP is one layer and TCP and UDP are on another. So you'd have to compare UDP with TCP and then you're back to that design decision again.
> Typical applications generally want some sort of transport protocol on top of the datagrams (whether UDP-in-IP or just plain IP), that cares about reliability and often about fairness.
Fairness is something that is usually not under control of the endpoints of a conversation but something that routers in between influence. They can decide to let a packet through or drop it (this goes for TCP as well as UDP), if a line is congested your UDP packets will usually (rules can be set to configure this) be dropped before your TCP packets will be in spite of the fact that TCP will re-try any lost packets. UDP packets can also be duplicated and routed in such a way that they arrive out-of-order.
> That could be TCP-over-IP, but some (like MixApp, which wanted TCP but not the kernel's default implementation) use TCP-over-UDP-over-IP, BitTorrent uses LEDBAT-over-UDP, Mosh uses SSP-over-UDP, Chrome uses QUIC-over-UDP and falls back to SPDY-over-TCP and HTTP-over-TCP, etc.
Running alternative protocols packaged inside other protocols is a time honored practice. See also: TCP over carrier pigeons and tunneling HTTP over DNS traffic (effectively using UDP). This is not in any way special, it's just a means to an end.
> The idea that using UDP somewhere in the stack means "designing and implementing a reliable transport protocol from scratch" is silly. It doesn't mean that any more than using IP does.
It actually comes down to exactly that. If you use UDP as your base and your application requires reliable transmission of data then you're going to have to deal with loss/duplication/sequencing at some other point in your application or put another (pre existing) protocol on top of it in order to mitigate these.
If your application can tolerate those errors (or if they are not considered errors) then a naive implementation will do.
> The question of SOCK_STREAM vs. SOCK_DGRAM is more typically, "Does the application want to use the one transport protocol that the kernel implements, namely a particular flavor of TCP, or does it want to use some other transport protocol implemented in a userspace library?"
TCP is the default for anything requiring virtual circuits if you have demands that are not well describe by that model and/or need real time, low overhead and you're willing to do the work required to deal with UDPs inherent issues (if those are a problem) then you're totally free to do so.
But the question is usually not 'do I need TCP', it usually is 'how do I avoid re-implementing TCP if I need its features'.
It's a tough choice because at a minimum it means that you're going to have to write software for both endpoints.
This is one of the reasons why we see HTTP over TCP in so many places where it wasn't originally intended: it is more or less guaranteed to be well tested and there are tons of tools available to use this protocol combination, especially browsers, fetchers and servers in all kinds of flavors. For UDP that situation is much less rosy and using UDP always translates into having to do a bunch of plumbing yourself.
The thing that bites a lot of naive protocol designers who use UDP is that it doesn't guarantee order either. So you can get delivery out of order, and that gets more likely the more your packet crosses IP subnets. In part because some folks do traffic shaping, in part because UDP traffic is considered "less important" by a lot of ISPs, and in part because new switches, like the latest compilers, have enough cpu in the control plane to play games on packets flying about.
> You're comparing apples with oranges, IP is one layer and TCP and UDP are on another.
That is what the books say, but I don't think it is right. When you consider that raw IPv4 doesn't work in practice because of NAT, UDP is a defacto minimum internet layer in practice.
NAT is a kludge. The existence of NAT does not remove the fact that both TCP and UDP are layered on top of it and there is plenty of stuff happening that is layered directly on IP besides UDP, for instance ICMP.
> It's mostly a way to send data in such a manner that you don't need to do the full 'call-setup-data-transmission-and-terminate' that would be required for other virtual circuit based protocols such as TCP when you don't need all that luxury. So for protocols that carry small amounts of data in a manner where retrying is not a problem and where loss of a packet is not an immediate disaster. It's also more suitable for real-time applications because of this than TCP (especially true for the first packet). Because of the fact that there is no virtual circuit a single listener can handle data from multiple senders.
It is possible that someone wants a virtual circuit but can do better than TCP for their application. I think the parent's explanation was more apt - it's a small layer on top of IP for you to implement your own protocol logic.
Lots of people talk about a "TCP or UDP" design decision, but that's usually not the relevant question for an application developer. Most UDP applications wouldn't just blat their data directly into UDP datagrams, just like almost nobody blats their data directly into IP datagrams.
Typical applications generally want some sort of transport protocol on top of the datagrams (whether UDP-in-IP or just plain IP) that cares about reliability and often about fairness.
That could be TCP-over-IP, but some (like MixApp, which wanted TCP but not the kernel's default implementation) use TCP-over-UDP-over-IP, BitTorrent uses LEDBAT-over-UDP, Mosh uses SSP-over-UDP, Chrome uses QUIC-over-UDP and falls back to SPDY-over-TCP and HTTP-over-TCP, etc.
The idea that using UDP somewhere in the stack means "designing and implementing a reliable transport protocol from scratch" is silly. It doesn't mean that any more than using IP does.
The question of SOCK_STREAM vs. SOCK_DGRAM is more typically, "Does the application want to use the one transport protocol that the kernel implements, namely a particular flavor of TCP, or does it want to use some other transport protocol implemented in a userspace library?"