TCP/IP
TCP/IP protocol is the vehicle of choice for transferring data between database client and servers.
The focus of this White Paper is to examine TCP/IP implementation in Solaris 2. This paper describes the new techniques that Sun has developed for achieving higher TCP/IP performance, and adapted TCP/IP to take advantage of multiple CPUs. It also describes algorithms within the protocol code that have been restructured to allow them to scale up to much larger installations. These changes have been incorporated without changing the application program interface, thus maintaining complete compatibility for the installed base of Internet applications.
- The Internet
- The Intranet
- Solaris 2 TCP/IP Implementation
- TCP/IP Is Optimized for High Performance
- Logical Interfaces for Virtual Host Services
- IP Multicasting Supports the New Generation of Computer Applications
- Router Discovery and Multiple Default Routes for Robust IP Routing
- Path MTU Discovery Provides Efficient Transport of Datagrams
- Internet Services
- References
Introduction
The Internet has evolved a great deal from its beginning 25 years ago as a small, experimental network linking university and research centers. The Internet is currently enjoying highly publicized and global interest from a new wave of Internet user. Driving this interest are the World Wide Web, the Electronic Commerce, and Sun Microsystem's Java network-specific programming language. Providing the underlying infrastructure for the Internet is the Transmission Control Protocol and the Internet Protocol, or TCP/IP, and the suite of protocol standards that allow computers to share resources, exchange data, and communicate.
The main functions of the Internet protocols have remained fundamentally the same since their inception, that is, they allow sharing of information among diverse computers and networks, yet they have evolved hand-in-hand with new data communications technologies, thereby securing the users' investment in Internet applications.
Today, network administrators are rapidly deploying TCP/IP for use in their corporate Intranetworks. In addition to the Internet file and resource sharing tools, network administrators are tapping TCP/IP's continually evolving collection of data communication standards and acknowledging it as the vehicle of choice for supporting management information systems (MIS) and database services. Network administrators are also finding TCP/IP a fast and reliable vehicle for carrying data from today's interactive computer applications.
Sun's implementation of TCP/IP reflects performance features that have made Sun a leader in this arena. Sun has optimized TCP/IP to take full advantage of multiprocessor technology. Additionally, Sun has improved data throughput and optimized performance over wide-area networks. When running on a network server, Sun's implementation of TCP/IP supports thousands of simultaneous connections and thousands of remote logins.
In the corporate Intranet, Sun's implementation of TCP/IP supports collaborative workgroup applications and other multicast applications, and is a proven workhorse for database servers.
Whether on the Intranet or the Internet, Solaris 2 continually provides a well- tested, high-performance implementation of TCP/IP.
This paper looks at the TCP/IP implementation in Solaris 2. It begins with a look at how the Internet and its body of standards have developed, then describes how the Solaris 2 TCP/IP implementation meets the needs of both the Internet user and the network administrator designing a network for today's corporate Intranet.
The Internet
The Internet began over 25 years ago with an experimental network called ARPAnet that linked computers at universities and government agencies. The early ARPAnet is notable because it was the first project that successfully explored the use of packet switched network technology, giving birth to the Internet protocol suite, of which the Transmission Control Protocol (TCP) and the Internet Protocol (IP) are the foundation.
The ARPAnet evolved into the Internet and grew rapidly through the 1980's. This was due in large part to the creation and connection of computer networks at large universities and research centers, which encouraged their faculty and students to use the resources available on the Internet. Many of these universities were Sun's original customers. By the early 1990's, commercial Internet Service Providers (ISPs) became involved in operating portions of the Internet, further growing the network extending access to individuals out side the university and research communities.
The Internet's rapid expansion in the past few years is due in great part to the burgeoning popularity of the World Wide Web, which has come to be regarded as a useful database search tool as well as a vast marketing arena. TCP/IP products currently include Web browsers such as the Netscape Navigator 2.0 offered in the Solaris 2 and PC - NFS Pro TCP/IP software.
While Internet arenas such as the World Wide Web and Electronic Commerce allow universal sharing of information, Sun Microsystem's new Java computer language was designed specifically for developing interactive TCP/IP applications and allows the universal sharing of computer applications. TCP/IP provides the necessary foundation for applications such as Web browsers and Java to function.
The Intranet
We might look at the corporate Intranet as a microcosm to the Internet's macrocosm. Intranet users perform the same tasks as Internet users - they exchange files, share resources, and send and receive mail - and the same tools available to Internet users for performing these tasks are also available to local internet users. For example, users on both an Intranet and the Internet can use a Web browser to seek information.
Other similarities exist in the type of data being transmitted on these networks. Links on both the Intranet and the Internet are increasingly being used by applications such as integrated video and voice, all of which contend for bandwidth on the network links. Intranets have additional requirements to support collaborative workgroup applications, which again vie for network bandwidth. And TCP/IP is still the vehicle of choice for transferring data between database client and servers. A large number of Intranet and Internet servers are based on Solaris 2 and its TCP/IP implementation.
The current trend is that the number of users of both Intranets and the Internet are increasing. The network administrator will want a TCP/IP stack robust enough to handle the resulting increases in network traffic.
Introducing Solaris 2 TCP/IP
Sun's founders made a bet on Internet technology when they decided in 1982 to base Sun's entire network architecture on the TCP/IP protocol suite. The bet was a good one. The use of TCP/IP has been validated for both Sun and its customers with the tremendous success of the Internet protocols and the rapid growth of the global Internet. Every Sun workstation shipped since 1982 has included TCP/IP. Sun continues to promote this success by building in high performance features that minimize processing cycles and streamline data transmissions across the network.
Internet protocols and services have evolved considerably since the early Internet. The original Internet services - file transfer, remote login, and electronic mail - were augmented in the mid-1980's by Sun's introduction of the Network File System (NFS) and Remote Procedure Call (RPC). The next major advance was the World Wide Web, introduced in the early 1990's, and Sun's recent introduction of the Java programming language. Internet services continue to evolve. The next major advance will be the introduction of multicast-based audio and video conferencing over the Internet (see Figure 1).
The original intent of the Internet was to allow the sharing of information, and it has been highly successful at providing this function. One might ask how it is possible that the Internet has been able to continue supporting these functions on rapidly evolving computers and networks over such an extended period of time? The answer is by developing and following standards.
The early developers of the Internet recognized that to allow the diverse set of computers and networks to interoperate, they needed a model that provided structure and order to the system, but that was at the same time flexible. They sought an open system, that is, one that would make its implementation easily available. The system they adopted is based on a model that has come to be known as Open Systems Interconnection (OSI).
This model forms the basis of an architecture that governs networking operation in terms of layers. Conceptually, there are seven layers (see Figure 2). The lowest are the physical and data link layers, which control network access and handle the details of physically interfacing to the network media - a cable running Ethernet or the Fiber Distributed Data Interface (FDDI), for example. The highest layer is the application layer, which governs how applications such as electronic mail readers function. In between are various layers that handle tasks such as opening and closing a connection and routing packets to the appropriate destination.
The TCP/IP standards are developed in an open process by the Internet Engineering Task Force (IETF). Sun's business model since its inception has been based on open standards and multiple implementations. Sun has been key proponents of open systems in the industry. Sun is an active participant in the IETF and in leading IETF working groups. Additionally, the Solaris 2 TCP/IP stack is state of the art, reflecting the latest Internet standards.
The TCP/IP protocol suite roughly follows the OSI model, but is implemented in four layers, rather than the seven described by OSI. The Internet Protocol (IP) is often referred to as the foundation upon which the other protocols are built. In terms of the OSI model, it is equivalent to a network-layer protocol. IP uses an address scheme to facilitate internetwork routing and transmission of information in units called datagrams. It handles fragmentation and reassembly of the datagrams and provides error reporting.
The Transmission Control Protocol (TCP) establishes a reliable means of data transmission between two devices by providing flow control mechanisms, retransmission and other reliability capabilities, and is equivalent to the OSI transport layer.
There are other protocols which are integral to the functions of the Internet, and the Solaris 2 TCP/IP implementation supports these standard protocols:
- File Transfer Protocol (FTP) - Handles file transfers between computers, regardless of their type and format.
- Telnet - Internet standard that handles remote computer logins.
- Simple Mail Transport Protocol (SMTP) - Handles the exchange of electronic mail.
- Domain Name System (DNS) - Maps domain names to IP addresses.
- Trivial File Transfer Protocol (TFTP) - Resides in read-only memory and is used to bootstrap diskless systems.
- User Datagram Protocol (UDP) - Establishes a means of data transmission similar to TCP, but without TCP's reliability features.
- Internet Control Message Protocol (ICMP) - Generates packets containing error messages.
- Internet Group Management Protocol (IGMP) - Allows systems on a physical network to know which hosts currently belong to which multicast group.
- Address Resolution Protocol (ARP) - Binds IP addresses to Ethernet and IEEE 802.2 addresses.
Figure 2 shows the relationship between the OSI model and the Solaris 2 TCP/IP protocol suite.
One key to the success of the Internet protocols is their ability to operate over a wide variety of data link technologies. The protocols that were used primarily over Ethernet links capable of transmitting data at the rate of 10 megabits per second (Mbps) in the 1980's have evolved in the 1990's to layer above new technologies that span a wide spectrum of data rates that include:
- Asynchronous Transfer Mode (ATM) operating at speeds of 155 Mbps or 622 Mbps
- FDDI with throughputs of 100 Mbps
- Fast Ethernet capable of transmitting data at 100 Mbps
- Traditional Ethernet at 10 Mbps
- Synchronous leased circuits with transmission rates of 64 Kbps to 45 Mbps or higher
- Integrated Services Digital Network (ISDN) with speeds up to 128 Kbps
- Asynchronous modem links over the dial-up telephone network providing maximum signaling rates up to 28.8 Kbps
The Internet protocols have remained fundamentally the same - securing the users' investment in Internet applications - while the data links have evolved to span orders of magnitude of bandwidth, from kilobits per second to hundreds of megabits per second.
The core Internet and transport layer protocols have also evolved, maintain-ing complete compatibility along the way. New congestion control algorithms were introduced into TCP in the late 1980's. The Path MTU discovery protocol, an algorithm that prevents performance-degrading IP fragmen-tation over wide area networks, was developed in the early 1990's. The Router Discovery Protocol, which greatly improves IP routing robustness, is also a recent introduction. The core protocols continue to evolve. The next big change will be the introduction of IP version 6 (IPv6), the next generation Internet Protocol, which will allow the Internet to continue to grow and support billions of nodes. It should begin deployment within the next few years.
Solaris 2 TCP/IP Implementation
Sun's implementation of the Internet protocols has evolved in parallel with the IETF protocols themselves. Sun has developed new techniques for achieving higher TCP/IP performance, and adapted TCP/IP to take advantage of multiple central processing units (CPUs). And the algorithms within the protocol code have been restructured to allow them to scale up to much larger installations. These changes have been incorporated without changing the application program interface, thus maintaining complete compatibility for the installed base of Internet applications. These features and enhancements are described more fully in the next sections.
TCP/IP Is Optimized for High Performance
The Solaris 2 TCP/IP software has been fully multi-threaded for improved performance and greater scaling on multi-processor (MP) machines. By processing data in parallel, TCP/IP can support more simultaneous connection and achieve better total throughput. For TCP bulk data transfers, the granularity of parallelism is a single TCP connection. This means that multiple bulk-data TCP connections can run at the same time on different processors of an MP machine.
The TCP and UDP software was specifically designed to maximize performance on the main data path for an established connection. This optimization is achieved in part by placing some of the functions of TCP and UDP - such as demultiplexing received packets and calculating checksums - within the IP code. In addition, the Solaris 2 TCP implementation fully implements the RFC 1122 requirements for Slow Start, Nagle Algorithm, Round Trip Estimates, Congestion Avoidance and Fast Retransmit.
To provide the high performance needed to handle bulk data applications such as file transfer or Web access, the Solaris 2 TCP/IP implementation combines a previously two-pass process that involved calculating a TCP checksum and copying data between the kernel and the user application into a single pass. Termed integrated copy and checksum, this technique eliminates one entire pass over the user's data, significantly improving raw data throughput and reducing the CPU cycles to process the data.
Solaris 2 TCP has been further optimized with a hashed TCP connection lookup, which allows the system to efficiently support thousands of simultaneous connection. The kernel's per-connection data structure, which must be located for every received packet, is stored in a hashed data structure for rapid access. This enhancement supports the large number of simultaneous TCP connections required on login servers, FTP servers, and Web servers, and ensures that the cost to process a received packet grows little as the number of TCP connections increases.
Hashing is also used to improve performance at the IP layer. The IP routing table lookup code uses a hashing algorithm to locate the correct routing table entry for each packet sent or received. IP merges the functions of the ARP cache and the IP routing table into a single integrated IP routing table. This feature turns the previously two-stage process of looking up a destination first in the IP routing table and then in the ARP cache, into a single, fast hashed lookup. These enhancements ensures that the system can support large IP routing tables and improves performance for large servers with many network interfaces.
Finally, to greatly improve both raw performance and scaling for machines operating as login servers, the telnet and rlogin server functions have been moved into the kernel in Solaris 2. This reduces context switching and data copying over the previous approach, in which both services were structured as user-level daemon processes. Now, Solaris 2 servers can support large numbers of remote users logged in via telnet or rlogin.
Figure 3 TCP supports thousands of fast, reliable connections
Logical Interfaces for Virtual Host Services
The Solaris 2 TCP/IP implementation provides, at the Network layer, logical interfaces - the ability to configure multiple IP addresses on a computer system with only one or a few physical network interfaces. This feature is widely used by Web server operators. Using logical interfaces, a single Web server can be made to appear as multiple virtual hosts to its clients. The system administrator typically configures the system to have multiple host names listed in the Domain Name Service (DNS). Each host name translates to one of the system's IP addresses, which are configured using logical interfaces. The system can then be configured to export different Web pages on each address.
IP Multicasting Supports the New Generation of Computer Applications
The current trend in computer applications has been moving from shared workgroup applications using one-to-one and application-to-application connections, toward collaborative computing and multimedia applications with a need for one-to-many and many-to-many communications. Examples of such applications include:
- Live video feeds to multiple systems on a network
- Desktop and video conferencing applications
- Collaborative workgroup applications such as shared whiteboards
Other applications that could make use of multicast communications are systems that need to deliver updates to multiple servers simultaneously. Database systems could also update their servers in one session instead of establishing multiple point-to-point sessions between systems. These new applications present challenges for the network administrator tasked with adding them to a busy network. A solution is available, however, with the concept of IP multicasting.
IP multicasting is the transmission of an IP datagram to a group of hosts. An IP multicast router transmits only one copy of each datagram over a network, thereby saving network bandwidth and time by allowing one IP multicast datagram to be transmitted to multiple sites. The datagram is delivered to all members of the destination host group with the same "best effort" reliability as regular unicast IP datagrams.
The Internet Group Management Protocol (IGMP), per RFC 1112, provides the membership reporting mechanism that allows multicasting to work. The Solaris 2 TCP/IP implementation provides full host-side support for IP multicast and kernel support for multicast routing. A multicast routing protocol daemon is available separately.
Figure 4 IP Multicasting makes efficient use of network bandwidth by allowing one IP Multicast datagram to be transmitted to multiple sites.
Router Discovery and Multiple Default Routes for Robust IP Routing
The ICMP Router Discovery protocol uses a pair of ICMP messages - Router Advertisements and Router Solicitations - on multicast links. It allows routers to be discovered dynamically, eliminating the need for manually configuring lists of routes.
Each router periodically multicasts a Router Advertisement from each of its multicast interfaces, announcing the IP address of that interface. Hosts discover the addresses of their neighboring routers by listening for advertisements. When a host attached to a multicast link starts up, it can also multicast a Router Solicitation to ask for immediate advertisements, rather than wait for the periodic one to arrive.
ICMP Router Discovery is not a routing protocol, since it only allows a router to discover neighboring routers, not necessarily which router provides the best route to a destination.
The Solaris 2 TCP/IP implementation of ICMP Router Discovery is as a user- level daemon process. It allows a Solaris 2 host to automatically and dynamically configure IP routing without needing to participate in the routing protocol. When operating as an IP router, Solaris 2 TCP/IP supports the router specific parts of RFC 1256.
Each discovered router is treated as a default router. The IP routing code employs each default router in a round-robin fashion, providing robustness against the failure of an individual router.
Figure 5 Protocols at the Network Layer optimize the data path for fast, efficient transmission of datagrams
Path MTU Discovery Provides Efficient Transport of Datagrams
Path MTU Discovery is a product of the IETF MTU Discovery Working Group. It is an algorithm that automatically determines the maximum transmission unit (MTU) - the size of the largest packet that can be transmitted to any IP destination. The path MTU is the minimum MTU on any network link currently in the path between two hosts.
Path MTU (PMTU) Discovery is useful when two hosts are communicating over a wide-area network containing links with different MTUs. For example, the MTU on an FDDI network is 4352 bytes, while the MTU for a point-to- point link may only be 296 bytes. The difference in the sizes of the MTUs for these networks requires that datagrams transmitted between these two links be fragmented.
Once fragmented, each datagram becomes its own packet with enough information to be routed independently of any other packet, and so should be able to transparently make it to its destination where it can be reassembled. If a fragment is lost during transmission, however, the entire datagram must be retransmitted, and therein lies a problem: In wide-area networks, retransmitting lost datagrams significantly increases network traffic. Today's wide-area networks are already experiencing increases in traffic form both additional users and bandwidth-intensive applications that contend for network bandwidth; they do not need additional traffic.
PMTU Discovery avoids IP fragmentation over wide-area paths by using the Don't Fragment (DF) bit in the IP header to dynamically discover the MTU of a path. It works by allowing the source host to initially assume that the MTU of a path is the known MTU of its first hop and sends all datagrams on that path with the DF bit set. If any of the datagrams are too large to be forwarded without fragmentation by some router along the path, that router discards the datagrams and returns an ICMP Destination Unreachable message that says in effect "packet too big to be transmitted without fragmentation." When the source receives the message, it reduces its assumed MTU for the path. The discovery process ends when the source determines that the PMTU is low enough that datagrams can be delivered without any fragmentation.
In the Solaris 2 TCP/IP implementation, TCP uses the PMTU Discovery to optimally segment data and avoid fragmentation. UDP uses the PMTU Discovery to optimally fragment datagrams. The result is less fragmentation and less retransmission of lost datagrams. The benefit is an optimized data path and more efficient transport of datagrams.
Internet Services
The first set of application-layer protocols developed for the experimental ARPAnet were Telnet, an acronym for telecommunications network protocol, and FTP, the file transfer protocol. Since that time, many more protocols have been developed and there is a standard set of services included with almost every implementation of TCP/IP, including Solaris 2 TCP/IP. These are:
- File Transfer Protocol (FTP)
- Telnet
- Simple Mail Transport Protocol (SMTP)
- Trivial File Transfer Protocol (TFTP)
- Domain Name System (DNS)
Additionally, Solaris 2 TCP/IP includes the UNIX rsh, rcp, and rlogin utilities, which allow UNIX users to login to remote systems, execute commands, and copy files.
Figure 6 Internet services are structured at the upper layers of the OSI Model
Summary
In the dozen or so years since Sun Microsystems decided to base their network technology on TCP/IP, data link technologies have advanced at a pace that was unimaginable in the early days of the Internet. That TCP/IP evolved with these technologies is a result of its concept of open standards. Yet TCP/IP and the Internet continue to evolve. IPv6, TCP/IP's next major evolution, will significantly increase the number of addresses available to support the rapidly growing number of users. Sun supports the open systems philosophy, and will support the IETF and their next version of TCP/IP. Sun has contributed much time and resources to the IETF and its various working groups, and will continue to do so.
Solaris 2 TCP/IP fully supports the Internet's visually appealing Web pages and the Electronic Commerce. The Java programming language promises access to thousands of new applets (small applications), and it is TCP/IP that will make the applets easily accessible to Internet users.
Sun workstations have long been fast Internet server performance leaders. To maintain that lead, Sun designed Solaris 2 TCP/IP to take full advantage of today's multiprocessor hardware technology. Additional performance enhancements include improved data throughput and optimized performance over wide-area networks.
The Path MTU Discovery algorithm provides efficient transport of datagrams by keeping them from being fragmented over differently sized network links, thereby preventing time-consuming retransmission of the datagrams.
IP multicasting makes more efficient use of network bandwidth by allowing one multicast datagram to be transmitted to multiple, predetermined sites, rather than flooding the network with datagrams that are transmitted to all sites.
The ICMP Router Discovery mechanism allows a Solaris 2 host to dynamically configure IP routing, and therefore protects against the failure of an individual router by providing alternate paths for delivering the datagram.
Sun continues its high performance leadership by designing Solaris 2 TCP/IP to support thousands of simultaneous TCP connections and thousands of simultaneous remote logins. Sun also integrated internal TCP processes to significantly speed up functions such as the bulk data transfers associated with files on the Web. At the network layer, Sun workstations feature a logical interface capable of supporting multiple IP addresses per physical network interface. This feature allows Web server operators to configure a single Web server to appear as multiple virtual hosts to clients.
Sun has been shipping TCP/IP with their operating systems since 1982, and a large number of Intranet and Internet servers are based on Solaris 2 and its TCP/IP implementation. The network administrator can feel confident choosing Solaris 2 TCP/IP for the corporate Intranet. Today Solaris 2 TCP/IP is a well-tested and proven workhorse for transferring data on the corporate Intranet, and it is robust enough to handle the increases in network traffic tomorrow.
References
Cortese, Amy, "The Software Revolution," BusinessWeek, December 4, 1995.
Deering, S., "Host Extensions for IP Multicasting," RFC 1112, DDN Network Information Center, Government Systems, Inc., Chantilly, VA 22021.
Deering, S., "ICMP Router Discovery Messages," RFC 1256, DDN Network Information Center, Government Systems, Inc., Chantilly, VA 22021.
IETF, "Requirements for Internet Hosts - Communication Layers," RFC 1122, DDN Network Information Center, Government Systems, Inc., Chantilly, VA 22021.
JavaSoft press release, Programming environment available free for developers, Sun Microsystems, Inc., 1996.
Mogul, J., "Path MTU Discovery," RFC 1191, DDN Network Information Center, Government Systems, Inc., Chantilly, VA 22021.
Reynolds, J. and J. Postel, "The Request for Comments Reference Guide," RFC 1000, DDN Network Information Center, Government Systems, Inc., Chantilly, VA 22021.
Stevens, W. Richard, TCP/IP Illustrated, Volume 1: The Protocols, Addison-Wesley Publishing Company, Reading, MA, 1994.
Sun Microsystems, Inc., Solaris 2 TCP/IP Software Overview, 1995.
SunSoft Internet Engineering Group slides, Solaris 2 TCP/IP Overview, Sun Microsystems, Inc., 1995.
