LinuxHPC.org/Cluster Builder 1.3
Ethernet
Ethernet
By LinuxHPC.org and Cluster Resources
| Ethernet |
|
Ethernet is a large and diverse family of frame-based computer networking technologies for local area networks (LANs). The name comes from the physical concept of the ether. It defines a number of wiring and signaling standards for the physical layer, two means of network access at the Media Access Control (MAC)/data link layer, and a common addressing format.
Ethernet has been standardized as IEEE 802.3. Its star-topology, twisted-pair wiring form became the most widespread LAN technology in use from the 1990s to the present, largely replacing competing LAN standards such as Coaxial-cable Ethernet, Token Ring, FDDI, and ARCNET. In recent years, WiFi, the wireless LAN standardized by IEEE 802.11, has been used instead of Ethernet in many installations.
General description
Ethernet was originally based on the idea of computers communicating over a shared coaxial cable acting as a broadcast transmission medium. The methods used show some similarities to radio systems (though there are major differences, like the fact that it is much easier to detect collisions in a cable broadcast system than a radio broadcast). The common cable providing the communication channel was likened to the ether and it was from this reference that the name 'Ethernet' was derived.
From this early and comparatively simple concept Ethernet evolved into the complex networking technology that today powers the vast majority of local computer networks. The coaxial cable was later replaced with point-to-point links connected together by hubs and/or switches in order to reduce installation costs, increase reliability, and enable point-to-point management and troubleshooting. StarLAN was the first step in the evolution of Ethernet from a coaxial cable bus to a hub-managed, twisted pair network. The advent of twisted-pair wiring enabled Ethernet to become a commercial success.
On top of the physical layer Ethernet stations communicate to each other by sending each other data packets, small blocks of data that are individually sent and delivered. As with other IEEE 802 LANs, each Ethernet station is given a single 48-bit MAC address, which is used both to specify the destination and the source of each data packet. Network interface cards (NICs) or chips normally do not accept packets addressed to other Ethernet stations. Adapters generally come programmed with a globally unique address but this can be overridden either to avoid an address change when an adapter is replaced or to use locally administered addresses.
Despite the huge changes in Ethernet from a thick coaxial cable bus running at 10 Mbit/s to point-to-point links running at 1 Gbit/s and beyond, the different variants remain essentially the same from the programmer's point of view and are easily interconnected using readily available inexpensive hardware. This is because the frame format remains the same, even though network access procedures are radically different.
Due to the ubiquity of Ethernet, the ever-decreasing cost of the hardware needed to support it and the reduced panel space needed by twisted pair Ethernet, most manufacturers now build the functionality of an Ethernet card directly into PC motherboards obviating the need for installation of a separate network card.
Physical layer
Ethernet evolved over a considerable time span and encompasses quite a few physical media interfaces. The commonly installed Gigabit Ethernet over copper wiring use a PAM-5 modulation scheme and over fiber uses 8B/10B encoding.
Dealing with multiple users
CSMA/CD shared medium Ethernet
Ethernet originally used a shared coaxial cable (the shared medium) winding around a building or campus to every attached machine. A scheme known as carrier sense multiple access with collision detection (CSMA/CD) governed the way the computers share the channel. The scheme was relatively simple compared to competing technologies token ring or Token Bus. When one computer wanted to send some information, it followed the following algorithm:
Main procedure
This works something like a dinner party, where all the guests talk to each other through a common medium (the air). Before speaking, each guest politely waits for the current guest to finish. If two guests start speaking at the same time, both stop and wait for short, random periods of time (measured in microseconds). The hope is that by each choosing a random period of time, both guests will not choose the same time to try to speak again, thus avoiding another collision. Exponentially increasing back-off times (determined using the truncated binary exponential backoff algorithm) are used when there is more than one failed attempt to transmit.- Frame ready for transmission
- Is medium idle? If not, wait until it becomes ready and wait the Interframe gap period (9.6 µs in 10 Mbit/s Ethernet).
- Start transmitting
- Does a collision occur? If so, go to collision detected procedure.
- End successful transmission
- Continue transmission until minimum packet time is reached (jam signal) to ensure that all receivers detect the collision
- Is maximum number of transmission attempts reached? If so, abort transmission.
- Calculate and wait random backoff period
- Re-enter main procedure at stage 1
Computers were connected to an Attachment Unit Interface (AUI) transceiver, which in turn connected to the cable. While a simple passive wire was highly reliable for small Ethernets, it was not reliable for large extended networks, where damage to the wire in a single place, or a single bad connector could make the whole Ethernet segment unusable. Multipoint systems are also prone to very strange failure modes when an electrical discontinuity reflects the signal in such a manner that some nodes would work just fine while others would work slowly due to excessive retries or not at all (see standing wave for an explanation of why); these could be much more painful to diagnose than a complete failure of the segment. Debugging such failures often involved several people crawling around wiggling connectors while others watched the displays of computers running ping and shouted out reports as performance changed.
Since all communications happen on the same wire, any information sent by one computer is received by all, even if that information was intended for just one destination. The network interface card filters out information not addressed to it, interrupting the CPU only when applicable packets are received unless the card is put into "promiscuous mode". This "one speaks, all listen" property is a security weakness of shared-medium Ethernet, since a node on an Ethernet network can eavesdrop on all traffic on the wire if it so chooses. Use of a single cable also means that the bandwidth is shared, so that network traffic can slow to a crawl when, for example, the network and nodes restart after a power failure.
Ethernet repeaters and hubs
For signal degradation and timing reasons, coaxial Ethernet segments had a restricted size which depended on the medium used. For example, 10BASE5 coax cables had a maximum length of 500 metres (1,640 feet). Also, as was the case with most other high-speed buses, Ethernet segments had to be terminated with a resistor at both ends. For coaxial cable based Ethernet, each end of the cable had a 50-ohm resistor and heat sink attached. Typically this was built into a male BNC or N connector and attached to the last device on the bus (or if vampire taps were in use to a socket mounted on the end of the cable just past the last device). If this was not done or if there was a break in the cable the AC signal on the bus was reflected, rather than dissipated, when it reached the end. This reflected signal was indistinguishable from a collision, and so no communication could take place.
A greater length could be obtained by an Ethernet repeater, which took the signal from one Ethernet cable and repeated it onto another cable. If a collision was detected, the repeater transmitted a jam signal onto all ports to ensure collision detection. Repeaters could be used to connect segments such that there were up to five Ethernet segments between any two hosts, three of which could have attached devices. Repeaters could detect an improperly terminated link from the continuous collisions and stop forwarding data from it. Hence they alleviated the problem of cable breakages: when an Ethernet coax segment broke, while all devices on that segment were unable to communicate, repeaters allowed the other segments to continue working (though depending on which segment was broken and the layout of the network the partitioning that resulted may have made other segments unable to reach important servers and thus effectively useless).
People recognized the advantages of cabling in a star topology (primarily that only faults at the star point will result in a badly partitioned network), and network vendors started creating repeaters having multiple ports, thus reducing the number of repeaters required at the star point; multiport Ethernet repeaters became known as "hubs". Network vendors such as DEC and SynOptics sold hubs that connected many 10BASE2 thin coaxial segments. There were also "multi-port transceivers" or "fan-outs". These could be connected to each other and/or a coax backbone. The best-known early example was DEC's DELNI. These devices allowed multiple hosts with AUI connections to share a single transceiver. They also allowed creation of a small standalone Ethernet segment without using a coaxial cable.
Ethernet on unshielded twisted-pair cables (UTP), beginning with StarLAN and continuing with 10BASE-T was designed for point-point links only and all termination was built into the device. This changed hubs from a specialist device used at the center of large networks to a device that every twisted pair-based network with more than two machines had to use. This structure made Ethernet networks more reliable by preventing faults with (but not deliberate misbehaviour of) one peer or its associated cable from affecting other devices on the network (though a failure of a hub or an inter hub link could still affect lots of users). Also as twisted-pair ethernet is point-to-point and terminated inside the hardware the total empty panel space required around a port is much reduced (making it easier to design hubs with lots of ports and to integrate ethernet onto computer motherboards).
Despite the physical star topology, hubbed Ethernet networks use half-duplex and CSMA/CD, with only minimal activity by the hub, primarily the Collision Enforcement signal, in dealing with packet collisions. Every packet is sent to every port on the hub, so bandwidth and security problems aren't addressed. The total throughput of the hub is limited to that of a single link and all links must operate at the same speed.
Collisions reduce throughput by their very nature. In the worst case, when there are lots of hosts with long cables that attempt to transmit many short frames, excessive collisions can reduce throughput dramatically. However, a Xerox report in 1980 summarized the results of having 20 fast nodes attempting to transmit packets of various sizes as quickly as possible on the same Ethernet segment[citation needed] . The results showed that, even for minimal Ethernet frames (64B), 90% throughput on the LAN was the norm. This is in comparison with token passing LANs (Token Ring, Token Bus), all of which suffer throughput degradation as each new node comes into the LAN, due to token waits.
This report was wildly controversial, as modeling showed that collision-based networks became unstable under loads as low as 40% of nominal capactity.
Bridging and Switching
While repeaters could isolate some aspects of Ethernet segments, such as cable breakages, they still forwarded all traffic to all Ethernet devices. This created significant limits on how many machines could communicate on an Ethernet network. To alleviate this, bridging was created to communicate at the data link layer while isolating the physical layer. With bridging, only well-formed packets are forwarded from one Ethernet segment to another; collisions and packet errors are isolated. Bridges learn where devices are, by watching MAC addresses, and do not forward packets across segments when they know the destination address is not located in that direction.
Early bridges examined each packet one by one, and were significantly slower than hubs (repeaters) at forwarding traffic, especially when handling many ports at the same time. In 1989 the networking company Kalpana introduced their EtherSwitch, the first Ethernet switch. An Ethernet switch does bridging in hardware, allowing it to forward packets at full wire speed. Bridges and switches also allow mixing of speeds, an imporant feature when equipment of mixed age is in use. Even more importantly (especially with fast ethernet) they overcome the cascading limits of hubs as a collision does not have to be detected by equipment on the other side of the switch or bridge.
Initially, Ethernet bridges and switches work somewhat like Ethernet hubs, with all traffic being echoed to all ports. However, as the switch "learns" the end-points associated with each port, it ceases to send non-broadcast traffic to ports other than the intended destination. In this way, Ethernet switching can allow the full wire speed of Ethernet to be used by any given pair of ports on a single switch.
Since packets are typically only delivered to the port they are intended for, traffic on a switched Ethernet is slightly less public than on shared-medium Ethernet. Despite this, switched Ethernet should still be regarded as an insecure network technology, because it is easy to subvert switched Ethernet systems by means such as ARP spoofing and MAC flooding. The bandwidth advantages, the slightly better isolation of devices from each other and the elimination of the chaining limits inherent in hubbed Ethernet have made switched Ethernet the dominant network technology.
When only a single device (anything but a hub) is connected to a switch port, full-duplex Ethernet becomes possible. In full duplex mode both devices can transmit to each other at the same time and there is no collision domain. This doubles the aggregate bandwidth of the link and was sometimes advertised as double the link speed (e.g. 200 Mbit/s) to account for this. However, this is misleading as performance will only double if traffic patterns are symmetrical (which in reality they rarely are). The elimination of the collision domain also means that all the link's bandwidth can be used (collisions can occupy a lot of bandwidth as links get busy)[citation needed] and that segment length is not limited by the need for correct collision detection (this is most significant with some of the fiber variants of Ethernet).
Dual speed hubs
In the early days of Fast Ethernet, fast ethernet switches were relatively expensive devices. However, hubs suffered from the problem that if there were any 10BASE-T devices connected then the whole system would have to run at 10 Mbit. Therefore a compromise between a hub and a switch appeared known as a dual speed hub. These effectively split the network into two sections, each acting like a hubbed network at its respective speed then acted as a two port switch between those two sections. This meant they allowed mixing of the two speeds without the cost of a Fast Ethernet switch.
More advanced networks
Simple switched Ethernet networks still suffer from a number of issues:
- They suffer from single points of failure; e.g., if one link or switch goes down in the wrong place the network ends up partitioned.
- It is possible to trick switches or hosts into sending data to your machine even if it's not intended for it, as indicated above.
- It is possible for any host to flood the network with broadcast traffic forming a denial of service attack against any hosts that run at the same or lower speed as the attacking device.
- They suffer from bandwidth choke points where a lot of traffic is forced down a single link.
- Spanning-tree protocol to maintain the active links of the network as a tree while allowing physical loops for redundancy.
- Various port protection features (as it is far more likely an attacker will be on an end system port than on a switch-switch link)
- VLANs to keep different classes of users separate while using the same physical infrastructure.
- Fast routing at higher levels (to route between those VLANs).
- Link aggregation to add bandwidth to overloaded links and to provide some measure of redundancy, although the links won't protect against switch failure because they connect the same pair of switches.
Varieties of Ethernet
The first Ethernet networks, 10BASE5, used thick yellow cable with vampire taps as a shared medium (using CSMA/CD). Later, 10BASE2 Ethernet used thinner coaxial cable (with BNC connectors) as the shared CSMA/CD medium. The later StarLAN 1BASE5 and 10BASE-T used twisted pair connected to Ethernet hubs with RJ-45 connectors.
Currently Ethernet has many varieties that vary both in speed and physical medium used. Perhaps the most common forms used are 10BASE-T, 100BASE-TX, and 1000BASE-T. All three utilize twisted pair cables and RJ-45 connectors. They run at 10 Mbit/s, 100 Mbit/s, and 1 Gbit/s, respectively. However each version has become steadily more selective about the cable it runs on and some installers have avoided 1000baseT for everything except short connections to servers.
Fiber optic variants of Ethernet are commonly seen connecting buildings or network cabinets in different parts of a building but are rarely seen connected to end systems for cost reasons. Their advantages lie in performance (fiber versions of a new speed almost invariablly come out before copper), distance (up to tens of kilometers with some versions) and electrical isolation. 10-gigabit Ethernet is becoming more popular in both enterprise and carrier networks, with discussions starting on 40G and 100G Ethernet.
Through Ethernet's history there have also been RF versions of Ethernet, both wireline and wireless. The currently recommended RF wireless networking standards, 802.11 and 802.16, are not based upon Ethernet.
Related standards
- Networking standards that are not part of the IEEE 802.3 Ethernet standard, but support the Ethernet frame format, and are capable of interoperating with it.
- LattisNet — A SynOptics pre-standard twisted-pair 10 Mbit/s variant.
- 100BaseVG — An early contender for 100 Mbit/s Ethernet. It runs over Category 3 cabling. Uses four pairs. Commercial failure.
- TIA 100BASE-SX — Promoted by the Telecommunications Industry Association. 100BASE-SX is an alternative implementation of 100 Mbit/s Ethernet over fiber; it is incompatible with the official 100BASE-FX standard. Its main feature is interoperability with 10BASE-FL, supporting autonegotiation between 10 Mbit/s and 100 Mbit/s operation -- a feature lacking in the official standards due to the use of differing LED wavelengths. It is targeted at the installed base of 10 Mbit/s fiber network installations.
- TIA 1000BASE-TX — Promoted by the Telecommunications Industry Association, it was a commercial failure, and no products exist. 1000BASE-TX uses a simpler protocol than the official 1000BASE-T standard so the electronics can be cheaper, but requires Category 6 cabling.
- Networking standards that do not use the Ethernet frame format but can still be connected to Ethernet using MAC-based bridging.
- 802.11 — A standard for wireless networking often paired with an Ethernet backbone.
- Long Reach Ethernet
- Avionics Full-Duplex Switched Ethernet