Data Center Architecture

• Modern data centers are highly available, scalable network infrastructures designed to support enterprise applications, cloud services, and storage systems
• Built using three-tier or spine-leaf architectures to provide redundancy, high bandwidth, and low latency
• Core design principles include redundancy at every layer, predictable performance, and rapid scalability

Three-Tier Data Center Design

• Access Layer: Connects servers directly (ToR switches - Top of Rack)
• Aggregation/Distribution Layer: Aggregates access switches and provides policy enforcement
• Core Layer: High-speed backbone connecting aggregation switches

Limitations of Three-Tier

• Oversubscription ratios can create bottlenecks (typically 4:1 or 8:1 at access layer)
• East-West traffic (server-to-server) must traverse multiple hops
• Limited scalability due to Spanning Tree Protocol (STP) blocking redundant links

Spine-Leaf Architecture (Clos Network)

• Leaf Switches: Connect directly to servers (equivalent to access layer)
• Spine Switches: Interconnect all leaf switches (no leaf-to-leaf connections)
• Every leaf connects to every spine creating a full-mesh topology
• Enables Equal-Cost Multi-Path (ECMP) routing for optimal bandwidth utilization

Spine-Leaf Benefits

• Consistent latency (maximum 3 hops: leaf → spine → leaf)
• Linear scalability (add spine-leaf pairs as needed)
• No STP blocking - all links actively forward traffic
• Predictable bandwidth allocation

Data Center Protocols and Technologies

VXLAN (Virtual Extensible LAN)

• Overlay protocol that encapsulates Layer 2 frames in UDP packets
• Uses 24-bit VXLAN Network Identifier (VNI) allowing 16 million virtual networks (vs 4,094 VLANs)
• Enables Layer 2 connectivity across Layer 3 boundaries
• Default UDP port: 4789

EVPN (Ethernet VPN)

• Control plane protocol for VXLAN using BGP
• Provides MAC and IP learning through BGP route advertisements
• Eliminates flooding for unknown unicast traffic
• Supports multi-tenancy with Route Targets (RT) and Route Distinguishers (RD)

Fabric Extenders (FEX)

• Remote line cards that extend parent switch interfaces
• Appear as additional ports on parent switch (single management point)
• Typically used in Cisco Nexus environments for server connectivity
• Cannot perform local switching - all traffic returns to parent switch

Data Center Switching Protocols

Virtual Port Channel (vPC)

• Cisco technology allowing dual-homed devices to connect to two switches simultaneously
• Both links appear as single port-channel to connected device
• Requires vPC peer-link between switches (typically 10G+ with multiple links)
• vPC peer-keepalive uses management network to prevent split-brain scenarios

Storage Networking

Fibre Channel over Ethernet (FCoE)

• Encapsulates Fibre Channel frames in Ethernet frames
• Requires Data Center Bridging (DCB) for lossless transport
• Uses dedicated VLAN for storage traffic
• Eliminates need for separate FC infrastructure in converged networks

iSCSI (Internet Small Computer Systems Interface)

• Block storage protocol over TCP/IP networks
• Uses standard Ethernet infrastructure (no special hardware required)
• Typically implemented with dedicated storage VLANs
• Less expensive than Fibre Channel but higher CPU overhead

Vocabulary

Oversubscription Ratio: The ratio of total server bandwidth to uplink bandwidth (e.g., 48 × 1G servers with 4 × 10G uplinks = 48:40 or 1.2:1 ratio)

East-West Traffic: Server-to-server communication within the data center (typically 70-80% of modern DC traffic)

North-South Traffic: Traffic between data center and external networks (users, internet, other sites)

Top of Rack (ToR): Switch placement strategy where access switches are located in each server rack

End of Row (EoR): Alternative placement where fewer, larger switches serve multiple racks from end of row

Converged Network: Single network infrastructure carrying multiple traffic types (data, storage, voice)

Notes

• Modern data centers typically use 40G/100G+ spine connections with 10G/25G server connections
• Always implement redundant power, cooling, and network paths for high availability
• Leaf switches should have consistent uplink bandwidth to avoid creating bottlenecks
• Consider buffer sizes and latency requirements when selecting spine-leaf hardware for specific applications
• VXLAN requires hardware VTEP (VXLAN Tunnel Endpoint) support for line-rate performance
• In spine-leaf designs, never connect leaf switches directly to each other as this breaks the Clos topology benefits
• Plan for 3-5 years of growth when sizing spine switch port density
• Use dedicated out-of-band management network for device access during network failures