Editing Itanium (section)

=== Chipsets ===
Prior to the 9300-series ([[Tukwila (processor)|Tukwila]]), chipsets were needed to connect to the main memory and I/O devices, as the [[front-side bus]] to the [[chipset]] was the sole operational connection to the processor.{{efn| the processor supported TAP ([[JTAG]]) and [[SMBus]] for debugging and system configuration}} Two generations of buses existed: the original ''Itanium processor system bus'' (a.k.a. ''Merced bus'') had a 64 bit data width and 133&nbsp;MHz clock with [[Double data rate|DDR]] (266 MT/s), being soon superseded by the 128-bit 200&nbsp;MHz DDR (400 MT/s) ''Itanium 2 processor system bus'' (a.k.a. ''McKinley bus''), which later reached 533 and 667 MT/s. Up to four CPUs per single bus could be used, but prior to the 9000-series the bus speeds of over 400 MT/s were limited to up to two processors per bus.<ref>{{cite web |title=Intel® Itanium® 2 Processor Datasheet |url=http://download.intel.com/design/Itanium2/datashts/25094505.pdf |page=9 |publisher=Intel |archive-url=https://web.archive.org/web/20060621013059/http://download.intel.com/design/Itanium2/datashts/25094505.pdf |archive-date=21 June 2006 |url-status=dead}}</ref><ref>{{cite web |title=Dual-Core Intel® Itanium® 2 Processor 9000 Series Datasheet |url=http://download.intel.com/design/Itanium2/datashts/31405401.pdf |page=9 |publisher=Intel |archive-url=https://web.archive.org/web/20110607124058/http://download.intel.com/design/Itanium2/datashts/31405401.pdf |archive-date=7 June 2011 |url-status=dead}}</ref> As no Itanium chipset could connect to more than four sockets, high-end servers needed multiple interconnected chipsets.

The "Tukwila" Itanium processor model had been designed to share a common chipset with the Intel Xeon processor EX (Intel's Xeon processor designed for four processor and larger servers). The goal was to streamline system development and reduce costs for server OEMs, many of which develop both Itanium- and Xeon-based servers. However, in 2013, this goal was pushed back to be "evaluated for future implementation opportunities".<ref>{{cite news| url = https://www.theregister.com/2013/02/08/intel_kills_itanium_xeon_convergence_and_kittson/| title = Remember that Xeon E7-Itanium convergence? FUHGEDDABOUDIT| first = Timothy Prickett| last = Morgan| work = [[The Register]]| access-date = November 25, 2022}}</ref>

In the times before on-chip memory controllers and [[QPI]], enterprise server manufacturers differentiated their systems by designing and developing chipsets that interface the processor to memory, interconnections, and peripheral controllers. "Enterprise server" referred to the then-lucrative market segment of high-end servers with high [[reliability, availability and serviceability]] and typically 16+ processor sockets, justifying their pricing by having a custom system-level architecture with their own chipsets at its heart, with capabilities far beyond what two-socket "commodity servers" could offer. Development of a chipset costs tens of millions of dollars and so represented a major commitment to the use of Itanium.

Neither Intel nor IBM would develop Itanium 2 chipsets to support newer technologies such as [[DDR2 SDRAM|DDR2]] or [[PCI Express]].<ref name="ibm_ditching_itanium">{{cite news
 | url=https://www.cnet.com/tech/tech-industry/sources-ibm-ditching-itanium-altogether/
 | title=Sources: IBM ditching Itanium altogether
 | access-date=July 4, 2023
 | last=Shankland
 | first=Stephen
 | date=February 25, 2005
 | publisher=[[CNET|CNET News]]
 }}</ref>
Before "Tukwila" moved away from the FSB, chipsets supporting such technologies were manufactured by all Itanium server vendors, such as HP, Fujitsu, SGI, NEC, and Hitachi.

==== Intel ====
The first generation of Itanium received no vendor-specific chipsets, only Intel's 460GX consisting of ten distinct chips. It supported up to four CPUs and 64 GB of memory at 4.2 GB/s, which is twice the system bus's bandwidth. Addresses and data were handled by two different chips. 460GX had an [[Accelerated Graphics Port|AGP]] X4 graphics bus, two 64-bit 66&nbsp;MHz [[Peripheral Component Interconnect|PCI]] buses and configurable 33&nbsp;MHz dual 32-bit or single 64-bit PCI bus(es).<ref>{{cite web |title=Intel 460GX Chipset Datasheet |url=http://developer.intel.com/design/itanium/downloads/24870301.pdf |publisher=Intel |archive-url=https://web.archive.org/web/20040723073549/http://developer.intel.com/design/itanium/downloads/24870301.pdf |archive-date=23 July 2004 |url-status=dead}}</ref>

There were many custom chipset designs for Itanium 2, but many smaller vendors chose to use Intel's E8870 chipset. It supports 128 GB of [[DDR SDRAM]] at 6.4 GB/s. It was originally designed for [[Rambus]] [[RDRAM]] [[serial communication|serial]] memory, but when RDRAM failed, Intel added four DDR SDRAM-to-RDRAM converter chips to the chipset.<ref>{{cite book |last1=Mueller |first1=Scott |last2=Soper |first2=Mark Edward |last3=Sosinsky |first3=Barrie |title=Upgrading and Repairing Servers |date=2006 |publisher=Pearson Education |isbn=0-13-279698-8 |url=https://books.google.com/books?id=9cLFf_1PBnkC&pg=PT301 |access-date=6 April 2022}}</ref> When Intel had previously made such a converter for Pentium III chipsets 820 and 840, it drastically cut performance.<ref>{{cite web |last1=Shimpi |first1=Anand Lal |title=Intel's 820 Chipset - Performance using SDRAM |url=https://www.anandtech.com/show/465 |website=[[AnandTech]] |access-date=6 April 2022}}</ref><ref>{{cite web |last1=Shimpi |first1=Anand Lal |title=Rambus DRAM Part 2: Performance |url=https://www.anandtech.com/show/551 |website=[[AnandTech]] |access-date=6 April 2022}}</ref> E8870 provides eight 133&nbsp;MHz [[PCI-X]] buses (4.2 GB/s total because of bottlenecks) and a [[I/O Controller Hub#ICH4|ICH4]] hub with six [[USB 2.0]] ports.
Two E8870 can be linked together by two E8870SP Scalability Port Switches, each containing a 1MB (~200,000 cache lines) [[Bus snooping#Snoop filter|snoop filter]], to create an 8-socket system with double the memory and PCI-X capacity, but still only one ICH4. Further expansion to 16 sockets was planned.<ref>{{cite journal |display-authors=etal |last=Briggs |first=Fayé |title=Intel 870: a building block for cost-effective, scalable servers |journal=[[IEEE Micro]] |date=7 August 2002 |volume=22 |issue=2 (March–April) |pages=36–47 |doi=10.1109/MM.2002.997878 |citeseerx=10.1.1.140.2915 |s2cid=3201355 }}</ref><ref>{{cite web |title=Intel® E8870 Scalable Node Controller (SNC) Datasheet |url=http://www.intel.com/design/chipsets/datashts/25111203.pdf |publisher=Intel |archive-url=https://web.archive.org/web/20040701014149/http://www.intel.com/design/chipsets/datashts/25111203.pdf |archive-date=1 July 2004 |url-status=dead}}</ref><ref>{{cite web |title=Intel® E8870IO Server I/O Hub (SIOH) Datasheet |url=http://intel.com/design/chipsets/datashts/25111103.pdf |publisher=Intel |archive-url=https://web.archive.org/web/20030706004227/http://intel.com/design/chipsets/datashts/25111103.pdf |archive-date=6 July 2003 |url-status=dead}}</ref> In 2004 Intel revealed plans for its next Itanium chipset, codenamed ''Bayshore'', to support [[PCI-e]] and [[DDR2 SDRAM|DDR2]] memory, but canceled it the same year.<ref>{{cite web |title=Intel Outlines Platform Innovations For More Manageable, Balanced And Secure Enterprise Computing |url=https://www.intel.com/pressroom/archive/releases/2004/20040218corp.htm |publisher=Intel |access-date=7 April 2022}}</ref><ref name="ibm_ditching_itanium"/>

==== Hewlett-Packard ====
HP has designed four different chipsets for Itanium 2: zx1, sx1000, zx2 and sx2000. All support 4 sockets per chipset, but sx1000 and sx2000 support interconnection of up to 16 chipsets to create up to a 64 socket system. As it was developed in collaboration with Itanium 2's development, booting the first Itanium 2 in February 2001,<ref>{{cite web |title=Overview of the new Itanium® 2-based HP servers rx2600 and rx5670: how HP is putting Intel® Itanium 2 processors to work |url=http://www.hp.com/products1/itanium/infolibrary/whitepapers/5981_2240EN.pdf |page=17 |publisher=Hewlett-Packard |archive-url=https://web.archive.org/web/20030319214329/http://www.hp.com/products1/itanium/infolibrary/whitepapers/5981_2240EN.pdf |archive-date=19 March 2003 |url-status=dead}}</ref> zx1 became the first Itanium 2 chipset available and later in 2004 also the first to support 533 MT/s FSB. In its basic two-chip version it directly provides four channels of [[DDR SDRAM|DDR-266]] memory, giving 8.5 GB/s of bandwidth and 32 GB of capacity (though 12 DIMM slots).<ref>{{cite web |title=HP Integrity rx2620 Server |url=http://h18000.www1.hp.com/products/quickspecs/12141_div/12141_div.PDF |archive-url=https://web.archive.org/web/20061029145359/http://h18000.www1.hp.com/products/quickspecs/12141_div/12141_div.PDF |archive-date=29 October 2006 |url-status=dead}}</ref> In versions with memory expander boards memory bandwidth reaches 12.8 GB/s, while the maximum capacity for the initial two-board 48 DIMM expanders was 96 GB, and the later single-board 32 DIMM expander up to 128 GB. The memory latency increases by 25 nanoseconds from 80 ns due to the expanders. Eight independent links went to the PCI-X and other peripheral devices (e.g. [[Accelerated Graphics Port|AGP]] in workstations), totaling 4 GB/s.<ref>{{cite web |title=HP Integrity rx4640-8 Server |url=http://h18000.www1.hp.com/products/quickspecs/11847_div/11847_div.PDF |archive-url=https://web.archive.org/web/20060314004913/http://h18000.www1.hp.com/products/quickspecs/11847_div/11847_div.PDF |archive-date=14 March 2006 |url-status=dead}}</ref><ref>{{cite web |title=HP Integrity rx5670 Server summary |url=http://www.hp.com/products1/servers/integrity/entry_level/rx5670/index.html |archive-url=https://web.archive.org/web/20041209002029/http://www.hp.com/products1/servers/integrity/entry_level/rx5670/index.html |archive-date=9 December 2004 |url-status=dead}}</ref>

HP's first high-end Itanium chipset was sx1000, launched in mid-2003 with the [[HP Superdome|Integrity Superdome]] flagship server.
It has two independent front-side buses, each bus supporting two sockets, giving 12.8 GB/s of combined bandwidth from the processors to the chipset. It has four links to data-only memory buffers and supports 64 GB of HP-designed 125&nbsp;MHz memory at 16 GB/s. The above components form a system board called a ''cell''. Two cells can be directly connected together to create an 8-socket [[Glue logic|glue]]less system. To connect four cells together, a pair of 8-ported [[crossbar switch]]es is needed (adding 64 [[Nanosecond|ns]] to inter-cell memory accesses), while four such pairs of crossbar switches are needed for the top-end system of 16 cells (64 sockets), giving 32 GB/s of [[bisection bandwidth]]. Cells maintain cache coherence through in-memory [[Directory-based cache coherence|directories]], which causes the minimum memory latency to be 241 ns. The latency to the most remote ([[Non-uniform memory access|NUMA]]) memory is 463 ns. The per-cell bandwidth to the I/O subsystems is 2 GB/s, despite the presence of 8 GB/s worth of PCI-X buses in each I/O subsystem.<ref>{{cite web |last1=Turner |first1=Vernon |last2=Rau |first2=Shane |title=HP's sx1000 Chipset: Innovation Atop Standardization |url=http://h71028.www7.hp.com/ERC/downloads/sx1000_White_Paper_.pdf |publisher=[[International Data Group|IDC]] (sponsored by HP) |archive-url=https://web.archive.org/web/20050601104604/http://h71028.www7.hp.com/ERC/downloads/sx1000_White_Paper_.pdf |archive-date=1 June 2005 |url-status=dead}}</ref><ref>{{cite web |title=Meet the HP Integrity Superdome: A white paper from HP |url=http://www.hp.com/products1/servers/integrity/superdome_high_end/infolibrary/Superdome_wp.pdf |archive-url=https://web.archive.org/web/20040731205815/http://www.hp.com/products1/servers/integrity/superdome_high_end/infolibrary/Superdome_wp.pdf |archive-date=31 July 2004 |url-status=dead}}</ref><ref>{{cite web |title=Itanium®–based midrange servers from HP— the HP Integrity rx7620-16 and rx8620-32 Servers |url=http://h21007.www2.hp.com/dspp/files/unprotected/integritymidrangejan05.pdf |archive-url=https://web.archive.org/web/20050509234702/http://h21007.www2.hp.com/dspp/files/unprotected/integritymidrangejan05.pdf |archive-date=9 May 2005 |url-status=dead}}</ref>

HP launched sx2000 in March 2006 to succeed sx1000. Its two FSBs operate at 533 MT/s. It supports up to 128 GB of memory at 17 GB/s. The memory is of HP's custom design, using the [[DDR2 SDRAM|DDR2]] protocol, but twice as tall as the standard modules and with redundant address and control signal contacts. For the inter-chipset communication, 25.5 GB/s is available on each sx2000 through its three [[Serial communication|serial]] links that can connect to a set of three [[Redundancy (engineering)|independent]] [[Crossbar switch|crossbars]], which connect to other cells or up to 3 other sets of 3 crossbars. The multi-cell configurations are the same as with sx1000, except the parallelism of the sets of crossbars has been increased from 2 to 3. The maximum configuration of 64 sockets has 72 GB/s of sustainable [[bisection bandwidth]]. The chipset's connection to its I/O module is now serial with an 8.5 GB/s peak and 5.5 GB/s sustained bandwidth, the I/O module having either 12 [[PCI-X]] buses at up to 266&nbsp;MHz, or 6 PCI-X buses and 6 [[PCIe]] 1.1 ×8 slots. It is the last chipset to support HP's [[PA-RISC]] processors ([[PA-8000#PA-8900|PA-8900]]).<ref>{{Cite web|url=http://archive.org/details/manualzilla-id-7031299|title=User Service Guide HP Integrity Superdome/sx2000 and HP 9000 Superdome/sx2000 Servers|publisher=[[Hewlett-Packard]]|date=September 2009|via=Internet Archive}}</ref>

HP launched the first zx2-based servers in September 2006. zx2 can operate the FSB at 667 MT/s with two CPUs or 533 MT/s with four CPUs. It connects to the [[DDR2 SDRAM|DDR2]] memory either directly, supporting 32 GB at up to 14.2 GB/s, or through expander boards, supporting up to 384 GB at 17 GB/s. The minimum open-page latency is 60 to 78 ns. 9.8 GB/s are available through eight independent links to the I/O adapters, which can include PCIe ×8 or 266&nbsp;MHz PCI-X.<ref>{{cite web |title=Overview of the HP Integrity rx2660, rx3600, and rx6600 Servers |url=https://shoredata.us.com/wp-content/uploads/2016/03/rx6600.pdf |archive-url=https://web.archive.org/web/20170306041015/https://shoredata.us.com/wp-content/uploads/2016/03/rx6600.pdf |archive-date=2017-03-06 |url-status=live}}</ref><ref>{{cite web |title=HP Integrity systems Family guide |url=https://www.hp.com/ch-de/pdf/harness_family_brosch_re_4aa3-4519enw_tcm_179_1247486.pdf |access-date=24 May 2022|archive-url=https://web.archive.org/web/20220708214847/https://www.hp.com/ch-de/pdf/harness_family_brosch_re_4aa3-4519enw_tcm_179_1247486.pdf|archive-date=8 July 2022|url-status=dead}}</ref>

==== Others ====
In May 2003, IBM launched the XA-64 chipset for Itanium 2. It used many of the same technologies as the first two generations of XA-32 chipsets for [[Xeon]], but by the time of the third gen XA-32 IBM had decided to discontinue its Itanium products. XA-64 supported 56 GB of [[DDR SDRAM]] in 28 slots at 6.4 GB/s, though due to bottlenecks only 3.2 GB/s could go to the CPU and other 2 GB/s to devices for a 5.2 GB/s total. The CPU's memory bottleneck was mitigated by an off-chip 64 MB [[DRAM]] L4 cache, which also worked as a [[Bus snooping#Snoop filter|snoop filter]] in multi-chipset systems. The combined bandwidth of the four [[PCI-X]] buses and other I/O is bottlenecked to 2 GB/s per chipset. Two or four chipsets can be connected to make an 8 or 16 socket system.<ref>{{cite web |title=IBM Eserver xSeries 455 Planning and Installation Guide |url=https://lenovopress.com/sg247056.pdf |publisher=IBM/Lenovo |access-date=6 April 2022}}</ref>

[[Silicon Graphics|SGI]]'s [[Altix]] supercomputers and servers used the SHUB (Super-Hub) chipset, which supports two Itanium 2 sockets. The initial version used [[DDR SDRAM|DDR memory]] through four buses for up to 12.8 GB/s bandwidth, and up to 32 GB of capacity across 16 slots. A 2.4 GB/s [[XIO]] channel connected to a module with up to six 64-bit 133&nbsp;MHz [[PCI-X]] buses. SHUBs can be interconnected by the dual 6.4 GB/s [[NUMAlink]]4 link planes to create a 512-socket cache-coherent single-image system. A cache for the in-memory [[Directory-based cache coherence|coherence directory]] saves memory bandwidth and reduces latency. The latency to the local memory is 132 ns, and each crossing of a NUMAlink4 router adds 50 ns. I/O modules with four 133&nbsp;MHz PCI-X buses can connect directly to the NUMAlink4 network.<ref>{{cite web |last1=Woodacre |first1=Michael |last2=Robb |first2=Derek |last3=Roe |first3=Dean |last4=Feind |first4=Karl |title=The SGI® Altix 3000 Global Shared-Memory Architecture |url=http://www.sgi.com/pdfs/3474.pdf |website=sgi.com |archive-url=https://web.archive.org/web/20060314142114/http://www.sgi.com/pdfs/3474.pdf |archive-date=2006-03-14 |url-status=dead}}</ref><ref>{{cite web |last1=Vogelsang |first1=Reiner |title=SGI® Altix™ Hardware Architecture |url=https://moodle.risc.jku.at/file.php/50/altix_hardware.pdf |access-date=25 April 2022 }}</ref><ref>{{cite web |title=SGI® Altix™ 350 System User's Guide |url=http://techpubs.sgi.com/library/manuals/4000/007-4660-002/pdf/007-4660-002.pdf |archive-url=https://web.archive.org/web/20160121032040/http://techpubs.sgi.com/library/manuals/4000/007-4660-002/pdf/007-4660-002.pdf |archive-date=2016-01-21 |url-status=dead}}</ref><ref>{{cite web |title=SGI® Altix® 3000 Servers and Superclusters |url=http://www.sgi.com/pdfs/3392.pdf |archive-url=https://web.archive.org/web/20060314165928/http://www.sgi.com/pdfs/3392.pdf |archive-date=2006-03-14 |url-status=dead}}</ref> SGI's second-generation SHUB 2.0 chipset supported up to 48 GB of [[DDR2 SDRAM|DDR2]] memory, 667 MT/s FSB, and could connect to I/O modules providing [[PCI Express]].<ref>{{cite web |title=SGI® Altix® 4700 Servers and Supercomputers |url=http://www.sgi.com/pdfs/3867.pdf |archive-url=https://web.archive.org/web/20051124042540/http://www.sgi.com/pdfs/3867.pdf |archive-date=2005-11-24 |url-status=dead}}</ref><ref>{{cite web |last1=Vogelsang |first1=Reiner |title=SGI® Altix™ Hardware Architecture |url=https://wwwuser.gwdg.de/~parallel/parallelrechner/altix_documentation/Altix_Hardware_revised_4.pdf |access-date=4 July 2023}}</ref> It supports only four local threads, so when having two dual-core CPUs per chipset, [[Hyper-Threading]] must be disabled.<ref>{{cite web |title=SGI® L1 and L2 Controller Software User's Guide |url=http://techpubs.sgi.com/library/manuals/3000/007-3938-006/pdf/007-3938-006.pdf |archive-url=https://web.archive.org/web/20151203105150/http://techpubs.sgi.com/library/manuals/3000/007-3938-006/pdf/007-3938-006.pdf |archive-date=2015-12-03 |url-status=dead}}</ref>