NCAR / UCAR a research site with a long association with Cray supercomputers.

The National Centre for Atmospheric Research NCAR/UCAR in Colorado has a long association with Cray supercomputers. Starting in 1977 with the Cray-1 SN3 the site invested in some the best supercomputers available. More recently a Cray Petaflop class machines will be used to explore climate research.

In 2024 the site has commissioned Derecho a 19 Peta Flop Cray system from HPC. This system will replace Cheyenne a 5.5 Peta Flop system with 313 Tbyte of working memory.

After announcing the move of the venerable Cray-1 SN3 system from Mesa lab to the Wyoming supercomputer centre, on 7th January 2021 the site hosted a virtual tour with some of the folks involved in the long history of using and caring for the Cray systems at the site. <<Link to call recording when available.>>. SN3 will be sitting along side Derecho and visible from the visitor centre when it reopens.

Here are some further links related to NCAR/UCAR. The centre has excellent web and visitor information well worth a visit.

Derecho Commissioning
Derecho Information
Tom Engel Presentation: HPC at NCAR Past Present and Future. Since that date the centre has had “YellowStone” “Cheyenne” and soon to install “Derecho” systems.
NCAR Supercomputing History
Honoring the Cray1-A Interactive Page
NCAR: Cray-1A S/N3 history & All NCAR Supercomputing history
NCAR Virtual Visit Interactive Page
NCAR OpenSky Repository
Personal account, Paul Rotar: The Evolution of Supercomputing at NCAR
See also C90 Install photos over here.

Derecho hardware

323,712 processor cores	3rd Gen AMD EPYC™ 7763 Milan processors
2,488 CPU-only computation nodes	Dual-socket nodes, 64 cores per socket 256 GB DDR4 memory per node
82 GPU nodes	Single-socket nodes, 64 cores per socket 512 GB DDR4 memory per node 4 NVIDIA 1.41 GHz A100 Tensor Core GPUs per node 600 GB/s NVIDIA NVLink GPU interconnect
328 total A100 GPUs	40GB HBM2 memory per GPU 600 GB/s NVIDIA NVLink GPU interconnect
6 CPU login nodes	Dual-socket nodes with AMD EPYC™ 7763 Milan CPUs 64 cores per socket 512 GB DDR4-3200 memory
2 GPU development and testing nodes	Dual-socket nodes with AMD EPYC™ 7543 Milan CPUs 32 cores per socket 2 NVIDIA 1.41 GHz A100 Tensor Core GPUs per node 512 GB DDR4-3200 memory
692 TB total system memory	637 TB DDR4 memory on 2,488 CPU nodes 42 TB DDR4 memory on 82 GPU nodes 13 TB HBM2 memory on 82 GPU nodes
HPE Slingshot v11 high-speed interconnect	Dragonfly topology, 200 Gb/sec per port per direction 1.7-2.6 usec MPI latency CPU-only nodes – one Slingshot injection port GPU nodes – 4 Slingshot injection ports per node
~3.5 times Cheyenne computational capacity	Comparison based on the relative performance of CISL’s High Performance Computing Benchmarks run on each system.
> 3.5 times Cheyenne peak performance	19.87 peak petaflops (vs 5.34)

NCAR also hosted the single delivered Cray-3 system.

A few photos of systems at NCAR – Pictures and text from NCAR and other sources.

4196-2. NCAR accepts the first production model of the CRAY-1A supercomputer im 1977. The five-ton machine was lowered through the ceiling of a newly built underground computing center. Subsequent machines from Cray Research dominated NCAR computing into the 1990s.

NCAR accepted the first production model of the CRAY-1A supercomputer in 1977. The supercomputer weighed 5-1/2 tons and arrived in two refrigerated electronic vans. More than 30 construction workers, engineers, and helpers were needed to move it into the computer room. The CRAY-1A was particularly adapted to the needs of the scientific community, permitting advances in modeling climate and severe storms. It was decommissioned in 1989 after 12 years of service.

6951c-39, 1/27/1987, computer center with new XMP (display)

computer areas for display update, 5/2/1989, 7650c-2

The CRAY Y-MP8/864 that would become NCAR’s flagship computer for seven years was installed in the Computer Room on May 21, 1990. It was named shavano after a 14,000-foot peak in the Colorado Rockies, which in turn had been named for Chief Che-Wa-No, chief of the Tabeguache branch of the Ute Indians.The Y-MP had eight processors that could run independently or in parallel, and could achieve more than a gigaflop on an NCAR ocean climate model. The machine also had 64 megawords of directly addressable central memory and a 256-million-word Solid-state Storage Device (SSD) that functioned as a dedicated high-speed disk drive. It had a clock speed of 6 nanoseconds.

Productivity of NCAR’s Climate Simulation Laboratory (CSL) was dramatically enhanced in December 1996 with the addition of a CRAY C90 named antero. The C90 offered more than a factor of four increase in computing power over its predecessor (a CRAY Y-MP/8I).As the linchpin of the Climate System Laboratory (CSL), antero was dedicated to extensive climate simulations, providing 10,400 CPU hours per month for CSL users. The NCAR Climate Systems Model (CSM) averaged approximately 5.2 gigaflops on the C90 and could simulate five years per day of wall-clock time; less than one month was required to simulate a century.

Cheyenne – Cheyenne is a 5.34-petaflops HPC system that has 145,152 processor cores on 4,032 nodes. This supercomputer was installed at the NCAR-Wyoming Supercomputing Center in late 2016 and became operational in January 2017. It will be decommissioned in late 2022.

Lynx was acquired by NCAR as a test and evaluation platform, rather than a full production supercomputer like its counterpart, Bluefire. CISL determined that the investment in a test Cray XT5m was of strategic importance, and while the system was used by CISL system administrators and computational scientists, many NCAR users also used Lynx for production computing to offload work from Bluefire. Lynx was installed in the NCAR Mesa Laboratory’s computer facility and used for three and a half years. Despite being a test and evaluation platform, over its lifetime Lynx’s average utilization was 49.3%. It was decommissioned October 1, 2013, after the opening of the NCAR-Wyoming Supercomputing Center (NWSC) and CISL’s deployment of Yellowstone, the inaugural supercomputer at the NWSC, in production

CRAY Supercomputer formerly on display at the NCAR Mesa Lab in Boulder CO, moved to the NCAR / Wyoming Supercomputing Center (NWSC) near Cheyenne WY. Amazing to have this original machine in the same facility as the new HP (CRAY) system named Derecho.

Table of Cray system specifications — NCAR systems table ( the good bits ) : from HPC at NCAR Past Present and Future.

NCAR systems table ( the good bits ) : from HPC at NCAR Past Present and Future.

Entries for articles in Cray Channels about NCAR

Search:

Organisation	Topic	Section	Product	AuthorCredit	Title	Abstract

ECMWF, GFDL, NOAA, NCAR, LASL, UCS	Environment	Feature Article	Cray-1		CRAY-1 to participate in worldwide weather study	Sites that have Cray-1s, Printed circuit boards and power supply components are exposed as engineers Install the CRAY-1 at UCS picture
NCAR	Environment	User News	Cray-1		CCOPE	CCOPE project at NCAR deemed a success
NCAR	Cray Systems for customers	Corporate Register	Cray-1		NCAR to receive second Cray-1 System	A one-million word CRAY system w ill be installed at the NCAR facility in Boulder, Colorado early in 1983. The CRAY system will be NCAR’s second; back in 1977, NCAR installed the second CRAY-1 system ever produced by Cray Research.
NCAR	Environment	User News	Unspecified		NCAR’s CRAY takes on acid rain modeling
ECMWF, NCAR, CMC	Environment	Feature Article	Unspecified		CRAY finishes first in the weather forecasting race	Supercomputer power is of critical importance in weather research and forecasting- here’s why. We also highlight some projects using the CRAY at NCAR, ECMWF, and CMC. Large-scale computing in weather centers
NCAR	Environment	User News	Cray-1		El Nino under close scrutiny	Scientists working with a CRAY-1 at the National Center for Atmospheric Research (NCAR) have developed an accurate model of El Nino.
NCAR	Environment	User News	Unspecified		Not in Kansas anymore	tornado modeling
NCAR	Environment	User News	Unspecified		NCAR offers insights to petroleum researchers	GAMM Areospace challenge
NCAR	Cray Systems for customers	Corporate Register	Cray-XMP		More new customers for Cray computer systems	Cray Research announced the order of a CRAY X-MP/48 computer system with a 128-million-word SSD solid-state storage device by the UCAR installed at NCAR in Boulder CO.
NCAR	Environment	Feature Article	Unspecified		Supercomputing at NCAR	Computer models probe the puzzles of nature
NCAR	Graphics	About the cover	Unspecified		On the cover	Researchers use Cray systems for a universe of applications- from examining molecules to mapping global climate patterns.
NCAR	Cray Systems for customers	Corporate Register	Cray-YMP		Cray Research’s customers span the globe	University Corporation for Atmospheric Research has ordered a CRAY X-MP/18 system to be installed at the National Center for Atmospheric Research in Boulder Colorado will be used to convert users’ codes from the Cray operating system COS to Cray Research’s UNlCOS operating system, which is based on AT&T UNIX System V. The used system will replace NCAR’s CRAY-1A system , which has served NCAR for 11 years. NCAR will continue to apply its CRAY X-MP/48 system to climate and weather research.
NCAR	Environment	Feature Article	Unspecified	Robert Chervin, National Center for Atmospheric Research, Boulder, Colorado	Microtasking climate models at NCAR	By adapting global atmosphere and ocean models for multiprocessor execution NCAR researchers obtain more answers in less time.
NCAR	Cray Systems for customers	Corporate Register	Cray-YMP		Environmental organiations worldwide select Cray systems	The National Center for Atmospheric Research (NCAR) in Boulder, Colorado, has ordered an eight-processor CRAY Y-MP system,
NCAR	Cray Systems for customers	Applications update	Cray-YMP		Four first-time customers order entry level systems	National Center for Atmospheric Research (NCAR), Boulder, Colorado, has ordered a CRAY Y-MP supercomputer to be one of the few supercomputers in the world dedicated solely to climate research. NCAR w ill establish a numerical climate research laboratory with funds provided by the Model Evaluation Consortium for Climate Assessment (MECCA)
NCAR	Events	User News	Unspecified		Two oceanographers, one center win supercomputing awards
NCAR	Cray Systems for customers	Corporate Register	Cray-T3D		The National Center for Atmospheric Research (NCAR) has installed four Cray Research systems, for CSL	The National Center for Atmospheric Research (NCAR) has installed four Cray Research systems, including a 64-PE CRAY T3D system, a CRAY Y-MP8I system, and two CRAY EL92 deskside systems. The CRAY T3D and CRAY Y-MP8I systems will provide the foundation for the establishment of the Climate Simulation Laboratory (CSL) at NCAR
NCAR	Cray Systems for customers	Corporate Register	Cray-YMP		The National Center for Atmospheric Research (NCAR) has installed four Cray Research systems, for CSL	The National Center for Atmospheric Research (NCAR) has installed four Cray Research systems, including a 64-PE CRAY T3D system, a CRAY Y-MP8I system, and two CRAY EL92 deskside systems. The CRAY T3D and CRAY Y-MP8I systems will provide the foundation for the establishment of the Climate Simulation Laboratory (CSL) at NCAR
NCAR	Cray Systems for customers	Corporate Register	Cray-EL		The National Center for Atmospheric Research (NCAR) has installed four Cray Research systems, for CSL	The National Center for Atmospheric Research (NCAR) has installed four Cray Research systems, including a 64-PE CRAY T3D system, a CRAY Y-MP8I system, and two CRAY EL92 deskside systems. The CRAY T3D and CRAY Y-MP8I systems will provide the foundation for the establishment of the Climate Simulation Laboratory (CSL) at NCAR

Showing 1 to 19 of 19 entries (filtered from 1,408 total entries)

Derecho hardware

Entries for articles in Cray Channels about NCAR

Share this:

Like this: