Academic (UB-HPC) Compute Cluster Hardware Specs

CCR core 10G Arista networking equipment

The academic compute cluster available to all faculty at UB is comprised of various Linux "nodes" (a.k.a. servers) with differing hardware specs and manufactured by several different vendors.  The hardware is similar enough that when networked together, users can run complex problems across many nodes to complete their problems faster.  This is known as a "Beowulf cluster." 

 

Front-end for SLURM Partitions

  • Hostname = rush.ccr.buffalo.edu (front end for logins, job submission and interactive use)
  • Vendor = DELL
  • Architecture = INTEL E7
  • Number of Processor Cores = 32
  • Processor Description:
    • 32x2.13GHz Intel Xeon E7-4830 Processor Cores
    • Main memory size: 256GB
    • Instruction cache size: 24576 Kbytes
    • Data cache size: 24576 Kbytes
    • Secondary unified instruction/data cache size: 8192 Kbytes
  • Operating System: Linux (CentOS release 6.4, 2.6.32 Kernel)

(interactive use, debugging, cpu time limit of 30m)

Fun Fact!

In the history of the English language, Beowulf is the earliest surviving epic poem written in English. It is a story about a hero with the strength of many men who defeated a fearsome monster called Grendel.  In computing, a Beowulf class cluster computer is a multicomputer architecture used for parallel computations, i.e., it uses many computers together so that it has the brute force to defeat fearsome number crunching problems.

Production Machine - DELL

(accessible only through the batch system)

  • Number of nodes = 372
  • Primary DELL 12 core Compute Nodes
  • Vendor = DELL
  • Architecture = DELL E5645
  • Number of Processor Cores = 12
  • Processor Description:
    • 12x2.40GHz Intel Xeon E5645 Processor Cores
    • Main memory size: 48 GB
    • Instruction cache size: 24576 Kbytes
    • Data cache size: 24576 Kbytes
    • Secondary unified instruction/data cache size: 8 MBytes
  • Operating System: Linux (RedHat Enterprise Linux 6.1 2.6.32 Kernel)
  • InfiniBand Q-Logic InfiniPath_QLE7340 Network Card
    • QDR InfiniBand 40Gb/s
  • Local scratch is approximately 884GB

 

Production Machine - DELL

(accessible only through the batch system)

  • Number of nodes = 8
  • Primary DELL 32 core Compute Nodes
  • Vendor = DELL
  • Architecture = DELL E7-4830
  • Number of Processor Cores = 32
  • Processor Description:
    • 32x2.13GHz Intel Xeon CPU E7-4830 Processor Cores
    • Main memory size: 256 GB
    • Instruction cache size: 24576 Kbytes
    • Data cache size: 24576 Kbytes
    • Secondary unified instruction/data cache size: 8 MBytes
  • Operating System: Linux (RedHat Enterprise Linux 6.1 2.6.32 Kernel)
  • InfiniBand Q-Logic InfiniPath_QLE7340 Network Card
    • QDR InfiniBand 40Gb/s
  • Local scratch is approximately 3.1TB

 

Production Machine - IBM

(accessible only through the batch system)

  • Number of nodes = 8
  • Primary IBM 32 core Compute Nodes
  • Vendor = IBM
  • Architecture = IBM 6132 HE
  • Number of Processor Cores = 32
  • Processor Description:
    • 32x2.20GHz AMD Opteron 6132 HE Processor Cores
    • Main memory size: 256 GB
    • Instruction cache size: 24576 Kbytes
    • Data cache size: 24576 Kbytes
    • Secondary unified instruction/data cache size: 8 MBytes
  • Operating System: Linux (RedHat Enterprise Linux 6.1 2.6.32 Kernel)
  • InfiniBand Q-Logic InfiniPath_QLE7340 Network Card
    • QDR InfiniBand 40Gb/s
  • Local scratch is approximately 3.1TB

 

Production Machine - DELL

(accessible only through the batch system)

  • Number of nodes = 1
  • Primary DELL 32 core Compute Nodes
  • Vendor = DELL
  • Architecture = DELL E7-4830
  • Number of Processor Cores = 32
  • Processor Description:
    • 32x2.13GHz Intel Xeon CPU E7-4830 Processor Cores
    • Main memory size: 512 GB
    • Instruction cache size: 24576 Kbytes
    • Data cache size: 24576 Kbytes
    • Secondary unified instruction/data cache size: 8 MBytes
  • Operating System: Linux (RedHat Enterprise Linux 6.1 2.6.32 Kernel)
  • InfiniBand Q-Logic InfiniPath_QLE7340 Network Card
    • QDR InfiniBand 40Gb/s
  • Local scratch is approximately 3.1TB

 

Production Machine - IBM

(accessible only through the batch system)

  • Number of nodes = 128
  • iDataPlex - dual quad-core Compute Nodes
  • Vendor = IBM
  • Number of Processor Cores = 8
  • Processor Description:
    • 8x2.27GHz Intel Xeon L5520 "Westmere" (Nehalem-EP) Processor Cores
    • Main memory size: 24576 Mbytes
    • Instruction cache size: 128 Kbytes
    • Data cache size: 128 Kbytes
    • Secondary unified instruction/data cache size: 12 MBytes
  • Operating System: Linux (RedHat Enterprise Linux 6.1, 2.6.32 Kernel)
  • InfiniBand Mellanox Technologies MT26428 Network Card
    • QDR InfiniBand 40Gb/s
  • Local scratch is approximately 268GB

 

Production Machine - DELL

(accessible only through the batch system)

  • Number of nodes = 128
  • PowerEdge C6100 - dual quad-core Compute Nodes
  • Vendor = DELL
  • Number of Processor Cores = 8
  • Processor Description:
    • 8x2.13GHz Intel Xeon L5630 "Westmere" (Nehalem-EP) Processor Cores
    • Main memory size: 24576 Mbytes
    • Instruction cache size: 128 Kbytes
    • Data cache size: 128 Kbytes
    • Secondary unified instruction/data cache size: 12 MBytes
  • Operating System: Linux (RedHat Enterprise Linux 6.1, 2.6.32 Kernel)
  • InfiniBand Mellanox Technologies MT26428 Network Card
    • QDR InfiniBand 40Gb/s
  • Local scratch is approximately 268GB

 

Production Machine - DELL

(accessible only through the SLURM batch system)

  • Number of nodes = 32
  •  Dual 8-core Compute Nodes
  • Vendor = Dell
  • Architecture = PowerEdge Server
  • Number of Processor Cores = 16
  • Processor Description:
    • 16x2.20GHz Intel E5-2660 "Sandy Bridge" Xeon Processor Cores
    • Main memory size: 128 GB
    • Instruction cache size: 128 Kbytes
    • Data cache size: 128 Kbytes
    • Secondary unified instruction/data cache size: 20 Mbytes
  • InfiniBand Mellanox Technologies MT26428 Network Card
    • QDR InfiniBand 40Gb/s
  • Local scratch is approximately 770 GB
  • Operating System: Linux (CentOS release 6.4, 2.6.32 Kernel)

 

DELL GPU Cluster

(accessible only through the batch system, gpu queue)

  • Number of nodes = 32
    • PowerEdge C6100 - dual quad-core Compute Nodes
    • Vendor = DELL
    • Number of Processor Cores = 12
    • Processor Description:
      • 12x2.66GHz Intel Xeon X5650 "Westmere" (Nehalem-EP) Processor Cores
      • Main memory size: 48GB
      • Instruction cache size: 128 Kbytes
      • Data cache size: 128 Kbytes
      • Secondary unified instruction/data cache size: 12 MBytes
    • Local Hard Drives: 2x500GB SATA (/scratch), 1x100GB SSD (/ss_scratch)
    • Local scratch is approximately 0.9TB total
    • Two Nvidia M2050 "Fermi" Tesla GPUs (3GB memory per card)
  • Number of nodes = 1
    • PowerEdge R910 - quad socket, oct-core Compute Node
    • Vendor = DELL
    • Number of Processor Cores = 32
    • Processor Description:
      • 32x2.0GHz Intel Xeon X7550 "Beckton" (Nehalem-EX) Processor Cores
      • Main memory size: 256GB
      • Instruction cache size: 128 Kbytes
      • Data cache size: 128 Kbytes
      • Secondary unified instruction/data cache size: 18 MBytes
      • Local Hard Drives: 2x500GB SATA (/scratch), 14x100GB SSD (/ss_scratch)
      • Local scratch is approximately 1.9TB total
  • QDR InfiniBand 40Gb/s
  • Operating System: Linux (RedHat Enterprise Linux 5.5, 2.6.18 Kernel)
  • InfiniBand Mellanox Technologies MT26428 Network Card

 

Disk Layout

  • /home, User $HOME directories, NFS mounted from the CCR SAN to the compute nodes and front-ends.
  • /scratch, Primary high-performance scratch space, located on each compute node (see above for what is available on each /scratch, as it varies for each type of node).
    • Accessible through SLURM, which will automatically create a unique scratch directory in /scratch for each new batch job 
    • All scratch space will be scrubbed automatically at the end of each batch job. Files that need to be stored long term should be kept elsewhere.
  • /gpfs/scratch, globally accessible high-performance parallel scratch space for staging/preserving data between runs