Preface

Before using the Theoretical high-performance computing (HPC) cluster, you will need to obtain access by filling out the Theoretical Cluster Request Form.








News

Information About the Cluster

A. Hardware

The HPC Cluster servers consist of few head nodes and many compute nodes ("servers"). There are queuing systems with access to separate "partitions" for portions of hardware that contain different hardware generations.

Our tc nodes (in the "TC Cluster" partition) each has 12 CPU cores of 2.8 GHz, and 48 GB of RAM. Our px nodes ("PX Cluster" partition) each has 16 CPU cores of 2.9 GHz, and 64 GB of RAM. Our xl nodes (in the "XL Cluster" partition) contain four type hardware configurations. The type A of xl nodes each has 16 CPU cores of 2.6 GHz, and 64 GB of RAM. The type B of xl nodes each has 24 CPU cores of 2.5 GHz, and 256 GB of RAM. The type C of xl nodes each has 16 CPU cores of 2.6 GHz, and 256GB of RAM. The type D of xl nodes each has 16 CPU cores of 2.6 GHz, and 512 GB of RAM. All users log in at each different head node, and all user files on the shared file system (glusterfs and lustre) are accessible on all nodes. Additionally, all px and xl nodes are tightly networked (56 Gbit/s Infiniband) and all rest of nodes are tightly networked (1 Gbit/s Ethernet) so in the same partition they can work together as a single "supercomputer", depending on the number of CPUs you specify.

CPU Cluster
Cluster Cores Nodes CPU per node Memory per node Network Year
tc 612 50 X5660 @ 2.80GHz x 2 (12 Cores x 22),
X5670 @ 2.93GHz x 2 (12 Cores x 28)
48GB 1Gb Ethernet 2010
px 432 27 E5-2690 @ 2.90GHz x 2 (16 Cores) 64GB 1Gb Ethernet / 56Gb IB FDR 2012, 2013
xl 1,664 103 E5-2640 v3 @ 2.60GHz x 2 (16 Cores x 99),
E5-2680 v3 @ 2.50GHz x 2 (24 Cores x 4)
64GB,
256GB,
512GB
10Gb Ethernet / 56Gb IB FDR 2015, 2016


GPU Cluster
Hostname Node CPU per node GPU per node Memory Network
gp8~11 4 CPU E5-2698 v4 @ 2.20GHz x 2 (40 Cores) NVIDIA Tesla P100-SXM2 (3,584 CUDA Cores, 16GB) x 4 256GB 10Gb Ethernet

B. Logging In

In order to login head nodes, you need to login the gate server (gate.tiara.sinica.edu.tw), first. The Academia Sinica Institute of Astronomy and Astrophysics (ASIAA) networks are allowed to login, directly. When you are outside of the office, you could connect to the gate server via VPN or visit our internal page and add your current IP address into the gate's white list.

C. Data Storage

Data space (/tiara/ara/data) in the HPC file system is not backed-up and should be treated as temporary by users. Only files necessary for actively-running jobs should be kept on the file system, and files should be removed from the cluster when jobs complete. A copy of any essential files should be kept in an alternate, non-TIARA storage location.

Each TIARA user is initially allocated 500GB of data storage space in their home directory (/tiara/home/username/), though we can increase data quotas upon email request to sysman@tiara.sinica.edu.tw with a description of data space needing for concurrent, active work. Each non-TIARA user is initially allocated 100 GB of data storage space in their home directory.

TIARA System Staff reserve the right to remove any significant amounts of data on the HPC Cluster in our efforts to maintain filesystem performance for all users, though we will always first ask users to remove excess data and minimize file counts before taking additional action.

Scratch space of is available on each execute node in /scratch/data and is automatically cleaned out upon 45 days.

more details please reference Storage Usage Policy.

D. Partition Configuration and Job Scheduling

The job scheduler on the HPC Cluster is using the PBS Pro. You can read more about submitting jobs to the queue on the PBS User Guide, but we have provided a simple guide in our wiki page for getting started.

The OC Cluster partition is available to all users. This partition is best for debugging or testing your jobs. If you are not familiar with our cluster or your codes are under development , you are suggested to run in this partition.

The TC Cluster partition is available to all users. This partition is good for running small, medium or serial jobs.

The PX Cluster partition is available to all users. This partition is best for running production jobs. The large and medium jobs are welcome in this partition.

The XL Cluster partition is available to all TIARA researchers. This partition is best for running production and large jobs.