arrow-left

All pages
gitbookPowered by GitBook
1 of 1

Loading...

Instructions

This simplified tutorial use the Kestrel (HPC@NREL) as an example

Written by Cheng-Wei Lee (clee2 [at] mines [dot] edu)


The very first step is to log into Kestrelarrow-up-right. Since there is no external login nodes available yet, you need to use login the SSH gateway first via

circle-info

[user name] is your user account

-AY is for X11 forwarding and is optional

Once on the ssh gateway, you can log into Kestrel by

circle-info

there are multiple login nodes for Kestrel, e.g. kl1 and kl2

Once on the login nodes of Kestrel, we can load the following modules

circle-info

We only need anaconda3 to set up pylada. The rest are the libraries used to compile VASP and thus needed to run VASP

We need to use the specific mpi, which can be different from the default one

Once anaconda is loaded, we can now use conda to create a python virtual environment

circle-info

[name] is the user defined name for the virtual environment

python 3.9 has been working fine for pylada and pymatgen

Once the virtual environment is set up, we can activate it via

circle-info

This requires the module of anaconda3 being loaded

With the environment set up, we also need git to install pylada

Now with everything set up, we can follow the instruction on the

(We use pip to install pylada)

circle-info

One common error message is likely due to lack of compiler

Once the pylada is installed, we need to create a file name "ipython_config.py" with the path of

And it needs to have the following content

In order to submit jobs using pylada, we also need the following file at home directory

with the following content

circle-info

There are two places requiring modification

  1. [user name] is your account

  2. [path to virtual environment] is the path to your python environment. you can check the python by

ssh -AY [user name]@hpcsh.nrel.gov
conda info -e
pylada's github websitearrow-up-right
ssh [user name]@kl2.hpc.nrel.gov
source /nopt/nrel/apps/env.sh
module purge
module load anaconda3/2022.05
module load PrgEnv-intel/8.5.0
module swap cray-mpich cray-mpich-abi
module unload cray-libsci
module load intel-oneapi-compilers/2023.2.0
module load intel-oneapi-mpi/2021.10.0-intel 
module load intel-oneapi-mkl/2023.2.0-intel
conda create -n [name] python=3.9
conda activate [name]
module load git
pip install git+https://github.com/pylada/pylada-light
~/.ipython/profile_default/ipython_config.py 
c = get_config()
c.InteractiveShellApp.extensions = [ "pylada.ipython" ]
~/.pylada

vasp_has_nlep = False

################## QDEL definition ################

mpirun_exe = "srun --mpi=pmi2 -n {n} {program}"   # Kestrel requires --mpi=pmi2 for vasp

qdel_exe = "scancel"

qsub_exe = "sbatch"

################### QSTAT definition ##############
def ipython_qstat(self, arg):

  """ Prints jobs of current user. """
  from subprocess import Popen, PIPE
  from IPython.utils.text import SList
  # get user jobs ids
  jobs   = Popen(['squeue', '-u', '[user name]','--format','%.18i %.9P %j %.8u %.2t %.10M %.6D %R'], stdout=PIPE, encoding='utf-8').communicate()[0].split('\n')

  names  = [lines.strip().split()[2] for lines in jobs[1:-1]]
                                                  
  mpps   = [lines.strip().split()[0] for lines in jobs[1:-1]]
                                                  
  states = [lines.strip().split()[4] for lines in jobs[1:-1]]
                                                  
  ids    = [lines.strip().split()[0] for lines in jobs[1:-1]]

  return SList([ "{0:>10} {1:>4} {2:>3} -- {3}".format(id, mpp, state, name)   \
                 for id, mpp, state, name in zip(ids, mpps, states, names)])

##  return SList([ "{0}".format(name)   \
##                 for id, mpp, state, name in zip(ids, mpps, states, names)])	

##################### PBSSCRIPT #####################

pbs_string =  '''#!/bin/bash -x
#SBATCH --account={account}   
#SBATCH --nodes={nnodes}
#SBATCH --ntasks-per-node={ppn}
#SBATCH --export=ALL
#SBATCH --time={walltime}
#SBATCH --job-name={name}
###SBATCH --mem=176G.  # uncomment to requrest specific memory (usually not needed)
#SBATCH --partition={queue}
####SBATCH --qos=high # uncommented if high qos is needed (usually not needed)
###SBATCH -o out
###SBATCH -e err

# Go to the directoy from which our job was launched
cd {directory}

# Make sure the library are loaded properly for vasp
source /nopt/nrel/apps/env.sh
module purge
module load anaconda3/2022.05
module load PrgEnv-intel/8.5.0
module swap cray-mpich cray-mpich-abi
module unload cray-libsci
module load intel-oneapi-compilers/2023.2.0
module load intel-oneapi-mkl/2023.2.0
module load intel-oneapi-mpi/2021.10.0-intel 

source activate [path to virtual environment] 
export OMP_NUM_THREADS=1 #turns off multithreading, needed for VASP

{header}
python {scriptcommand}
{footer}
'''