| LS-DYNA |
|---|
| Description: Suite of programs for transient dynamic finite element program |
| SHARCNET Package information: see LS-DYNA software page in web portal |
| Full list of SHARCNET supported software |
Contents
Introduction
Before a research group can use LSDYNA on sharcnet, a license must be purchased directly from LSTC for the sharcnet license server. Alternately, if a research group resides at a institution that has sharcnet computing hardware (mac, uwo, guelph, waterloo) it maybe possible to use a pre-existing site license hosted on an accessible institutional license server.
Usage
These usage instructions apply to the module based installation of lsdyna found under directory /opt/sharcnet/lsdyna on licensed clusters shown in the above Availability Table. To access and use this software you must be a member of the lsdyna group. Users who belong to a licensed department will be given a 5 digit port number. The value should then be inserted into the appropriate departmental export statement before loading the module file as follows:
o UWO Civil and Environmental Engineering Dept
export LSTC_LICENSE_SERVER='PortNumber'@license1.uwo.sharcnet
o McGill Mechanical Engineering Department
export LSTC_LICENSE_SERVER='PortNumber'@license2.uwo.sharcnet
o UW Mechanical and Mechatronics Engineering Dept
export LSTC_LICENSE_SERVER='PortNumber'@license3.uwo.sharcnet
Module Selection
[roberpj@orc-login1:~] module avail lsdyna -------------------------------------- /opt/sharcnet/modules --------------------------------------- lsdyna/ls971mpp6.0.0 lsdyna/ls971mpp6.1.1 lsdyna/ls971smp6.1.0 lsdyna/ls980mppB1 lsdyna/ls971mpp6.1.0 lsdyna/ls971smp6.0.0 lsdyna/ls971smp6.1.1 lsdyna/ls980smpB1
o For serial or threaded jobs, using ls971 do:
module load lsdyna/ls971smp6.1.1
o For mpi jobs, using ls971 do:
module unload intel openmpi lsdyna module load intel/11.1.069 openmpi/intel/1.5.4 lsdyna/ls971mpp6.1.1
o For serial or threaded jobs, using version ls980 (and noting that the restart broken) do the following:
module load lsdyna/ls980smpB1
o For mpi jobs, using version ls980 (and noting that the restart broken) do the following:
module unload intel openmpi lsdyna module load intel/12.1.3 openmpi/intel/1.4.5 lsdyna/ls980mppB1
Job Submission
To run the single or double precision solvers specify lsdyna_s or lsdyna_d respectively:
o SUBMIT 1CPU SERIAL JOB
sqsub -r 7d -q serial -o ofile.%J --mpp=2G lsdyna_d i=airbag.deploy.k ncpu=1
o SUBMIT 4CPU SMP JOB
sqsub -r 7d -q threaded -n 4 -o ofile.%J --mpp=2G lsdyna_d i=airbag.deploy.k ncpu=4
o SUBMIT 4CPU MPI JOB
sqsub -r 7d -q mpi -n 4 -o ofile.%J --mpp=1G lsdyna_d i=airbag.deploy.k
Example Job
STEP1) The following shows a sample sqsub jobs submission for the airbag example to the mpi queue shown in this link http://www.dynaexamples.com/examples-manual/misc/airbag where you I suggest first changing endtim to 3.000E-00 in airbag.deploy.k to the job runs long enough to perform all steps in this section:
cp -a /opt/sharcnet/lsdyna/ls971smp6.1.1/examples /work/$USER/test-lsdyna cd /work/$USER/test-lsdyna/misc/airbag gunzip airbag.deploy.k.gz module unload intel openmpi lsdyna module load intel/11.1.069 openmpi/intel/1.5.4 lsdyna/ls971mpp6.1.1 export LSTC_LICENSE_SERVER=#####@license#.uwo.sharcnet cp airbag.deploy.k airbag.deploy.restart.k sqsub -r 1h -q mpi --mpp=4G -n 8 -o ofile.%J lsdyna_d i=airbag.deploy.k
STEP2) With the job still running, use the echo command as follows to create a file called "D3KIL" that will trigger generation of restart files at which point the file D3KIL itself will be erased. Do this once a day if data loss is critical to you OR once or twice just before the sqsub -r time limit is reached. Further information can be found here http://www.dynasupport.com/tutorial/ls-dyna-users-guide/sense-switch-control:
echo "sw3" > D3KIL sqkill job#
STEP3) Before the job can be restarted, the following two lines must be added to the airbag.deploy.restart.k file:
*CHANGE_CURVE_DEFINITION
1
STEP4) Now resubmit the job as follows using "r=" to specify the restart file:
sqsub -r 1h -q mpi --mpp=4G -n 8 -o ofile.%J lsdyna_d i=airbag.deploy.restart.k r=d3dump01
General Notes
Using lstc_qsun
The available options for the lstc_qrun are given by:
[roberpj@hnd19:~] lstc_qrun -help o PRINT HELP AND QUIT: lstc_qrun -help o PRINT VERSION INFO AND QUIT: lstc_qrun -v o PRINT SERVER LIST: lstc_qrun -i o PRINT JOB INFO: lstc_qrun [-s server] o PRINT LICENSE INFORMATION: lstc_qrun [-s server] -r o PRINT VERBOSE LICENSE INFORMATION: lstc_qrun [-s server] -R
To check the status of your job in the queue use the sqjobs command. On x86_64 systems you can check whether any of your jobs are queued on the license server use lstc_qrun. Its possible that sqsub will start your job but it will sit idle until enough license are available on the license server and the lstc_qrun command will reveal this:
[roberpj@hnd19:~] lstc_qrun Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable Running Programs User Host Program Started # procs ----------------------------------------------------------------------------- dgierczy 20277@saw61.saw.sharcn LS-DYNA_971 Mon Apr 8 17:44 1 dgierczy 8570@saw32.saw.sharcn LS-DYNA_971 Mon Apr 8 17:44 1 dscronin 25486@hnd6 MPPDYNA_971 Tue Apr 9 20:19 6 dscronin 14897@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6 dscronin 14971@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6 dscronin 15046@hnd18 MPPDYNA_971 Tue Apr 9 21:48 6 dscronin 31237@hnd16 MPPDYNA_971 Tue Apr 9 21:53 6 dscronin 31313@hnd16 MPPDYNA_971 Tue Apr 9 21:54 6 dscronin 6396@hnd15 MPPDYNA_971 Tue Apr 9 21:54 6 csharson 28890@saw175.saw.sharc MPPDYNA_971 Wed Apr 10 16:48 6 roberpj 11257
At the time of this writing UWO is running on a 4cpu demo license with details:
[roberpj@hnd19:~] lstc_qrun -R Defaulting to server 1 specified by LSTC_LICENSE_SERVER variable **** LICENSE INFORMATION **** PROGRAM EXPIRATION CPUS USED FREE MAX | QUEUE ---------------- ---------- ----- ------ ------ | ----- LS-DYNA_971 12/31/2013 - 966 1024 | 0 dgierczy 20277@saw61.saw.sharcnet 1 dgierczy 8570@saw32.saw.sharcnet 1 MPPDYNA_971 12/31/2013 - 966 1024 | 0 dscronin 25486@hnd6 6 dscronin 14897@hnd18 6 dscronin 14971@hnd18 6 dscronin 15046@hnd18 6 dscronin 31237@hnd16 6 dscronin 31313@hnd16 6 dscronin 6396@hnd15 6 csharson 28890@saw175.saw.sharcnet 6 roberpj 11257@hnd15 8 LICENSE GROUP 58 966 1024 | 0 PROGRAM EXPIRATION CPUS USED FREE MAX | QUEUE ---------------- ---------- ----- ------ ------ | ----- LS-OPT 12/31/2013 0 1024 1024 | 0 LICENSE GROUP 0 1024 1024 | 0
Legacy Instructions
The following binaries remain available on saw and orca for backward compatibility testing:
[roberpj@orc-login2:~] cd /opt/sharcnet/local/lsdyna [roberpj@orc129:/opt/sharcnet/local/lsdyna] ls ls971* ls971_d_R3_1 ls971_d_R4_2_1 ls971_s_R3_1 ls971_s_R4_2_1 ls971_s_R5_1_1 ls971_d_R4_2_0 ls971_d_R5_0 ls971_s_R4_2_0 ls971_s_R5_0
There are currently no sharcnet modules for these versions, hence jobs should be submitted as follows:
module load lsdyna export PATH=/opt/sharcnet/local/lsdyna:$PATH export LSTC_LICENSE_SERVER=XXXXX@license3.uwo.sharcnet cp /opt/sharcnet/local/lsdyna/examples/airbag.deploy.k airbag.deploy.k SERIAL JOB: sqsub -q serial -r 1d -o ofile.%J ls971_d_R4_2_1 i=airbag.deploy.k THREADED JOB: sqsub -q threaded -n 4 -r 1d -o ofile.%J ls971_d_R4_2_1 ncpu=4 para=2 i=airbag.deploy.k
Note! the ls971_s_R3_1 and ls971_d_R3_1 binaries do not work, a fix is being looked for.
References
o LSTC LS-DYNA Homepage
http://www.lstc.com/lsdyna.htm
o LS-DYNA and d3VIEW Blog
http://blog2.d3view.com/a-few-words-on-memory-settings-in-ls-dyna/
o Convert Words to GB (500000000w=3.73gb,800000000w=5.96gb )
http://deviceanalytics.com/memcalc.php
o LS-DYNA Support Environment variables http://www.dynasupport.com/howtos/general/environment-variables