Configuring and Running Simulations
Traditionally, PSCAD was used to run simulations of small power networks to study localized effects of electromagnetic transients. A single processor/core was sufficient in those times. Over time, with the advancement in power systems and the availability of increased computing power, in the form of number of cores in a computer, power systems engineers created sizable networks that surpassed the software’s capability of only using a single core for running the simulations.
PSCAD has evolved to become adept at parallel processing in the recent years. With features such as, Parallel Network Interface (PNI) and Parallel Multiple Run (PMR), PSCAD can now utilize the computing power of multiple cores that are at its disposal. It not only utilizes multiple cores on the computer where PSCAD is running, but also can utilize multiple-cores on remote computers connected via a Local Area Network (LAN).
The Cluster Launch System (CLS) is a utility used by PSCAD to launch simulation processes on remote computers. The CLS uses PsExec.exe, available from Windows Sysinternals, in order to achieve this.
The utilization of high-performance and parallel processing functionalities requires multiple processor cores – the number of which may exceed those available locally on a typical workstation. In such situations, users may either purchase a large and expensive, high-performance computer (ex. 64-core), or connect a collection of multi-core computers across a LAN. The latter option is referred to as a computational cluster. Running processes across multiple computers requires additional software to manage the simulation processes. Enter the CLS, a utility that can both launch and manage simulations on a computational cluster.
High-Level Representation of the Cluster Launch System
A user wanting to run 10 simulations on a computer (Host 1) comprising of 4 cores, can use the CLS to configure a computational cluster and run 10 simulations using several multi-core computers connected across a LAN. As shown in the figure above, Host 1 can be treated as the master, since PSCAD is running on this computer. PSCAD launches the CLS, which is then used to configure a computational cluster comprising of Host 1, Host 2 and Host 3. The CLS then launches all 10 simulations. The first 4 simulations are launched on the local computer and the remaining 6 simulations are distributed on Host 2 and Host 3.
There are some prerequisites that must be dealt with before the Cluster Launch System can be utilized. The following is the required setup process for the CLS to function.
PsExec.exe must exist on the computer where PSCAD is running. PsExec.exe is a third party utility and can be downloaded from the following link:
https://docs.microsoft.com/en-us/sysinternals/downloads/psexec
PsExec.exe comes as part of PsTools, which is part of the Windows Sysinternals suite of utilities. The downloaded *.zip file can be unzipped/extracted to a suitable location of the Master computer. The path to PsExec.exe can then be provided to the CLS utility.
The CLS can only be used if all the files corresponding to the simulation projects reside on a shared network drive. This includes the workspace file (*.pswx), project files (*.pscx) and all other files required by the simulation project. The CLS requires the shared network drive to be mapped on to a common drive letter. This drive letter must be the same on all the computers participating in running the simulations.
Example:
In the example above, the shared network location \\workspaces\public\PSCADProject is mapped on to drive letter Q: on a computer. The drive letter Q must be the same drive letter on all the computers in the cluster as shown in the figure below:
PsExec.exe needs to have access to the mapped networked drive on all participating remote computers in the cluster. This can be performed by running a command in the Windows Command Prompt, in the folder where PsExec.exe has been downloaded (on the Master computer). The command is as follows:
PsExec.exe -s \\<IPAddress> net use P: \\<Network>\<shared>\<folder> /USER:<domain>\<username> <password>
Examples:
PsExec.exe -s \\<10.xx.xx.35> net use Q: \\workspaces\public\PSCADProject /USER:mhi\joshsmith pwd123
PsExec.exe -s \\<10.xx.xx.52> net use Q: \\workspaces\public\PSCADProject /USER:mhi\johnsmith pwd123
PsExec.exe -s \\<10.xx.xx.65> net use Q: \\workspaces\public\PSCADProject /USER:mhi\johnsmith pwd123
PsExec.exe -s \\<10.xx.xx.78> net use Q: \\workspaces\public\PSCADProject /USER:mhi\johnsmith pwd123
To verify that the network drives mapped on remote computers are accessible to PsExec.exe (running from the Master computer), run the following command:
PsExec.exe -s \\<IPAddress> net use
Example:
PsExec.exe -s \\<10.xx.xx.35> net use
There must be an OK under status for the drive letter that was mapped on all remote computers participating in the cluster.
The CLS as launched as an external utility, via the Cluster Launch System button on the Tools tab of the PSCAD ribbon control bar:
The CLS must be launched with USER privileges and NOT with ADMIN privileges. This is because the network resources are not treated the same as all other resources between ADMIN and USER privileged environments.
The CLS may be launched in two modes: Configuration Mode or Run Mode.
The user may choose to launch the CLS simply to configure the cluster and the resources. See Launching above.
In this mode, the user will not be able to run the simulations.
To launch the CLS in Run Mode, the user must first enable some settings. Further, in run mode the CLS can be launched in two sub-modes: GUI Mode and Command Line Mode.
In the PSCAD Application Options dialog, set the Simulation launch method option in the Simulations category to Cluster launch utility.
Selecting this option instructs PSCAD to not launch the simulations locally, but to use the CLS instead.
In the PSCAD Application Options dialog, set the GUI mode display option in the Tools & Utilities category to Always (GUI Mode):
GUI Mode allows the user control over the cluster configuration and resource settings. It also shows any errors and messages in the GUI window and will always bring up the GUI of CLS when running the simulations.
Automatic (GUI or Command Line Mode): Selecting this option will let the CLS decide whether to bring up the CLS application GUI or run in command line. If all system settings are valid, the CLS may choose to run in command mode, transparent to the user. Whereas, if some parameter is missing or not configured, it will bring up the GUI so users can provide the missing information.
The CLS is primarily used for launching simulations on multiple, multi-core computers connected via a LAN. It operates on two files – a Simulation Task Description file and a Configuration file
A simulation task description file (with extension *.clsx) is an XML-based file, generated by PSCAD when a simulation set is run and the application option Simulation Startup Method is set to Use Cluster Launch System.
The *.clsx file is created within the same folder, in which workspace file (*.pswx) is saved. The *.clsx filename corresponds to the workspace file. For example, if the workspace name is Examples_v1.pswx then the simulation task description file name will be named Examples_v1.clsx.
The path to the *.clsx file is supplied by PSCAD as an argument to the CLS utility and does not need to be selected by the user, as shown in the figure below:
An example of “.clsx” file is given below:
A configuration file (with extension *.cfg) is an XML-based file that describes the cluster configuration and resource settings. This file is located in the AppData folder of the local computer:
C:\Users\johnsmith\AppData\Local\Manitoba HVDC Research Centre\CLSData\config.cfg
This file is generated by the CLS, after the user has entered the required information. An example of the *.cfg file is given below:
The CLS requires the user to input cluster information (ie. IP Address and Core Count) and resources settings the first time it is launched. When the user enters this information, it is saved in the configuration (*.cfg) file. This information is then retrieved and used/shown in the GUI when CLS is launched thereafter.
A Parallel Network Interface (PNI) example is shown in the figure below:
When we run the simulation set named default, the CLS is launched as shown below:
Notice that cluster information (IP Address, Core Count) and Resource Settings (Shared Storage and Utility) are empty. This is because this is the CLS' first launch and there was no configuration (*.cfg) file, from which the settings could be loaded. Also, the simulation task description (*.clsx) file is already present, as we have launched CLS in run mode.
On first use, the user may be required to fill in some information, such as IP Address and Core Count of the computers taking part in the execution environment, shared network drive information and PsExec.exe location.
Fill in the IP Address of the computers (both local and remote) that are to take part in the computational cluster. Under Core Count enter the number of cores to be utilized on that particular computer.
For Example:
Specify the shared storage network path information and the PsExec.exe path as shown below:
Cluster configuration information and Resource Settings get saved in the configuration (*.cfg) file.
C:\Users\johnsmith\AppData\Local\Manitoba HVDC Research Centre\CLSData\config.cfg
The user may save the entered information by pressing the Run or Close button in the CLS utility. If changes are detected in the form, the user will be prompted to save the information.
When the CLS is launched, the application looks for the configuration (*.cfg) file and loads the configuration and settings in the form.
Once the information is filled in the forms, press the Run button to launch the simulations.
After the simulations have finished running, the user must Close the CLS utility to exit back to PSCAD.