🖥 Start and access AWS submission box

Spin up an Ubuntu submission box if not already running. To do this, log onto AWS Console and start the EC2 instance.

Update IP address in .ssh/config file. To do this, open a terminal and type the command below. This will open your config file where you can change the IP to the IP4 assigned to the AWS EC2 instance (see AWS Console for this):

SSH into the box. In the terminal, SSH into your box. Typically we name these instances "staging", so usually the command is:

🧱 Setup

Now you should be logged onto the AWS submission box. If you haven't yet, set up your directory structure.

🗂 Create the directory structure (ONCE PER USER)

Type the following commands:

Have your Github ssh key passphrase handy so you can paste it when prompted (possibly multiple times) with the git pull command. Alternatively, you can add your github key to your batch box so you don't have to enter your token 6 times per day.

🚀 Run inference using AWS (do everytime)

🛳 Initiate the Docker

Start up and log into the docker container, and run setup scripts to setup the environment. This setup code links the docker directories to the existing directories on your box. As this is the case, you should not run job submission simultaneously using this setup, as one job submission might modify the data for another job submission.

Setup environment

To set up the environment for your run, run the following commands. These are specific to your run, i.e., change VALIDATION_DATE, FLEPI_RUN_INDEX and RESUME_LOCATION as required. If submitting multiple jobs, it is recommended to split jobs between 2 queues: Compartment-JQ-1588569569 and Compartment-JQ-1588569574.

NOTE: If you are not running a resume run, DO NOT export the environmental variable RESUME_LOCATION.

Additionally, if you want to profile how the model is using your memory resources during the run, run the following commands

Then prepare the pipeline directory (if you have already done that and the pipeline hasn't been updated (git pull says it's up to date). You need to set $PROJECT_PATH to your data folder. For a COVID-19 run, do:

for Flu do:

Now for any type of run:

For now, just in case: update the arrow package from 8.0.0 in the docker to 11.0.3 ;

Now flepiMoP is ready 🎉 ;

Do some clean-up before your run. The fast way is to restore the $PROJECT_PATH git repository to its blank states (⚠️ removes everything that does not come from git):

Then run the preparatory data building scripts and you are good

Now you may want to test that it works :

If this fails, you may want to investigate this error. In case this succeeds, then you can proceed by first deleting the model_output:

Launch your inference batch job on AWS

Assuming that the initial test simulation finishes successfully, you will now enter credentials and submit your job onto AWS batch. Enter the following command into the terminal:

You will be prompted to enter the following items. These can be found in a file you received from Shaun called new_user_credentials.csv.

• Access key ID when prompted

• Secret access key when prompted

• Default region name: us-west-2

Now you're fully set to go 🎉

To launch the whole inference batch job, type the following command:

This command infers everything from you environment variables, if there is a resume or not, what is the run_id, etc., and the default is to carry seeding if it is a resume (see below for alternative options).

If you'd like to have more control, you can specify the arguments manually:

We allow for a number of different jobs, with different setups, e.g., you may not want to carry seeding. Some examples of appropriate setups are given below. No modification of these code chunks should be required ;

Document the submission

After the job is successfully submitted, you will now be in a new branch of the data repo. Commit the ground truth data files to the branch on github and then return to the main branch:

Send the submission information to slack so we can identify the job later. Example output:

These details cover how to install and initialize flepiMoP on an HPC environment and submit a job with slurm.

For getting access to one of the supported HPC environments please refer to the following documentation before continuing:

• for UNC users, or

• for JHU users.

External users will need to consult with their PI contact at the respective institution.

Installing flepiMoP

This task needs to be ran once to do the initial install of flepiMoP.

Download and run the the appropriate installation script with the following command:

Substituting <cluster-name> with either rockfish or longleaf. This script will install flepiMoP to the correct locations on the cluster. Once the installation is done the conda environment can be activated and the script can be removed with:

Updating flepiMoP

Updating flepiMoP is designed to work just the same as installing flepiMoP. First change directory to your flepiMoP installation and then make sure that your clone of the flepiMoP repository is set to the branch you are working with (if doing development or operations work) and then run the flepimop-install-<cluster-name> script, substituting <cluster-name> with either rockfish or longleaf.

Initialize The Created flepiMoP Environment

These steps to initialize the environment need to run on a per run or as needed basis.

Change directory to where a full clone of the flepiMoP repository was placed (it will state the location in the output of the script above). And then run the hpc_init script, substituting <cluster-name> with either rockfish or longleaf. This script will assume the same defaults as the script before for where the flepiMoP clone is and the name of the conda environment. This script will also ask about the path to your flepiMoP installation and project directory. It will also ask if you would like to set a default configuration file, if you plan to use the flepimop batch-calibrate command below we recommend pressing enter to skip setting this environment variable. If this is your first time initializing flepiMoP it might be helpful to use configs out of flepiMoP/examples/tutorials directory as a test.

Upon completing this script it will output a sample set of commands to run to quickly test if the installation/initialization has gone okay.

Submitting A Batch Inference Job To Slurm

The main entry point for submitting batch inference jobs is the flepimop batch-calibrate action. This CLI tool will let you submit a job to slurm once logged into a cluster. For details on the available options please refer to flepimop batch-calibrate --help. As a quick example let's submit an R inference and EMCEE inference job. For the R inference run execute the following once logged into either longleaf or rockfish:

This command will produce a large amount of output, due to -vvv. If you want to try the command without actually submitting the job you can pass the --dry-run option. This command will submit a job to calibrate the sample 2 population configuration which uses R inference. The R inference supports array jobs so each chain will be run on an individual node with 1 CPU and 1GB of memory a piece. Additionally the extra option allows you to provide additional info to the batch system, in this case what partition to submit the jobs to but email is also supported with slurm for notifications. After running this command you should notice the following output:

• config_sample_2pop-YYYYMMDDTHHMMSS.yml: This file contains the compiled config that is actually submitted for inference,

• manifest.json: This file contains a description of the submitted job with the command used, the job name, and flepiMoP and project git commit hashes,

For operational runs these files should be committed to the checked out branch for archival/reproducibility reasons. Since this is just a test you can safely remove these files after inspecting them.

Now, let's submit an EMCEE inference job with the same tool. Importantly, the options we'll use won't change much because flepimop batch-calibrate is designed to provide a unified implementation independent interface.

One notable difference is, unlike R inference, EMCEE inference only supports running on 1 node so resources for this command are adjusted accordingly:

• Swapping 4 nodes with 1 cpu each to 1 node with 4 cpus, and

• Doubling the memory usage from 4 nodes with 1GB each for 4GB total to 1 node with 8GB for 8GB total.

The extra increase in memory is to run a configuration that is slightly more resource intense than the previous example. This command will also produce a similar set of record keeping files like before that you can safely remove after inspecting.

Estimating Required Resources For A Batch Inference Job

When inspecting the output of flepimop batch-calibrate --help you may have noticed several options named --estimate-*. While not required for the smaller jobs above this tool has the ability to estimate the required resources to run a larger batch estimation job. The tool does this by running smaller jobs and then projecting the required resources for a large job from those smaller jobs. To use this feature provide the --estimate flag, a job size of the targeted job, resources for test jobs, and the following estimation settings:

• --estimate-runs: The number of smaller jobs to run to estimate the required resources from,

• --estimate-interval: The size of the prediction interval to use for estimating the resource/time limit upper bounds,

• --estimate-vary: The job size elements to vary when generating smaller jobs,

Effectively using these options requires some knowledge of the underlying inference method. Sticking with the simple usa state level example above try submitting the following command (after cleaning up the output from the previous example):

In short, this command will submit 6 test jobs that will vary simulations and measure time and memory. The number of simulations will be used to project the required resources. The test jobs will range from 1/5 to 1/10 of the target job size. This command will take a bit to run because it needs to wait on these test jobs to finish running before it can do the analysis, so you can check on the progress by checking the output of the simple_usa_statelevel_estimation.log file.

Once this command finishes running you should notice a file called USA_influpaint_resources.json. This JSON file contains the estimated resources required to run the target job. You can submit the target job with the estimated resources by using the same command as before without the --estimate-* options and using the --from-estimate option to pull the information from the outputted file:

Saving Model Outputs On Batch Inference Job Finish

For production runs it is particularly helpful to save the calibration results after a successful run to long term storage for safe keeping. To accomplish this flepimop batch-calibrate can chain a call to flepimop sync after a successful run via the --sync-protocol option. For more details on the flepimop sync command in general please refer to the guide.

For a quick demonstration of how to use this option start with the config_sample_2pop_inference.yml configuration file and add the following section:

Where /path/to/an/example-folder and s3://my-bucket/and-sub-bucket are place holders for paths to your desired location. Importantly, note that there is no trailing slash on the model_output directory name. This will cause flepimop sync to sync the model_output directory itself and not just it's contents. You can also apply additional filters to the sync protocols here, say to limit the backed up model outputs to certain folders or exclude llik outputs, but the --sync-protocol option will add filters to limit the synced directories to those corresponding to the run submitted. Note that users do not need to specify run/job ids or configuration file names in the sync protocol. The flepimop batch-calibrate CLI will take advantage of flepimop sync's options to set paths appropriately to accommodate for run/job ids.

Modifying the first flepimop batch-calibrate command from before:

This command will submit an array job just like before, but will also add a dependent job with the same name prefixed with 'sync_'. This should looks like:

After those jobs finish the results can be found in a subdirectory named after the job and whose contents will look like:

Note that this contains the model_output directory but only limited to the batch run named 'sample_2pop-20250521T190823_Ro_all_test_limits' as well as a file called manifest.json which can be used to reproduce the run from scratch if needed.

Saving Model Outputs To S3 For Hopkins Users

For Hopkins affiliated users there is a configuration file patch included with flepiMoP that can be used to add S3 syncing for model outputs to s3://idd-inference-runs. Taking the example before of running the config_sample_2pop_inference.yml configuration we can slightly modify the command to:

This will take advantage of the patching abilities of the flepimop batch-calibrate to add a sync protocol named s3-idd-inference-runs that will save the results to the s3://idd-inference-runs bucket.

Access model files

See the section to ensure you have access to the correct files needed to run. On your local machine, determine the file paths to:

For running the model locally, especially for testing, non-inference runs, and short chains, we provide a guide for setting up and running in a conda environment, and provide a Docker container for use. A Docker container is an environment which is isolated from the rest of the operating system i.e. you can create files, programs, delete and everything but that will not affect your OS. It is a local virtual OS within your OS. We recommend Docker for users who are not familiar with setting up environments and seek a containerized environment to quickly launch jobs ;

For longer inference runs across multiple slots, we provide instructions and scripts for two methods to launch on SLURM HPC and on AWS using Docker. These methods are best for launching large jobs (long inference chains, multi-core and computationally expensive model runs), but not the best methods for debugging model setups.

Running locally

Running longer inference runs across multiple slots