Short internal tutorial on running locally using a "Docker" container.
There are more comprehensive directions in the How to run -> Running with Docker locally section, but this section has some specifics required to do US-specific, COVID-19 and flu-specific runs
Run Docker Image
Current Docker image: /hopkinsidd/flepimop:latest-dev
Docker is a software platform that allows you to build, test, and deploy applications quickly. Docker packages software into standardized units called containers that have everything the software needs to run including libraries, system tools, code, and runtime. This means you can run and install software without installing the dependencies in the system.
A docker container is an environment which is isolated from the rest of the operating system i.e. you can create files, programs, delete and everything but that will not affect your OS. It is a local virtual OS within your OS.
For flepiMoP, we have a docker container that will help you get running quickly!
In this command we run the docker image hopkinsidd/flepimop
. The -v
command is used to allocate space from Docker and mount it at the given location.
This mounts the data folder <dir1>
to a path called drp
within the docker environment, and the COVIDScenarioPipeline <dir2>
in flepimop
.
First, populate the folder name variables:
Then, export variables for some flags and the census API key (you can use your own):
Go into the Pipeline repo (making sure it is up to date on your favorite branch) and do the installation required of the repository:
Note: These installations take place in the docker container and not the Operating System. They must be made once while starting the container and need not be done for every time you are running tests, provided they have been installed once.
Everything is now ready. 🎉 Let's do some clean-up in the data folder (these files might not exist, but it's good practice to make sure your simulation isn't re-using some old files).
Stay in $DATA_PATH
, select a config, and build the setup. The setup creates the population seeding file (geodata) and the population mobility file (mobility). Then, run inference:
where:
n
is the number of parallel inference slots,
j
is the number of CPU cores it'll use in your machine,
k
is the number of iterations per slots.
It should run successfully and create a lot of files in model_output/
.
The last few lines visible on the command prompt should be:
[[1]]
[[1]][[1]]
[[1]][[1]][[1]]
NULL
Other helpful tools
To understand the basics of docker refer to the following: Docker Basics
To install docker for windows refer to the following link: Installing Docker
The following is a good tutorial for introduction to docker: Docker Tutorial
To run the entire pipeline we use the command prompt. To open the command prompt type “Command Prompt" in the search bar and open the command prompt. Here is a tutorial video for navigating through the command prompt: Command Prompt Tutorial
To test, we use the test folder (test_documentation_inference_us in this case) in the CovidScenariPipeline as the data repository folder. We run the docker container and set the paths.
or any HPC using the slurm workload manager
Rockfish administrators provided several partitions with different properties. For our needs (storage intensive and shared environment), we work in the /scratch4/struelo1/
partition, where we have 20T of space. Our folders are organized as:
code-folder: /scratch4/struelo1/flepimop-code/
where each user has its own subfolder, from where the repos are cloned and the runs are launched. e.g for user chadi, we'll find:
/scratch4/struelo1/flepimop-code/chadi/covidsp/Flu_USA
/scratch4/struelo1/flepimop-code/chadi/COVID19_USA
/scratch4/struelo1/flepimop-code/chadi/flepiMoP
...
(we keep separated repositories by users so that different versions of the pipeline are not mixed where we run several runs in parallel. Don't hesitate to create other subfolders in the code folder (/scratch4/struelo1/flepimop-code/chadi-flusight
, ...) if you need them.
Note that the repository is cloned flat, i.e the flepiMoP
repository is at the same level as the data repository, not inside it!
output folder:/scratch4/struelo1/flepimop-runs
stores the run outputs. After an inference run finishes, it's output and the logs files are copied from the $DATA_PATH/model_output
to /scratch4/struelo1/flepimop-runs/THISRUNJOBNAME
where the jobname is usually of the form USA-DATE.
Using ssh from your terminal, type in:
and enter your password when prompted. You'll be into rockfish's login node, which is a remote computer whose only purpose is to prepare and launch computations on so-called compute nodes.
Load the right modules for the setup:
Now, type the following line so git remembers your credential and you don't have to enter your token 6 times per day:
Now you need to create the conda environment. You will create the environment in two shorter commands, installing the python and R stuff separately. This can be extremely long if done in one command, so doing it in two helps. This command is quite long you'll have the time to brew some nice coffee ☕️:
type the following commands. $USER is a variable that contains your username.
This can be done in a second step -- but is necessary in order to push and pull to s3. Setup your AWS credentials by:
Then run ./aws-cli/bin/aws configure
and use the following :
log-in to rockfish via ssh, then type:
which will prepare the environment and setup variables for the validation date, the resume location and the run index for this run. If you don't want to set a variable, just hit enter.
Note that now the run-id of the run we resume from is automatically inferred by the batch script :)
Check that the conda environment is activated: you should see(flepimop-env)
on the left of your command-line prompt.
Then prepare the pipeline directory (if you have already done that and the pipeline hasn't been updated (git pull
says it's up to date) then you can skip these steps
Now flepiMoP is ready 🎉.
Next step is to setup the data. First $DATA_PATH to your data folder, and set any data options. If you are using the Delph Epidata API, first register for a key here: https://cmu-delphi.github.io/delphi-epidata/. Once you have a key, add that below where you see [YOUR API KEY]. Alternatively, you can put that key in your config file in the inference
section as gt_api_key: "YOUR API KEY"
.
For a COVID-19 run, do:
for Flu do:
Now for any type of run:
Do some clean-up before your run. The fast way is to restore the $DATA_PATH
git repository to its blank states (⚠️ removes everything that does not come from git):
Then run the preparatory script and you are good
If you want to profile how the model is using your memory resources during the run:
Now you may want to test that it works :
If this fails, you may want to investigate this error. In case this succeeds, then you can proceed by first deleting the model_output:
When an inference batch job is launched, a few post processing scripts are called to run automatically postprocessing-scripts.sh.
You can manually change what you want to run by editing this script.
Now you're fully set to go 🎉
To launch the whole inference batch job, type the following command:
This command infers everything from you environment variables, if there is a resume or not, what is the run_id, etc. The part after the "2" makes sure this file output is redirected to a script for logging, but has no impact on your submission.
If you'd like to have more control, you can specify the arguments manually:
If you want to send any post-processing outputs to #flepibot-test
instead of csp-production
, add the flag --slack-channel debug
Commit files to Github. After the job is successfully submitted, you will now be in a new branch of the data repo. Commit the ground truth data files to the branch on github and then return to the main branch:
but DO NOT finish up by git checking main like the aws instructions, as the run will use data in the current folder.
TODO JPSEH WRITE UP TO HERE
Two types of logfiles: in `$DATA_PATH`: slurm-JOBID_SLOTID.out and and filter_MC logs:
```tail -f /scratch4/struelo1/flepimop-runs/USA-20230130T163847/log_FCH_R16_lowBoo_modVar_ContRes_blk4_Jan29_tsvacc_100.txt
```
When approching the file number quota, type
to find which subfolders contains how many files
Check that the python comes from conda with which python
if some weird package missing errors arrive. Sometime conda magically disappears.
Don't use ipython
as it breaks click's flags
cleanup:
We use ntfy.sh for notification. Install ntfy on your Iphone or Android device. Then subscribe to the channel ntfy.sh/flepimop_alerts
where you'll receive notifications when runs are done.
End of job notifications goes as urgent priority.
Check your running jobs:
where job_id has your full array job_id and each slot after the under-score. You can see their status (R: running, P: pending), how long they have been running and soo on.
To cancel a job
To check your code prior to submitting a large batch job, it's often helpful to run an interactive session to debug your code and check everything works as you want. On 🪨🐠 this can be done using interact
like the below line, which requests an interactive session with 4 cores, 24GB of memory, for 12 hours.
The options here are [-n tasks or cores]
, [-t walltime]
, [-p partition]
and [-m memory]
, though other options can also be included or modified to your requirements. More details can be found on the ARCH User Guide.
Often you'll need to move files back and forth between Rockfish and your local computer. To do this, you can use Open-On-Demand, or any other command line tool.
scp -r <user>@rfdtn1.rockfish.jhu.edu:"<file path of what you want>" <where you want to put it in your local>
These steps are already done an affects all users, but might be interesting in case you'd like to run on another cluster
So our 🤖-friend can send us some notifications once a run is done.