Basic Usage

Setting up a Project

DIPLOMAT works on both single-animal and multi-animal DeepLabCut based projects, with no adjustments. So if you plan on using the DeepLabCut frontend, you’ll need to first setup a DeepLabCut project. This can be done by following the Single Animal DLC Project or Multi-Animal DLC Project guides included in the DeepLabCut documentation. It is recommended you follow the multi-animal guide as it will allow you to create a skeleton and label multiple individuals (DIPLOMAT will automatically pull the skeleton from DeepLabCut if one isn’t manually specified when running the diplomat tracking commands).

You’ll want to follow these user guides until you have finished training the network. Once the network is trained, you can begin using DIPLOMAT to begin tracking videos.

Tracking a Video

To run DIPLOMAT on a video, simply run either the diplomat track_and_interact or diplomat track command, as shown below (you’ll need to replace the paths below with actual paths to the project config file and the video):

# Run diplomat with UI intervention...
diplomat track_and_interact -c path/to/dlc/project/config.yaml/or/zipped/dlc/project -v path/to/video/to/run/on.mp4
# Run without the UI...
diplomat track -c path/to/dlc/project/config.yaml/or/zipped/dlc/project -v path/to/video/to/run/on.mp4

These commands will produce a .csv and .dipui file after tracking is done. Both of these commands support running on a list videos by simply passing paths using a comma separated list encased in square brackets:

diplomat track_and_interact -c path/to/dlc/project/config.yaml/or/zipped/dlc/project -v [path/to/video1.mp4, path/to/video2.webm, path/to/video3.mkv]

They also support working on more or less individuals by specifying the --num_outputs or -no flag:

# Video has 2 individuals.
diplomat track_and_interact -c path/to/dlc/project/config.yaml/or/zipped/dlc/project -no 2 -v path/to/video1.mp4
# Video has 5 individuals.
diplomat track_and_interact -c path/to/dlc/project/config.yaml/or/zipped/dlc/project -no 5 -v path/to/video2.mp4

Producing a Labeled Video

Once tracking is done, one can produce a labeled video using the diplomat annotate command and passing video/csv pairs to it, as shown below:

diplomat annotate -v path/to/video.mp4 -c path/to/video-track-results.csv

This will cause DIPLOMAT to produce another video placed next to the original video with _labeled tacked on to the end of its name. Solid markers indicate a tracked and detected part, hollow markers indicate the part is occluded or hidden.

Note

This command supports the following additonal final trace formats on top of DIPLOMAT csv files:

  • DeepLabCut .csv

  • DeepLabCut .h5

  • SLEAP .h5

Restoring UI to Make Major Adjustments

DIPLOMAT, when run in interactive or non-interactive mode with the “storage_mode” set to “disk” or “hybrid” (“hybrid” is the default setting), will save the video, all run session info, and frame data to a .dipui file. If the DIPLOMAT UI either crashes or you would like to edit your saved results in the feature complete version of the UI, you can restore the UI state using the diplomat interact command, as shown below:

diplomat interact -s path/to/ui/state/file.dipui

Making Minor Tweaks to Results

DIPLOMAT provides a stripped down version of the UI editor, which allows you to make minor modifications to results and also view results after tracking has already been done. This can be done passing video/csv pairs diplomat tweak command.

diplomat tweak -v path/to/video.mp4 -c path/to/video-track-results.csv

Note

This command supports the following additonal final trace formats on top of DIPLOMAT csv files:

  • DeepLabCut .csv

  • DeepLabCut .h5

  • SLEAP .h5

Saving Model Outputs for Later Analysis (All Frontends)

DIPLOMAT is capable of grabbing model outputs (confidence maps and location references) and dumping them to a file, which can improve performance when analyzing the same video multiple times or allow analysis to be completed somewhere else on a machine that lacks a GPU. To create a frame store for later analysis, run tracking with the frame store exporting predictor:

diplomat track_with -c path/to/config -v path/to/video -p FrameExporter -no 1

The above command will generate a .dlfs file next to the video. To run tracking on it, run one of DIPLOMAT’s tracking methods, but with the -fs flag passing in the frame store(s) instead of the video. Also, the project config is not needed when running on frame stores.

# Run DIPLOMAT with no UI...
diplomat track -fs path/to/fstore.dlfs -no <num_bodies>
# Run DIPLOMAT with UI...
diplomat track_and_interact -fs path/to/fstore.dlfs -no <num_bodies>
# Run DIPLOMAT with some other prediction algorithm
diplomat track_with -fs path/to/fstore.dlfs -p NameOfPredictorPlugin -no <num_bodies>

Video Utilities

The diplomat split_videos command provides functionality for both trimming and splitting videos into segments. It allows for splitting a video into fixed length segments or at exact second based offsets, as shown below:

# Split a video into 2 minute (120 second) chunks (-sps stands for seconds per segment).
diplomat split_videos -v path/to/video.mp4 -sps 120

# Split a video at exactly 30.25, 70.001, and 500 seconds in.
diplomat split_videos -v path/to/video.mp4 -sps [30.25, 70.001, 500]

# Like all other commands, multiple videos can be passed.
diplomat split_videos -v [path/to/video1.mov, path/to/video2.avi] -sps 120

# Can specify an alternative output format via fourcc code and file extension...
diplomat split_videos -v path/to/video1.mov -sps 120 -ofs mp4v -oe .mp4