The International Planning Competition is organized in the context of the International Conference on Planning and Scheduling (ICAPS). It empirically evaluates state-of-the-art planning systems on a number of benchmark problems. The goals of the IPC are to promote planning research, highlight challenges in the planning community and provide new and interesting problems as benchmarks for future research.
Since 2004, probabilistic tracks have been part of the IPC under different names (as the International Probabilistic Planning competition or as part of the uncertainty tracks). After 2004, 2006, 2008, 2011 and 2014, the 6th edition of the IPC probabilistic tracks will be held in 2018 and conclude together with ICAPS, in June 2018, in Delft (Netherlands). This time it is organized by Thomas Keller, Scott Sanner and Buser Say.
The deterministic part of IPC is organized by Florian Pommerening and Alvaro Torralba. You can find information about it on ipc2018-classical.bitbucket.io.
The temporal part of IPC is organized by Andrew Coles, Amanda Coles and Moises Martinez. You can find information about it on ipc2018-temporal.bitbucket.io.
To stay up-to-date with the probabilistic tracks, register for the mailing list at https://groups.google.com/forum/#!forum/ipc2018-probabilistic
|Call for domains / expression of interest||July 14, 2017|
|Domain submission deadline||November 30, 2017|
|Demo problems provided||January 17, 2018|
|Expression of interest||February 4, 2018|
|Initial planner submission||March 25, 2018|
|Final planner submission||April 15, 2018|
|Planner abstract submission deadline||May 20, 2018|
|Contest run||May-June, 2018|
|Results announced||June, 2018|
|Result analysis deadline||July, 2018|
The competition is run in the tracks discrete MDP, continuous MDP and discrete SSP. These tracks focus on the maximization of the expected reward in a discrete or continuous environment (discrete and continuous MDP tracks) or on the minimization of the expected cost to reach a goal (discrete SSP track).
Additionally, there are the novel discrete data-based MDP and continuous data-based MDP tracks, which are versions of the discrete and continuous MDP tracks where a set of sample traces is provided as input rather than a declarative model of the MDP.
Update March 8, 2018:The data-based tracks have been cancelled due to a lack of participants.
otherwise, where is the highest average accumulate reward of all participants.
otherwise, where is the lowest average total cost of all participants.
Unlike in IPC 2011 and 2014, competitors do not run their planners themselves. This time, the competitors must submit the source code of their planners, and the organizers will run all planners on the actual competition domains/problems, unknown to the competitors until this time. This way, no fine-tuning of the planners will be possible.
All competitors must submit an abstract (max. 300 words) and a 4-page paper describing their planners. After the competition we encourage the participants to analyze the results of their planner and submit an extended version of their abstract. An important requirement for IPC 2018 competitors is to give the organizers the right to post their paper and the source code of their planners on the official IPC 2018 web site.
There are three important dates for the registration of planners (the deadlines are only important for the discrete tracks; information on deadlines for the continuous MDP track will follow later). This starts with an expression of interest in participation in one or more tracks, which is due February 4, 2018. Please let us know which tracks you are interested in and which input language you are planning to use by sending a mail to email@example.com.
We will use the container technology "Singularity" this year to promote reproducibility and help with compilation issues that have caused problems in the past.
The second step is to register your planner until March 25, 2018. To do so, send an email to firstname.lastname@example.org and let us know if you wish to use a mercurial or a git repository and provide us your bitbucket user names. We will then set up your repository on bitbucket, give you write access and let you know that you can submit your code to the repository. To do so, create one branch per track you want to participate in and name it according to the following list:
We will build all planners once a day and run them on a number of test cases. You can see the results for your planner on the build status page. Test your Singularity file locally (see below) and make sure it passes our automated tests. Please note that the test runs are shorter than the actual competition runs (only 10 iterations instead of 100, and tighter time constraints). The quality of your planner's results is not important at this point, so don't worry if the time limit seems unreasonably small.
A planner is officially registered in a track if it has a green box for that track on the build status page on March 25. You can still make any code changes you want until the final submission deadline on April 15. The build status on the website will update (once a day) when you push new changes the registered branches.
On the final submission deadline on April 15, 2018, we will change your access rights to the repository (or repositories) from write access to read access. If you find any bugs in your code afterwards, you can still fork the repository, fix the bug in the fork and create a pull request for the ipc2018-probabilistic-bot. If we detect any bugs while running your code, we'll let you know and you are also allowed to provide a bug fix. However, only bug fixes will be accepted after the deadline (in particular, we will not accept patches modifying behavior or tuning parameters).
In an effort to increase reproducibility and reduce the effort of running future IPCs, we are using software containers that contain the submitted planner and everything required to run it. We are using Singularity which is an alternative to the well-known Docker. Singularity (in contrast to Docker) is specifically designed for scientific experiments on HPC clusters and has low overhead.
Singularity containers can be viewed as light-weight alternatives to virtual machines that carry a program and all parts of the OS that are necessary to run it. They can be based on any docker image. We created an example submission (Singularity file) that uses the latest Ubuntu as a basis and uses apt-get to install required packages for Prost. It then builds the planner from the files that are next to the Singularity file in the repository.
In the following, we collect and answer frequently asked questions about Singularity. We'll update this section as we get more questions. If you run into problems using Singularity and your problem is not answered here, let us know.
You can install Singularity locally with the following commands (See the Singularity quick start guide for more details):
sudo apt install automake git clone https://github.com/singularityware/singularity.git cd singularity git checkout 2.4 ./autogen.sh ./configure --prefix=/usr/local make sudo make install
To test your Singularity script, please install Singularity (see above) and rddlsim, start rddlsim on the provided demo instances and run the following commands (replacing our demo submission with your repository):
hg clone https://bitbucket.org/ipc2018-probabilistic/demo-submission -r ipc2018-disc-mdp sudo singularity build planner.img demo-submission/Singularity mkdir rundir RUNDIR="$(pwd)/rundir" singularity run -C -H $RUNDIR planner.img recon_demo_inst_mdp__1 2323
The last command also shows how we will call the container during the competition: the parameter "-H" mounts the user's home directory. The parameter "-C" then isolates the container from the rest of the system. Only files written to the mounted directory will be stored permanently. Other created files (for example in /tmp) only persist for the current session and are cleaned up afterwards. When running the container on two instances at the same time, their run directories and sessions will be different, so the two runs cannot interact. The container itself is read-only after its creation.
We will also build your code about once per day and show the results for all planners on the build status page
, but files written there will be deleted after the run and there is a limit on the amount of data you are allowed to write to
. If possible, you should write to your home directory instead. See the question above for how to set up Singularity in this way for testing.
Please contact us if your license does not permit you to package the library into the container.
If we can acquire a license, we will mount the
installation files for the library while building
the container. You can then copy the installation
file into the container in the
step and install it in the
%setupsection will copy the files from the correct branch into the container. In the
%postsection you can then build your planner in the directory /planner.
Bootstrap: dockeris then parsed as
Bootstrap: docker\rand not recognized. Using Linux-style line endings should fix the issue.
To estimate the quality of the computed policies, we execute the policies 100 times and use the average as a metric for the policy's quality. We do so by having your planner interact as a client with the rddlsim server. Detailed information on the protocol that is used to exchange messages between client and server can be found in the file PROTOCOL.txt in the root directory of rddlsim.
Here, we just discuss the changes for IPC 2018:
If you want to test your planner locally, please update to the latest version of rddlsim, recompile and start the server with the command
./run rddl.competition.Server PROBLEM_DIR PORT 100 1 1 1800 ./ 1
where PORT is the port where the client can connect
(by default, this is 2323) and where 1800 is the
allowed time in seconds (this will differ from
instance to instance at IPC 2018).
PROBLEM_DIR is a directory that contains two subdirectories "client" and "server". The "client" directory contains the rddl or pddl files that are sent to your planner as part of the session-init message, and the "server" subdirectory contains matching rddl files that are used by the server. Even though it is still possible to set PROBLEM_DIR to a directory that contains rddl files that are shared by client and server, the tarballs that provide the demo domains and instances all contain separate client and server files.