The International Planning Competition is organized in the context of the International Conference on Planning and Scheduling (ICAPS). It empirically evaluates state-of-the-art planning systems on a number of benchmark problems. The goals of the IPC are to promote planning research, highlight challenges in the planning community and provide new and interesting problems as benchmarks for future research.
Since 2004, probabilistic tracks have been part of the IPC under different names (as the International Probabilistic Planning competition or as part of the uncertainty tracks). After 2004, 2006, 2008, 2011 and 2014, the 6th edition of the IPC probabilistic tracks will be held in 2018 and conclude together with ICAPS, in June 2018, in Delft (Netherlands). This time it is organized by Thomas Keller, Scott Sanner and Buser Say.
The deterministic part of IPC is organized by Florian Pommerening and Alvaro Torralba. You can find information about it on ipc2018.bitbucket.io.
|Call for domains / expression of interest||July, 2017|
|Domain submission deadline||November, 2017|
|Demo problems provided||November, 2017|
|Initial planner submission||January, 2018|
|Planner submission deadline||February, 2018|
|Planner abstract submission deadline||May, 2018|
|Contest run||May, 2018|
|Results announced||June, 2018|
|Result analysis deadline||July, 2018|
The competition is run in the tracks discrete MDP, continuous MDP and discrete SSP. These tracks focus on the maximization of the expected reward in a discrete or continuous environmen (discrete and continuous MDP tracks) or on the minimization of the expected cost to reach a goal (discrete SSP track).
Additionally, there are the novel discrete data-based MDP and continuous data-based MDP tracks, which are versions of the discrete and continuous MDP tracks where a set of sample traces is provided as input rather than a declarative model of the MDP.
Important: Most of the rules are not set in stone (yet), and we are open to discuss almost everything. If there is a rule that prevents you from competing in any track, and there is a modification that would allow you to compete, please let us know by writing to the mailing list or via email.
All five tracks have in common that the quality of the policy of each competitor is estimated as the average reward (for the MDP tracks) or cost (for the SSP track) over 100 runs (see the server documentation below for more information on the technical details on the evaluation procedure). Even though the time limit to complete the runs is selected such that there is a deliberation time of 0.5 seconds for each decision, it is also possible to compute a policy in a preprocessing step that takes almost all of the total available time and execute that policy shortly before the time limit is reached. As the total time limit is significantly higher than the time limit of the last two competitions (for a problem with the same horizon), we expect that both strategies are competetive options with different advantages and disadvantages.
Scores for each participant are computed with the same procedure that has been used at the probabilistic tracks of IPC 2011 and of IPC 2014: along with the competitors, we will run a simple planner that serves as the baseline. All competitors that
From the per-instance results, we compute per-domain results as the average over all instances, and from the per-domain results we compute the total score as the average over all domains. The total scores are finally used to rank the planners (the planner with the highest total score wins the competition).
Your planner interacts as a client with the rddlsim server that is used for evaluation. For now, we refer to the file PROTOCOL.txt which can be found in the root directory of rddlsim for information on the used protocol. Please note, though, that there will be an additional message the client has to send to indicate that it is about to start to execute the 100 runs that are actually used to compute the planner's score (you can find the reason why this change is necessary in this paper). We'll make sure that a list of all TCP/IP messages that are required to compete in any track of the IPC 2018 probabilistic tracks is available here as soon as possible.
Potential participants are requested to subscribe to the mailing list and send us an email expressing interest. Information on submitting the planner will be provided on the mailing list and the homepage later on.
The competitors must submit the source code of their planners that will be run by the organizers on the actual competition domains/problems, unknown to the competitors until this time. This way no fine-tuning of the planners will be possible.
All competitors must submit an abstract (max. 300 words) and a 4-page paper describing their planners. After the competition, we encourage the participants to analyze the results of their planner and submit an extended version of their abstract. An important requirement for IPC 2018 competitors is to give the organizers the right to post their paper and the source code of their planners on the official IPC 2018 web site.