Call for Challenge Cases

The idea of the challenge track is to provide participants with a set of case studies that tackle relevant SPL problems and challenge the state of the art. The challenge track happens in two phases.

In the first phase, there will be a call for cases. The challenge co-chairs will review the submitted cases to ensure that all required information is clearly described. Accepted cases will be part of the official conference proceedings. Authors of the accepted cases must attend the challenge track and participate in the discussion.

In the second phase, there will be a call for solutions to the accepted cases and cases from previous years. Submitted case solutions will be peer-reviewed by the challenge program committee. Accepted solutions will also be part of the official conference proceedings. Authors of accepted solutions must present their papers during the conference.

You can check the challenges from previous years on the official website of the challenge track: https://variability-challenges.github.io/

Proposed challenges should be new and should be kept simple, as too complex ones may discourage people from trying to propose solutions.

A case description can be:

  1. a particular data set with specific questions, or
  2. a call for a solution to a specific problem in a given context.

The following information is required for both types of case descriptions:

For datasets with specific questions:

  • What is your dataset, and how was it obtained?
  • What is the size of the dataset?
  • How can this data be accessed?
  • Do you provide tools to process the data?
  • Provide at least one concrete question you want participants to answer. You can provide multiple questions.
  • For each question, provide the criteria for evaluating an answer/solution.

For calls to solutions:

  • What is the concrete problem you want participants to solve?
  • How can a solution be evaluated? The following are some ideas on how to specify evaluation criteria:
    • Concrete evaluation metrics (e.g., precision, recall, accuracy etc. depending on the problem)
    • Concrete test cases participants can evaluate their solution against (e.g., provide inputs and expected outputs and participants are expected to provide a solution that gets from one point to the other)
    • A list of systems to evaluate their solution against (e.g., a list of C systems that have a large number of nested #ifdefs, because your problem only makes sense in the context of higher-order variability)
    • A reference implementation to compare against, according to particular metrics

Additional requirements for both types of cases:

  • The description should contain the URL of a public repository or artifact page containing all the instructions needed to initiate the case study.
  • The description should describe the requirements for the solution, or a requirement specification document should be published in the repository. The requirements’ description should help researchers who want to build the solution of or answer to the specific questions. 
  • Optional: Case authors may include a list of 5 senior PhD students or junior researchers (postdocs or junior professors) who have the expertise required to evaluate submitted solutions. The case authors themselves may be part of this list. The challenge track co-chairs will consider this list when creating the SPLC challenge program committee.

The challenge track co-chairs will select a small but representative set of case studies to be used for the challenge. After the description of accepted cases is finalized, a call for solutions will be released. Please note that case authors cannot submit a solution to their own case study.

Submission Information

If you have a suitable case study, we encourage you to write a short description with all the necessary details and submit it. Case study proposals have a minimum length of 2 pages and a maximum length of 6 pages, including all references and figures.

Submissions must adhere to the latest ACM Master Article Template: https://www.acm.org/publications/proceedings-template.

Latex users should use the “sigconf” option, so they are recommended to use the template that can be found in “sample-sigconf.tex”. In this way, the following latex code can be placed at the start of the latex document to create a double column layout:

\documentclass[sigconf,review]{acmart}
 
\acmConference[SPLC'23]{27th International Systems and Software Product Line Conference}{August 28--September 1, 2023}{Tokyo, Japan}

The SPLC proceedings will be published in the ACM Digital Library.

By submitting your article to an ACM Publication, you are hereby acknowledging that you and your co-authors are subject to all ACM Publications Policies, including ACM’s new Publications Policy on Research Involving Human Participants and Subjects. Alleged violations of this policy or any ACM Publications Policy will be investigated by ACM and may result in a full retraction of your paper, in addition to other potential penalties, as per ACM Publications Policy.

Please ensure that you and your co-authors obtain an ORCID ID, so you can complete the publishing process for your accepted paper. ACM has been involved in ORCID from the start and we have recently made a commitment to collect ORCID IDs from all of our published authors. The collection process has started and will roll out as a requirement throughout 2022. We are committed to improve author discoverability, ensure proper attribution and contribute to ongoing community efforts around name normalization; your ORCID ID will help in these efforts.

The author of the accepted submission must register and attend SPLC 2023 in order for the submission to be published.

Submission link

Submissions should be sent using EasyChair:

https://easychair.org/conferences/?conf=splc23 (Challenge Cases Track)

Important Dates (AoE time)

The following describes the important dates for the whole timeline of the challenge track.

  • Challenge case submission: February 3, 2023
  • Challenge case notification: February 18, 2023
  • Camera-ready of challenge case: March 1, 2023
  • Call for challenge solutions released: March 1, 2023
  • Deadlines of call for solutions
    • Challenge solution submission: June 12, 2023
    • Challenge solution notification: July 2, 2023
    • Camera-ready of challenge solution: July 9, 2023
  • Conference: August 28 – September 1, 2023

Chairs

Angelo Gargantini
University of Bergamo, Italy
Stefania Gnesi
Italian National Research Council, Italy

Contact: splc23challengecasestrack@easychair.org 

Program Committee

TBA