Research PapersICFP 2019

PACMPL (ICFP) seeks contributions on the design, implementations, principles, and uses of functional programming, covering the entire spectrum of work, from practice to theory, including its peripheries. Authors of papers published in this issue of PACMPL will present their work at ICFP in Berlin, providing an opportunity for researchers and developers to hear about the latest work in functional programming.

Accepted Papers

Call for Papers

Scope

PACMPL issue ICFP 2019 seeks original papers on the art and science of functional programming. Submissions are invited on all topics from principles to practice, from foundations to features, and from abstraction to application. The scope includes all languages that encourage functional programming, including both purely applicative and imperative languages, as well as languages with objects, concurrency, or parallelism. Topics of interest include (but are not limited to):

• Language Design: concurrency, parallelism, and distribution; modules; components and composition; metaprogramming; macros; pattern matching; type systems; type inference; dependent types; session types; gradual typing; refinement types; interoperability; domain-specific languages; imperative programming; object-oriented programming; logic programming; probabilistic programming; reactive programming; generic programming; bidirectional programming.

• Implementation: abstract machines; virtual machines; interpretation; compilation; compile-time and run-time optimization; garbage collection and memory management; runtime systems; multi-threading; exploiting parallel hardware; interfaces to foreign functions, services, components, or low-level machine resources.

• Software-Development Techniques: algorithms and data structures; design patterns; specification; verification; validation; proof assistants; debugging; testing; tracing; profiling; build systems; program synthesis.

• Foundations: formal semantics; lambda calculus; program equivalence; rewriting; type theory; logic; category theory; monads; continuations; control; state; effects; names and binding; program verification.

• Analysis and Transformation: control-flow; data-flow; abstract interpretation; partial evaluation; program calculation.

• Applications: symbolic computing; formal-methods tools; artificial intelligence; systems programming; distributed-systems and web programming; hardware design; databases; XML processing; scientific and numerical computing; graphical user interfaces; graphics and multimedia; GPU programming; scripting; system administration; security.

• Education: teaching introductory programming; parallel programming; mathematical proof; algebra.

Submissions will be evaluated according to their relevance, correctness, significance, originality, and clarity. Each submission should explain its contributions in both general and technical terms, clearly identifying what has been accomplished, explaining why it is significant, and comparing it with previous work. The technical content should be accessible to a broad audience.

PACMPL issue ICFP 2019 also welcomes submissions in two separate categories — Functional Pearls and Experience Reports — that must be marked as such at the time of submission and that need not report original research results. Detailed guidelines on both categories are given at the end of this call.

Please contact the principal editor if you have questions or are concerned about the appropriateness of a topic.

Preparation of submissions

Deadline: The deadline for submissions is Friday, March 1, 2019, Anywhere on Earth (https://en.wikipedia.org/wiki/Anywhere_on_Earth). This deadline will be strictly enforced.

Formatting: Submissions must be in PDF format, printable in black and white on US Letter sized paper, and interpretable by common PDF tools. All submissions must adhere to the “ACM Small” template that is available (in both LaTeX and Word formats) from https://www.acm.org/publications/authors/submissions. For authors using LaTeX, a lighter-weight package, including only the essential files, is available from http://sigplan.org/Resources/Author/#acmart-format.

There is a limit of 25 pages for a full paper or Functional Pearl and 12 pages for an Experience Report; in either case, the bibliography will not be counted against these limits. Submissions that exceed the page limits or, for other reasons, do not meet the requirements for formatting, will be summarily rejected. Supplementary material can and should be separately submitted (see below).

See also PACMPL’s Information and Guidelines for Authors at https://pacmpl.acm.org/authors.cfm.

Submission: Submissions will be accepted at https://icfp19.hotcrp.com/

Improved versions of a paper may be submitted at any point before the submission deadline using the same web interface.

Author Response Period: Authors will have a 72-hour period, starting at 14:00 UTC on Tuesday, April 16, 2019, to read reviews and respond to them.

Supplementary Material: Authors have the option to attach supplementary material to a submission, on the understanding that reviewers may choose not to look at it. This supplementary material should not be submitted as part of the main document; instead, it should be uploaded as a separate PDF document or tarball.

Supplementary material should be uploaded at submission time, not by providing a URL in the paper that points to an external repository.

Authors are free to upload both anonymized and non-anonymized supplementary material. Anonymized supplementary material will be visible to reviewers immediately; non-anonymized supplementary material will be revealed to reviewers only after they have submitted their review of the paper and learned the identity of the author(s).

Authorship Policies: All submissions are expected to comply with the ACM Policies for Authorship that are detailed at https://www.acm.org/publications/authors/information-for-authors.

Republication Policies: Each submission must adhere to SIGPLAN’s republication policy, as explained on the web at http://www.sigplan.org/Resources/Policies/Republication.

Resubmitted Papers: Authors who submit a revised version of a paper that has previously been rejected by another conference have the option to attach an annotated copy of the reviews of their previous submission(s), explaining how they have addressed these previous reviews in the present submission. If a reviewer identifies him/herself as a reviewer of this previous submission and wishes to see how his/her comments have been addressed, the principal editor will communicate to this reviewer the annotated copy of his/her previous review. Otherwise, no reviewer will read the annotated copies of the previous reviews.

Review Process

This section outlines the two-stage process with lightweight double-blind reviewing that will be used to select papers for PACMPL issue ICFP 2019. We anticipate that there will be a need to clarify and expand on this process, and we will maintain a list of frequently asked questions and answers on the conference website to address common concerns.

PACMPL issue ICFP 2019 will employ a two-stage review process. The first stage in the review process will assess submitted papers using the criteria stated above and will allow for feedback and input on initial reviews through the author response period mentioned previously. At the review meeting, a set of papers will be conditionally accepted and all other papers will be rejected. Authors will be notified of these decisions on May 3, 2019.

Authors of conditionally accepted papers will be provided with committee reviews (just as in previous conferences) along with a set of mandatory revisions. After four weeks (May 31, 2019), the authors will provide a second submission. The second and final reviewing phase assesses whether the mandatory revisions have been adequately addressed by the authors and thereby determines the final accept/reject status of the paper. The intent and expectation is that the mandatory revisions can be addressed within four weeks and hence that conditionally accepted papers will in general be accepted in the second phase.

The second submission should clearly identify how the mandatory revisions were addressed. To that end, the second submission must be accompanied by a cover letter mapping each mandatory revision request to specific parts of the paper. The cover letter will facilitate a quick second review, allowing for confirmation of final acceptance within two weeks. Conversely, the absence of a cover letter will be grounds for the paper’s rejection.

PACMPL issue ICFP 2019 will employ a lightweight double-blind reviewing process. To facilitate this, submitted papers must adhere to two rules:

1. author names and institutions must be omitted, and
2. references to authors’ own related work should be in the third person (e.g., not “We build on our previous work …” but rather “We build on the work of …”).

The purpose of this process is to help the reviewers come to an initial judgement about the paper without bias, not to make it impossible for them to discover the authors if they were to try. Nothing should be done in the name of anonymity that weakens the submission or makes the job of reviewing the paper more difficult (e.g., important background references should not be omitted or anonymized). In addition, authors should feel free to disseminate their ideas or draft versions of their paper as they normally would. For instance, authors may post drafts of their papers on the web or give talks on their research ideas.

Information for Authors of Accepted Papers

• As a condition of acceptance, final versions of all papers must adhere to the new ACM Small format. The page limit for the final versions of papers will be increased by two pages to help authors respond to reviewer comments and mandatory revisions: 27 pages plus bibliography for a regular paper or Functional Pearl, 14 pages plus bibliography for an Experience Report.

• Authors of accepted submissions will be required to agree to one of the three ACM licensing options: open access on payment of a fee (recommended, and SIGPLAN can cover the cost as described next); copyright transfer to ACM; or retaining copyright but granting ACM exclusive publication rights. Further information about ACM author rights is available from http://authors.acm.org.

• PACMPL is a Gold Open Access journal. It will be archived in ACM’s Digital Library, but no membership or fee is required for access. Gold Open Access has been made possible by generous funding through ACM SIGPLAN, which will cover all open access costs in the event authors cannot. Authors who can cover the costs may do so by paying an Article Processing Charge (APC). PACMPL, SIGPLAN, and ACM Headquarters are committed to exploring routes to making Gold Open Access publication both affordable and sustainable.

• ACM offers authors a range of copyright options, one of which is Creative Commons CC-BY publication; this is the option recommended by the PACMPL editorial board. A reasoned argument in favour of this option can be found in the article Why CC-BY? published by OASPA, the Open Access Scholarly Publishers Association.

• We intend that the papers will be freely available for download from the ACM Digital Library in perpetuity via the OpenTOC mechanism.

• ACM Author-Izer is a unique service that enables ACM authors to generate and post links on either their home page or institutional repository for visitors to download the definitive version of their articles from the ACM Digital Library at no charge. Downloads through Author-Izer links are captured in official ACM statistics, improving the accuracy of usage and impact measurements. Consistently linking to the definitive version of an ACM article should reduce user confusion over article versioning. After an article has been published and assigned to the appropriate ACM Author Profile pages, authors should visit http://www.acm.org/publications/acm-author-izer-service to learn how to create links for free downloads from the ACM DL.

• At least one author of each accepted submissions will be expected to attend and present their paper at the conference. The schedule for presentations will be determined and shared with authors after the full program has been selected. Presentations will be videotaped and released online if the presenter consents.

• The official publication date is the date the papers are made available in the ACM Digital Library. This date may be up to two weeks prior to the first day of the conference. The official publication date affects the deadline for any patent filings related to published work.

Artifact Evaluation

Authors of papers that are conditionally accepted in the first phase of the review process will be encouraged (but not required) to submit supporting materials for Artifact Evaluation. These items will then be reviewed by an Artifact Evaluation Committee, separate from the paper Review Committee, whose task is to assess how the artifacts support the work described in the associated paper. Papers that go through the Artifact Evaluation process successfully will receive a seal of approval printed on the papers themselves. Authors of accepted papers will be encouraged to make the supporting materials publicly available upon publication of the papers, for example, by including them as “source materials” in the ACM Digital Library. An additional seal will mark papers whose artifacts are made available, as outlined in the ACM guidelines for artifact badging.

Participation in Artifact Evaluation is voluntary and will not influence the final decision regarding paper acceptance.

Special categories of papers

In addition to research papers, PACMPL issue ICFP solicits two kinds of papers that do not require original research contributions: Functional Pearls, which are full papers, and Experience Reports, which are limited to half the length of a full paper. Authors submitting such papers should consider the following guidelines.

Functional Pearls

A Functional Pearl is an elegant essay about something related to functional programming. Examples include, but are not limited to:

• a new and thought-provoking way of looking at an old idea

• an instructive example of program calculation or proof

• a nifty presentation of an old or new data structure

• an interesting application of functional programming techniques

• a novel use or exposition of functional programming in the classroom

While pearls often demonstrate an idea through the development of a short program, there is no requirement or expectation that they do so. Thus, they encompass the notions of theoretical and educational pearls.

Functional Pearls are valued as highly and judged as rigorously as ordinary papers, but using somewhat different criteria. In particular, a pearl is not required to report original research, but, it should be concise, instructive, and entertaining. A pearl is likely to be rejected if its readers get bored, if the material gets too complicated, if too much specialized knowledge is needed, or if the writing is inelegant. The key to writing a good pearl is polishing.

A submission that is intended to be treated as a pearl must be marked as such on the submission web page, and should contain the words “Functional Pearl” somewhere in its title or subtitle. These steps will alert reviewers to use the appropriate evaluation criteria. Pearls will be combined with ordinary papers, however, for the purpose of computing the conference’s acceptance rate.

Experience Reports

The purpose of an Experience Report is to help create a body of published, refereed, citable evidence that functional programming really works — or to describe what obstacles prevent it from working.

Possible topics for an Experience Report include, but are not limited to:

• insights gained from real-world projects using functional programming

• comparison of functional programming with conventional programming in the context of an industrial project or a university curriculum

• project-management, business, or legal issues encountered when using functional programming in a real-world project

• curricular issues encountered when using functional programming in education

• real-world constraints that created special challenges for an implementation of a functional language or for functional programming in general

An Experience Report is distinguished from a normal PACMPL issue ICFP paper by its title, by its length, and by the criteria used to evaluate it.

• Both in the papers and in any citations, the title of each accepted Experience Report must end with the words “(Experience Report)” in parentheses. The acceptance rate for Experience Reports will be computed and reported separately from the rate for ordinary papers.

• Experience Report submissions can be at most 12 pages long, excluding bibliography.

• Each accepted Experience Report will be presented at the conference, but depending on the number of Experience Reports and regular papers accepted, authors of Experience reports may be asked to give shorter talks.

• Because the purpose of Experience Reports is to enable our community to accumulate a body of evidence about the efficacy of functional programming, an acceptable Experience Report need not add to the body of knowledge of the functional-programming community by presenting novel results or conclusions. It is sufficient if the Report states a clear thesis and provides supporting evidence. The thesis must be relevant to ICFP, but it need not be novel.

The review committee will accept or reject Experience Reports based on whether they judge the evidence to be convincing. Anecdotal evidence will be acceptable provided it is well argued and the author explains what efforts were made to gather as much evidence as possible. Typically, more convincing evidence is obtained from papers which show how functional programming was used than from papers which only say that functional programming was used. The most convincing evidence often includes comparisons of situations before and after the introduction or discontinuation of functional programming. Evidence drawn from a single person’s experience may be sufficient, but more weight will be given to evidence drawn from the experience of groups of people.

An Experience Report should be short and to the point: it should make a claim about how well functional programming worked on a particular project and why, and produce evidence to substantiate this claim. If functional programming worked in this case in the same ways it has worked for others, the paper need only summarize the results — the main part of the paper should discuss how well it worked and in what context. Most readers will not want to know all the details of the project and its implementation, but the paper should characterize the project and its context well enough so that readers can judge to what degree this experience is relevant to their own projects. The paper should take care to highlight any unusual aspects of the project. Specifics about the project are more valuable than generalities about functional programming; for example, it is more valuable to say that the team delivered its software a month ahead of schedule than it is to say that functional programming made the team more productive.

If the paper not only describes experience but also presents new technical results, or if the experience refutes cherished beliefs of the functional-programming community, it may be better to submit it as a full paper, which will be judged by the usual criteria of novelty, originality, and relevance. The principal editor will be happy to advise on any concerns about which category to submit to.

Previous versions of this FAQ were developed and refined by SIGPLAN officers and conference organizers with input from their committees and the community. Please contact the current program chair with any questions or comments about this version of the document.

Last updated: January 31, 2017

General

• Why are you using double-blind reviewing?

• Do you really think blinding actually works? I suspect reviewers can often guess who the authors are anyway.

• Couldn’t blind submission create an injustice where a paper is inappropriately rejected based upon supposedly-prior work which was actually by the same authors and not previously published?

• What happens if the Program Chair has a conflict with the authors of a submitted paper?

• Can you provide pointers to more information about bias in merit reviewing?

For authors

• What exactly do I have to do to anonymize my paper?

• I would like to provide supplementary material for consideration, such as the code of my implementation or the proofs of theorems. How do I do this?

• Is there a way for me to submit anonymous supplemental material which could be considered by a reviewer before she submits her review (rather than potentially non-anonymous material that can only be viewed afterward)?

• Can I supplement my submission using a URL that links to auxiliary materials instead of submitting such materials to the HotCRP system directly?

• I am building on my own past work on the WizWoz system. Do I need to rename this system in my paper for purposes of anonymity, so as to remove the implied connection between my authorship of past work on this system and my present submission?

• I am submitting a paper that extends my own work that previously appeared at a workshop. Should I anonymize any reference to that prior work?

• Am I allowed to post my (non-blinded) paper on my web page? Can I advertise the unblinded version of my paper on mailing lists or send it to colleagues? May I give a talk about my work while it is under review?

• Will the fact that ICFP is double-blind have an impact on handling conflicts-of interest? When I am asked by the submission system to identify conflicts of interest, what criteria should I use?

For reviewers

• What should I do if I if I learn the authors’ identity? What should I do if a prospective ICFP author contacts me and asks to visit my institution?

• The authors have provided a URL to supplemental material. I would like to see the material but I worry they will snoop my IP address and learn my identity. What should I do?

• If I am assigned a paper for which I feel I am not an expert, how do I seek an outside review?

• May I ask one of my students to do a review for me?

• How do I handle potential conflicts of interest given that I cannot see the author names?

• Are PC members allowed to submit papers? If so, how are they handled?

• Are external reviewers allowed to submit papers? If so, how are they handled?

• How should I handle a paper I feel is very good, and yet would be a better fit for POPL (or PLDI or OOPSLA)?

• How should I handle a paper that is out of scope for ICFP?

General

Q: Why are you using double-blind reviewing?

A: Our goal is to give each a reviewer an unbiased “first look” at each paper. Studies have shown that a reviewer’s attitude toward a submission may be affected, even unconsciously, by the identity of the author (see link below to more details). We want reviewers to be able to approach each submission without such involuntary reactions as “Barnaby; he writes a good paper” or “Who are these people? I have never heard of them.” For this reason, we ask that authors to omit their names from their submissions, and that they avoid revealing their identity through citation. Note that many systems and security conferences use double-blind reviewing and have done so for years (e.g., SIGCOMM, OSDI, IEEE Security and Privacy, SIGMOD). POPL and PLDI have done it for the last several years.

A key principle to keep in mind is that we intend this process to be cooperative, not adversarial. If a reviewer does discover an author’s identity through a subtle clue or oversight, then the author will not be penalized.

For those wanting more information, see the list of studies about gender bias in other fields and links to CS-related articles that cover this and other forms of bias below.

Q: Do you really think blinding actually works? I suspect reviewers can often guess who the authors are anyway.

A: Studies of blinding with the flavor we are using show that author identities remain unknown 53% to 79% of the time (see Snodgrass, linked below, for details). Moreover, about 5-10% of the time (again, see Snodgrass), a reviewer is certain of the authors, but then turns out to be at least partially mistaken. So, while sometimes authorship can be guessed correctly, the question is, is imperfect blinding better than no blinding at all? If author names are not explicitly in front of the reviewer on the front page, does that help at all even for the remaining submissions where it would be possible to guess? Our conjecture is that on balance the answer is “yes.”

Q: Couldn’t blind submission create an injustice where a paper is inappropriately rejected based upon supposedly-prior work which was actually by the same authors and not previously published?

A: This is indeed a serious issue. In the approach we are taking for ICFP ’18, author names are revealed to reviewers after they have submitted their review. Therefore, a reviewer can correct their review if they indeed have penalized the authors inappropriately. Unblinding prior to the PC meeting also avoids abuses in which committee members end up advancing the cause of a paper with which they have a conflict.

Q: What happens if the Program Chair has a conflict with the authors of a submitted paper?

A: The reviewing process for papers with which the Program Chair has a conflict will be managed by another PC member.

Q: Can you provide pointers to more information about bias in merit reviewing?

A: More detailed information on this topic has been gathered in the section at the end of this document.

For Authors

Q: What exactly do I have to do to anonymize my paper?

A: Your job is not to make your identity undiscoverable but simply to make it possible for our reviewers to evaluate your submission without having to know who you are. The specific guidelines stated in the call for papers are simple: omit authors’ names from your title page (or list them as “omitted for submission”), and when you cite your own work, refer to it in the third person. For example, if your name is Smith and you have worked on amphibious type systems, instead of saying “We extend our earlier work on statically typed toads (Smith 2004),” you might say “We extend Smith’s (2004) earlier work on statically typed toads.” Also, be sure not to include any acknowledgements that would give away your identity. If you have any questions, feel free to ask the PC chair.

Q: I would like to provide supplementary material for consideration, such as the code of my implementation or the proofs of theorems. How do I do this?

A: On the submission site there will be an option to submit supplementary material along with your main paper. This supplementary material may or may not be anonymized; if not anonymized, it will only be revealed to reviewers after they have submitted their review of your paper and learned your identity. Reviewers are under no obligation to look at this material. The submission itself is the object of review and so it should strive to convince the reader of at least the plausibility of reported results; supplemental material only serves to confirm, in more detail, the idea argued in the paper. Of course, reviewers are free to change their review upon viewing supplemental material (or for any other reason). For those authors who wish to supplement, we encourage them to mention the supplement in the body of the paper so reviewers know to look for it, if necessary. For example, “The proof of Lemma 1 is included in the anonymous supplemental material submitted with this paper.”

Q: Is there a way for me to submit anonymous supplemental material which could be considered by a reviewer before she submits her review (rather than potentially non-anonymous material that can only be viewed afterward)?

A: Yes, submit it via HotCRP. Authors have been known to release a TR, code, etc. via an anonymous hosting service, and to include a URL to that material in the paper. However, we discourage authors from using such tactics except for materials that cannot, for some reason, be uploaded to the official site (e.g., a live demo). We emphasize that authors should strive to make their paper as convincing as possible within the submission page limit, in case reviewers choose not to access supplemental material. Also, see the next question.

Q: Can I supplement my submission using a URL that links to auxiliary materials instead of submitting such materials to the HotCRP system directly?

A In general, we discourage authors from providing supplementary materials via links to external web sites. It is possible to change the linked items after the submission deadline has passed, and, to be fair to all authors, we would like to be sure reviewers evaluate materials that have been completed prior to the submission deadline. Having said that, it is appropriate to link to items, such as an online demo, that cannot easily be submitted. Of course, attempting to discover the reviewers for your paper by tracking visitors to such a demo site would be a breach of academic integrity. Supplementary items such as PDFs should always be uploaded to HotCRP.

Q: I am building on my own past work on the WizWoz system. Do I need to rename this system in my paper for purposes of anonymity, so as to remove the implied connection between my authorship of past work on this system and my present submission?

A: No. The relationship between systems and authors changes over time, so there will be at least some doubt about authorship. Increasing this doubt by changing the system name would help with anonymity, but it would compromise the research process. In particular, changing the name requires explaining a lot about the system again because you can’t just refer to the existing papers, which use the proper name. Not citing these papers runs the risk of the reviewers who know about the existing system thinking you are replicating earlier work. It is also confusing for the reviewers to read about the paper under Name X and then have the name be changed to Name Y. Will all the reviewers go and re-read the final version with the correct name? If not, they have the wrong name in their heads, which could be harmful in the long run.

Q: I am submitting a paper that extends my own work that previously appeared at a workshop. Should I anonymize any reference to that prior work?

A: No. But we recommend you do not use the same title for your ICFP submission, so that it is clearly distinguished from the prior paper. In general there is rarely a good reason to anonymize a citation. One possibility is for work that is tightly related to the present submission and is also under review. But such works may often be non-anonymous. When in doubt, contact the PC Chair.

Q: Am I allowed to post my (non-blinded) paper on my web page? Can I advertise the unblinded version of my paper on mailing lists or send it to colleagues? May I give a talk about my work while it is under review?

A: As far as the authors’ publicity actions are concerned, a paper under double-blind review is largely the same as a paper under regular (single-blind) review. Double-blind reviewing should not hinder the usual communication of results.

That said, we do ask that you not attempt to deliberately subvert the double-blind reviewing process by announcing the names of the authors of your paper to the potential reviewers of your paper. It is difficult to define exactly what counts as “subversion” here, but generally speaking please refrain from sending individual e-mail to members of the PC or external reviewers about your work (unless they are conflicted out anyway).

It is perfectly fine, for example, to visit other institutions and give talks about your work, to present your submitted work during job interviews, to present your work at professional meetings (e.g., Dagstuhl), or to post your work on your web page. In general, PC members and external reviewers will not be asked to recuse themselves if they discover the (likely) identity of an author through such means.

Please refrain from posting the title of your paper on broadly visible social media or a major mailing list (e.g., TYPES). That's not so different from posting a submission on your web page, put the “push” nature of those mechanisms makes the information more difficult for potential reviewers to ignore. Posts that omit a paper’s title are less concerning. Of course, tweeting “submitted my ICFP paper” is encouraged!

If you are not sure about what constitutes “subversion,” please consult directly with the Program Chair.

Q: Will the fact that ICFP is double-blind have an impact on handling conflicts-of interest? When I am asked by the submission system to identify conflicts of interest, what criteria should I use?

A: Using DBR does not change the principle that reviewers should not review papers with which they have a conflict of interest, even if they do not immediately know who the authors are. Quoting (with slight alteration) from the ACM SIGPLAN review policies document:

A conflict of interest is defined as a situation in which the reviewer can be viewed as being able to benefit personally in the process of reviewing a paper. For example, if a reviewer is considering a paper written by a member of his own group, a current student, his advisor, or a group that he is seen as being in close competition with, then the outcome of the review process can have direct benefit to the reviewer’s own status. If a conflict of interest exists, the potential reviewer should decline to review the paper.

As an author, you should list PC members (and any others, because others may be asked for external reviews) that you believe have a conflict with you. While particular criteria for making this determination may vary, please apply the following guidelines, identifying a potential reviewer Bob as conflicted if

• Bob was your co-author or collaborator at some point within the last 2 years

• Bob is an advisor or advisee of yours

• Bob is a family member

• Bob has a non-trivial financial stake in your work (e.g., invested in your startup company)

Also please identify institutions with which you are affiliated; all employees or affiliates of these institutions will also be considered conflicted.

If a possible reviewer does not meet the above criteria, please do not identify him/her as conflicted. Doing so could be viewed as an attempt to prevent a qualified, but possibly skeptical reviewer from reviewing your paper. If you nevertheless believe that a reviewer who does not meet the above criteria is conflicted, you may identify the person and send a note to the PC Chair.

For Reviewers

Q: What should I do if I if I learn the authors’ identity? What should I do if a prospective ICFP author contacts me and asks to visit my institution?

A: If at any point you feel that the authors’ actions are largely aimed at ensuring that potential reviewers know their identity, you should contact the Program Chair. Otherwise you should not treat double-blind reviewing differently from regular blind reviewing. In particular, you should refrain from seeking out information on the authors’ identity, but if you discover it accidentally this will not automatically disqualify you as a reviewer. Use your best judgment.

Q: The authors have provided a URL to supplemental material. I would like to see the material but I worry they will snoop my IP address and learn my identity. What should I do?

A: Contact the Program Chair, who will download the material on your behalf and make it available to you.

Q: If I am assigned a paper for which I feel I am not an expert, how do I seek an outside review?

A: PC members and external reviewers should do their own reviews, not delegate them to someone else. If doing so is problematic for some papers (e.g., you don’t feel completely qualified), then consider the following options. First, submit a review for your paper that is as careful as possible, outlining areas where you think your knowledge is lacking. Assuming we have sufficient expert reviews, that could be the end of it: non-expert reviews are valuable too, since conference attendees are by-and-large not experts for any given paper. Second, if you feel like the gaps in your knowledge are substantial, submit a first cut review, and then work with the PC chair to solicit an external review. This is easy: after submitting your review the paper is unblinded, so you at least know not to solicit the authors! You will also know other reviewers of the paper that have already been solicited. If none of these expert reviewers is acceptable to you, just check with the PC Chair that the person you do wish to solicit is not conflicted with the authors. In addition, the PC chair will attempt to balance the load on external reviewers. Keep in mind that while we would like the PC to make as informed a decision as possible about each submitted paper, each additional review we solicit places a burden on the community.

As a last resort, if you feel like your review would be extremely uninformed and you’d rather not even submit a first cut, contact the PC Chair, and another reviewer will be assigned.

Q: May I ask one of my students to do a review for me?

A: Having students (or interns at a research lab) participate in the review process is good for their education. However, you should not just “offload” your reviews to your students. Each review assigned to you is your responsibility. We recommend the following process: If you are sure that your student’s conflicts of interest are a subset of your own, you and your student may both begin to do your own separate reviews in parallel. (A student’s review should never simply be a substitute for your own work.) If your student’s conflicts of interest are not a subset of your own, you may do your own first-cut review first and then unblind the authors so you can check, or you may consult with the PC chair. Either way, once the student has completed their review, you should check the review to ensure the tone is professional and the content is appropriate. Then you may merge the student’s review with your own.

Q: How do I handle potential conflicts of interest given that I cannot see the author names?

A: The conference review system will ask that you identify conflicts of interest when you get an account on the submission system. Please see the related question applied to authors to decide how to identify conflicts. Feel free to also identify additional authors whose papers you feel you could not review fairly for reasons other than those given (e.g., strong personal friendship).

Q: Are PC members allowed to submit papers? If so, how are they handled?

A: PC submissions are allowed (except for the PC chair) with the usual condition of a higher standard (clearly as good as or better than other accepted papers, that is, strong support and no significant doubt). PC papers will be reviewed by the ERC, and they will not be discussed at the physical PC meeting.

Q: Are ERC members and external reviewers allowed to submit papers? If so, how are they handled?

A: ERC members and external reviewers are allowed to submit papers. Their papers will be reviewed like any other paper. There is no “higher standard” for ERC or external reviewer papers.

Q: How should I handle a paper I feel is very good, and yet would be a better fit for POPL (or PLDI or OOPSLA)?

A: The scope of ICFP is broad and encompasses all topics that pertain to functional programming. Hence, if you feel a paper would be an excellent POPL (or PLDI or OOPSLA) paper then it might also be an excellent ICFP paper. To be accepted at ICFP, a paper must discuss functional programming in some way, shape or form and it must have the potential to make a lasting impact on our field.

Q: How should I handle a paper that is out of scope for ICFP?

A: The scope of ICFP is broad and encompasses all topics that pertain to functional programming. However, if you discover you have been assigned a paper that does not contribute to the study of functional programming, please contact the program chair. We will discuss it and may decide to reject the paper on grounds of scope. Of course, if we decide after all that the paper is within the scope of ICFP, then you should review it like any other paper.

More Information about Bias in Merit Reviewing

Note that the information in this section has been put together by past and current organizers of the conference; not all of the program committee members or external reviewers are necessarily persuaded by it.

Kathryn McKinley’s editorial makes the case for double-blind reviewing from a computer science perspective. Her article cites Richard Snodgrass’s SIGMOD record editorial, which collects many studies of the effects of potential bias in peer review. The POPL ’12 Program Chair’s Report provides details about the introduction of double-blind reviewing in another SIGPLAN conference.

Here are a few studies on the potential effects of bias manifesting in a merit review process, focusing on bias against women. (These were collected by David Wagner.)

• There’s the famous story of gender bias in orchestra try-outs, where moving to blind auditions seems to have increased the hiring of female musicians by up to 33% or so. Today some orchestras even go so far as to ask musicians to remove their shoes (or roll out thick carpets) before auditioning, to try to prevent gender-revealing cues from the sound of the auditioner’s shoes.

• One study found bias in assessment of identical CVs but with names and genders changed. In particular, the researchers mailed out CVs for a faculty position, but randomly swapped the gender of the name on some of them. They found that both men and women reviewers ranked supposedly-male job applicants higher than supposedly-female applicants — even though the contents of the CV were identical. Presumably, none of the reviewers thought of themselves as biased, yet their evaluations in fact exhibited gender bias. (However: In contrast to the gender bias at hiring time, if the reviewers were instead asked to evaluate whether a candidate should be granted tenure, the big gender differences disappeared. For whatever that’s worth.)

• The Implicit Association Test illustrates how factors can bias our decision-making, without us realising it. For instance, a large fraction of the population has a tendency to associate men with career (professional life) and women with family (home life), without realizing it. The claim is that we have certain gender stereotypes and schemas which unconsciously influence the way we think. The interesting thing about the IAT is that you can take it yourself. If you want to give it a try, select the Gender-Career IAT or the Gender-Science IAT from here. There is evidence that these unconscious biases affect our behavior. For instance, one study of recommendation letters written for 300 applicants (looking only at the ones who were eventually hired) found that, when writing about men, letter-writers were more likely to highlight the applicant’s research and technical skills, while when writing about women, letter-writers were more likely to mention the applicant’s teaching and interpersonal skills.

• There is a study of postdoctoral funding applications in Sweden, which found that women needed to be about 2.5 times as productive (in terms of papers published) as men, to be ranked equivalently. Other studies have suggested that the Swedish experience may be an anomaly. (For instance, one meta-analysis I saw estimated that, on average, it appears men win about 7% more grant applications than women, but since this is not controlled according to the objective quality of the application, it does not necessarily imply the presence of gender bias in reviewing of grant applications.)

• This study reports experience from an ecology journal that switched from non-blind to blind reviewing. After the switch, they found a significant (~8%) increase in the acceptance rate for female-first-authored submissions. To put it another way, they saw a 33% increase in the fraction of published papers whose first author is female (28% -> 37%). Keep in mind that this is not a controlled experiment, so it proves correlation but not causation, and there appears to be controversy in the literature about the work. So it as at most a plausibility result that gender bias could be present in the sciences, but far from definitive.

Snodgrass’ studies includes some of these, and more.