Quality Control of IEEE Conferences: Ensuring Excellence and Integrity

IEEE China Mini Panel of Conference Organizers (POCO) 19 September 2015

Bin Zhao IEEE Technical Program Integrity Committee (TPIC)

Agenda

9/19/2015 2

IEEE Conferences

Issues in Conference Quality

Quality Control and Assurance

Recommended Practice

Q & A

• R10/China is very important to IEEE from many perspectives • IEEE wants to be the favorite place for R10/China’s

Electronic and Electrical Engineering Community: • to hold conferences or conventions • to publish technical papers

IEEE Conferences

9/19/2015 3

Bright minds share the latest research at IEEE sponsored and co-sponsored conferences around the world.

1,600 Annual Conferences

Research Collaboration Publications

IEEE Global Conference Overview In 2014 IEEE….

9/19/2015 4

With Universities and other not-for-profit organizations

Partnered with over 1000 non-IEEE entities

Sponsored 1635 conferences

805 financially sponsored

technically co-sponsored 830

in 90+ countries

Over 480,000 attendees

266K Financially sponsored conferences

Technically co-sponsored conferences 215K

Acquired 1510 conference proceedings

Conference Quality Issues

9/19/2015 5

“Pay-to-Publish” behavior – most authors are no-shows

Paper Stuffing – 1000 papers from conference

w/100 attendees

Proceedings Hijacking – unauthorized people

substituting content

Citation Stacking – numerous self-references in a

single paper

Sponsorship Shopping – mainly in TCS

Machine-generated papers

Content outside of conference scope or Organization technical scope

Plagiarized papers

Poor English / grammar

Potential Consequences

9/19/2015 6

Loss of IEEE’s Reputation Loss of IEEE Conference Publications’ Ability to be Indexed

9/19/2015 7

The IEEE Brand Promise Advancing Technology for Humanity

Maintain IEEE leadership as the trusted source of advancements and applications of technology

Create standards, innovative research venues, and a global technology forum

Provide opportunities to conquer technical challenges with imagination

Enable today’s innovators to improve tomorrow’s quality of life, as our tagline promises

9/19/2015 8

The IEEE – Who We Serve Global Reach

Academia, government and corporate organizations around the world rely on IEEE publications via our online database: the IEEE Xplore Digital Library. – Search >3,650,000 items

– IEEE is the most cited publisher in new technology patents

Root Causes and Driving Forces

9/19/2015 9

Need for Authors to Publish – Major force in parts of Asia

Monetary Profit – $200k revenue per pay-to-publish conference

Different Cultural Viewpoints

Inexperienced Conference Organizers or IEEE OUs

Decentralized, Trust-Based Model in IEEE

Problems mostly associated with “hotspots” – Confluence of needy authors, predatory organizers, and

naïve IEEE organizational units (OUs) – Mostly occurs in areas of the world experiencing rapid

growth and in TCS conferences

Corrective Actions

9/19/2015 10 Result of Cross-IEEE Partnership

New conference approval policies

Joint Volunteer / Staff Conference Quality Team – TPIC: Technical Program Integrity Committee

Dedicated support staff – Tracking, Analyzing and Following Up on Complaints

– Screening 10% of Incoming Xplore® Content

Communications to change behavior

Education program developed and launched

Operation Screening

9/19/2015 11

Issues and shortcomings in conference organization – Insufficient engagement by co-sponsoring IEEE OUs

– Inadequate resources dedicated to peer review

– Implication of a pay-to-publish/no-show business model

– False advertising and misrepresentation

– Partnering with co-sponsors whose non-profit status is questionable

– Very short time span between conference initialization request and the actual event sometimes four to six months or less normal practice is one to two years

Content Screening

9/19/2015 12

Conference content scope and quality issues – Papers well outside of the stated subject matter scope of the

conference

– Papers well outside of IEEE’s engineering/technology related subject areas

– Papers that are poorly or unintelligibly written

– Fraudulent content, e.g., machine-generated papers

– Papers that lack some or all of the basic elements commonly found in scientific research papers: Introductory question or hypothesis -> brief summary of related

previous scholarly work by others -> research methods -> results -> conclusions

Decision Process

9/19/2015 13

Third-Party Company Evaluates Every Paper from Conference, Searching for Suspect Papers – Out of scope – Poor writing quality

Volunteers Screen Subset of Papers, Then Decide – Check third-party evaluation – Examine other factors (review process, program, etc.)

Accept / Reject Entire Conference – Evaluating the Conference, not Individual Papers

Scope Definition

9/19/2015 14

Would This Paper Be Presented at an Seminar in an EECS Department at a Major University?

Aerospace

Bioengineering

Communication, Networking & Broadcasting

Components, Circuits, Devices & Systems

Computing & Processing (hardware/software)

Engineered Materials, Dielectrics & Plasma

Engineering Profession

Fields, Waves & Electromagnetics

General Topics for Engineers (Math, Science & Engineering)

Geoscience

Nuclear Engineering

Photonics & Electo-Optics

Power, Energy Industry Applications

Robotics & Control Systems

Signal Processing & Analysis

Transportation

IEEE Covers Many Areas of Technology

15 9/19/2015

RENEWABLE ENERGY SMART GRID

AEROSPACE COMMUNICATIONS

OPTICS

INFORMATION TECHNOLOGY IMAGING SEMICONDUCTORS

CIRCUITS BIOMEDICAL ENGINEERING

LTE WIRELESS BROADBAND ELECTRONICS

NANOTECHNOLOGY

CLOUD COMPUTING

More areas than just Electrical Engineering & Computer Science

Writing and Content Quality

9/19/2015 16

Main Criteria – Does the writing quality interfere with the ability to understand the technical content? – We do not expect or require perfect English!

Fraudulent content – Plagiarism – Machine-generated papers

Also Look for Basic Scientific Paper Content – Hypothesis – Previous work in the field – Methods / Apparatus – Results / Data – Discussion – Does the Data support the Hypothesis? – Conclusions

Current TPIC/Quality Focus

9/19/2015 17

Plagiarism Checking – Being required for all conference papers – Software will be free if IEEE holds copyright – Conference organizers must evaluate output

Continue Refine Scope Definition – Engineering Management – Life Sciences / Bio-Engineering – Materials Science

Online Training Modules – Make training easy – Modules in Chinese

Closely Engage with Sponsoring OUs – Well-defined approval & oversight processes

Key Factors Influencing Conference Quality

Sponsorship

Technical Program

Paper Quality

18 9/19/2015

TPIC/Conference Quality team are no longer solely in reactive/enforcement mode; we are in proactive/outreach mode

9/19/2015 19

Sponsorship & Engagement

Goal: Run High Quality Conferences!

Vet Conferences Carefully Before Granting TCS

Don’t Grant TCS and Then Ignore

Require and Maintain “Direct and Substantial Involvement”

Pay Attention to Inquiries from MCE Staff

Communicate with Your Chapters

Don’t Promise that Content will be in Xplore®

Report Suspicious Activity to tpii@ieee.org

9/19/2015 20

High Quality Technical Program

The portion of the conference that contains the presentation and discussion of research, generally in the form of: – Oral Presentations, Poster Sessions, Workshops and Tutorials Technical papers are collected and published as the proceedings of the event, and are often accessed in a digital repository, such as IEEE Xplore – Peer review required for publication of conference proceedings in IEEE Xplore Effective Technical Program Development – see a reference: https://www.ieee.org/conferences_events/conferences/organizers/planning_technical_program.html

9/19/2015 21

Characteristics in Technical Program of High-Quality

Technical program flow and content – Technical scope determined tracks and sessions – Allocating appropriate time for presentation and interaction – Assigning appropriate space

Supported by high-quality peer review – Identifying and organizing reviewers – Establishing review guidelines – Indicating review process in Call for papers – Integrated approach in managing Paper submissions Reviews Final program development

9/19/2015 22

Conference Organization Timeline

Establish TP Committee

10-12 Months

Recruit Reviewers Plan Key Dates

9-10

Months

Call for Papers

6-8 Months

Abstract or full paper

submission deadline

4-6 Months

Peer Review Process

3-4 months

Notice of acceptances

sent out (final paper) to authors

10-12 weeks

Graphic by John Tracy

9/19/2015 23

Peer Review

– Number of reviews per paper, and papers per reviewer We suggest 3 reviews per paper (2 is an absolute minimum) A maximum of 8 to 10 papers per reviewer

– Abstract or extended abstract vs. full papers (or two-

step approach) We recommend a deep-dive review of an extended abstract,

followed by a later, shorter review of the full paper

– Open, single-blind, and double-blind review We recommend single-blind review (the reviewers are

unknown to the authors)

9/19/2015 24

Review Criteria

Scope – Determine whether the paper is in the conference’s

stated field(s) of interest (and IEEE’s)

Quality – Is the paper well-written? – If it is an original research paper, did the author follow

the traditional IMRAD model (introduction, methods, results, and discussion)?

Originality – Is the author proposing a new idea or a novel approach?

9/19/2015 25

Acceptance and Rejection

You can require that authors make certain changes to papers before they are accepted

Communicate to authors the results of the review process and any next steps – Provide link to PDF eXpress – Requirements for presentation at the conference – Final paper submission dates

Consideration to use pre-screening – Improved possibility to pass quality check although it is

not guaranteed – Understand the issues and root causes more specifically

for future events and taking corrective actions

9/19/2015 26

Screening for Plagiarism: An Introduction to CrossCheck

CrossCheck compares submitted manuscripts to a database of published articles (including content behind publishers’ paywalls)

Crosscheck identifies papers that surpass a predetermined threshold of similarity to previously-published papers

IEEE recommends a 30% similarity threshold (i.e., any submitted paper with 30% or more similarity to previously published content will be flagged for a visual, side-by-side review with similar, previously-published papers)

9/19/2015 27

Using CrossCheck

Conference Technical Program Chairs can register for CrossCheck through the CrossCheck Portal ohttps://crosscheck.ieee.org/crosscheck/

Several peer review tools used widely within the IEEE conferences community are integrated with CrossCheck

9/19/2015 28

Challenges in Peer Review

Almost every conference faces these challenges: – Gathering enough high-quality papers by the submission

deadline – Finding enough experts to complete their reviews by the

deadline

Good conferences attract good papers and good reviewers!

9/19/2015 29

Non-Presented Papers “No-shows”

IEEE empowers conference organizers to limit the distribution of any paper that was not presented in person at the conference – Authors unable to attend the conference should arrange for a

qualified substitute – Organizers must notify authors in the conference call for papers

if they choose to implement IEEE’s non-presented papers policy

If a paper is included in the proceedings distributed on site and is later identified as a non-presented paper, the paper should still be included in the proceedings delivered to IEEE – Flag the paper when generating the Packing List so the paper

can be archived in our internal content management system

9/19/2015 30

Conference Quality Opportunities

– Main quality risks:

Papers outside the stated subject matter scope of the conference and IEEE’s fields of interest

Papers that are poorly or unintelligibly written

Content that indicates author(s) wrongdoing, e.g., machine-generated papers, plagiarism, excessive self-citations

– Serious engagement needed by co-sponsoring IEEE OUs

– Appropriate resources need to be dedicated to peer review

– Beware pay-to-publish business models

The IEEE Technical Program Integrity Committee (TPIC) and the IEEE Conference Quality team co-manage an ongoing quality assurance process

9/19/2015 31

Working Together with A Common Goal – Working closely with Conference Organizer Engagement and

Education team to develop online and in-person training materials (e.g., peer review best practices, stringent criteria for IEEE co-sponsorship)

– IEEE volunteers and staff continue to help conference organizers and leaders of IEEE organizational units whose conferences have been compromised by poor quality or inappropriate content

– Communication is the key to our success: we are engaging with our community to identify, develop and refine best practices

TPIC and Conference Quality team want to work with conference organizers / IEEE OUs to improve quality, maintain the health of Xplore, boost technical activities

9/19/2015 32

Q & A