Peer Review Panel Functions and Organizations

(PDF version)

October 1994

Prepared by:
SG Associates, Inc.

Prepared for:
U.S. Department of Transportation
Federal Transit Administration
Federal Highway Administration
Office of the Secretary
U.S. Environmental Protection Agency

Table of Contents

The procedures used by transportation planning agencies to forecast future travel demand and the split of demand among drive alone, ridesharing and public transit are quite complex. Over the past thirty years, a basic process has been developed that involves four major elements.

  1. How many trips will be made? (Trip generation)
  2. What destinations will be selected by people living in specific areas? (Trip distribution)
  3. How will travel be divided between driving alone, ridesharing and public transit? (Mode choice)
  4. How many vehicles or people will want to use specific roadways or transit services? (Assignment)

Initially, the methods used to address each of these questions were relatively simple. Over time, as more research has been reported on personal travel behavior, new procedures were recommended. Advances in personal computer technology and reduced costs for computers has made it possible to implement many of the methods recommended by the research. As this has happened, the methodologies used to consider each of the questions have become more complex.

When an agency undertakes development of new travel forecasting methods, it must consider many questions related to the details of the methodology. The questions might include:

  • How should the population be stratified for estimating travel demand?
  • What factors are important in determining the number of trips to be made?
  • What factors affect an individual's choice of the destination for a trip? If both highway and transit are available, does this affect the choice of destination? If so, how are these travel opportunities represented?
  • When allocating travel to the available modes, what modes must be considered? Should different types of transit service be treated as different modes? Does the methodology need to differentiate between people who alk to transit and those who drive to transit?
  • What factors determine a traveler's choice of mode? How important is each factor?

Each of the questions listed above implies further questions relating to details of the computer programs and the techniques used to represent transportation facilities in the computer models. There are many possible pitfalls. Seemingly simple decisions made at the beginning of work to develop travel forecasting methods can make it difficult or impossible to answer questions that arise when specific investment actions are being considered.

Few individuals have had the opportunity to develop and apply more than one travel demand forecasting procedure. No individual can foresee all the issues that may arise in developing or applying a new model set. One approach to improving travel forecasting procedures has been the use of Peer Review Panels. These Panels, composed of individuals who have "hands-on" experience with both developing and applying travel forecasting models, assist local agency staff in both identifying possible problems and in developing workable solutions.

For a project involving development of new models, a Panel will typically schedule a two or three day meeting that functions much like a seminar. Documentation of existing travel fore- casting procedures and of available data sources are distributed to the Panel in advance of the meeting, permitting the Panel to read the items and become familiar with the methods. At the meeting, each aspect of the travel forecasting procedures is discussed in detail. Deficiencies, if any, in existing procedures are noted and suggestions for improving the methods are made. The objective is to assure that travel forecasting methods will be able to answer important questions and that the methods are consistent with the general "state-of-the-art." At the end of the Panel meeting, documentation is prepared of specific recommended and desirable actions.

For a project involving the application of travel forecasting models to analysis of a pro-posed transportation investment, the role of the Panel is slightly different. In these situations, the Panel will provide comments on the application of the existing models to the issues under study, suggest areas in which additional analyses are required, provide review of basic assumptions (e.g. demographic forecasts, parking costs) and design of alternatives to be tested, and comment on the interim and final results.

In the past few years, Panels have been formed in at least a dozen metropolitan areas. Some Panels have been established as part of FTA's review of Alternatives Analysis studies; others have been assembled by the local transit agency or Metropolitan Planning Organization.

The issue in forming a Panel and in conducting panel review activities includes selecting the panel members, organizing panel activities and conducting panel meetings.

Selecting Panel Members

A typical Peer Review Panel consists of five or six members. A smaller group limits Panel interaction while a larger group may be unwieldy. The individuals selected to serve on a panel should have experience and credibility in the specific topics under review. If the topic is travel demand forecasting, the panel members should have demonstrated expertise and experience in both the theoretical and practical aspects of developing and applying travel forecasting models. It is not necessary that all panel members be expert in both theoretical and practical issues – a mix of researchers and practitioners is adequate – but all should have some appreciation of both theory and practice.

Persons selected as panel participants will typically be drawn from other planning organizations, academic institutions, and consulting firms. Individuals from planning organizations and academic institutions will typically have no conflicts-of-interest. When selecting individual consultants or persons affiliated with consulting firms, it may be necessary to take care to avoid real or perceived conflicts.

If the panel will be reviewing travel forecasting model development or application work being performed by a consulting firm or if the panel will be making specific suggestions on travel forecasting work that is likely to be the subject of-a request-forproposals, potential conflicts must be avoided. Individuals selected should not be affiliated with consulting firms that were in competition for any on-going work for the sponsoring agency and should be precluded from bidding on any work that might result from the panel's activities.

Organizing the Panel

Organization of the panel may be undertaken directly by the sponsoring agency or the agency may choose to contract with an individual or firm. The activities related to organization include:

  • Identifying the scope for panel activities
  • Identifying possible panel members
  • Obtaining agreements to participate
  • Establishing panel meeting dates
  • Preparing and distributing pre-meeting materials
  • Arranging meeting logistics
  • Establishing the meeting agenda
  • Post-meeting activities including report preparation and payment of panel expenses.

Scope of Activities

Before assembly of a Peer Review Panel, it is necessary to determine the scope of the panel's activities. This includes both the nature of the technical issues to be covered and the extent of the panel's participation. These will be determined by the nature of the work planned by the agency.

Several examples illustrate how the scope and extent of panel activities can vary.

  • A metropolitan planning organization felt that its travel forecasting procedures were not consistent with the current state-of-the-practice. A panel was formed, composed of individuals with experience in travel forecasting, to review and critique the existing procedures, to suggest specific areas for improvement given the nature of the issues to be addressed by the agency, and to identify needed data collection activities. This panel met only once to conduct the review and prepare recommendations.

  • A metropolitan planning organization had embarked on a project to upgrade its travel forecasting procedures. The project was a joint effort of agency staff and a consultant. A panel was formed to assist the agency in guidance of the project and review of both interim and final products. This panel met several times including a meeting prior to start of the work and periodically during the course of the project.

  • A transit agency was conducting an AA/DEIS study of investment alternatives in a specific corridor. The studies were being performed by a consultant. To assist the agency in review of travel forecasts, a panel was formed. The scope of the panel's activity included the overall work program, development of alternatives, proposed travel forecasting methodologies, validation of the model set, and review of interim and final forecasts, This panel met several times throughout the project. Three or four meetings is typical for such an effort. A similar schedule would be appropriate for a Major Investment Study (MIS).

  • A transit agency was engaged in preliminary engineering studies for a proposed light rail facility. The studies were being performed by a consultant. The agency sought review, not only of the travel forecasts, but also of all aspects of the project including engineering, operations and finance. A multi-disciplinary panel was formed, composed of individuals with expertise in each of the subject areas. This panel was scheduled to meet three or four times over the course of a twenty-month effort.

Identifying Possible Panel Members

Panel members may be drawn from academic, governmental or consulting organizations and should be individuals with practical experience in the subject areas to be reviewed. Sources for identifying possible panel members include agency staff contacts or the membership lists of Transportation Research Board or American Public Transit Association committees. In addition, FTA can provide a list of individuals who have previously served on selected panels or have been selected for panel membership.

It is useful to strive for diversity of background and experience in establishing panel membership. This helps to generate broad consideration of issues.

Obtaining Agreement to Participate

The individual charged with assembling the panel should contact potential members either by letter or phone. The purpose of the panel, the panel's duration, approximate meeting dates, the expected time commitment and remuneration should be explained. At a minimum, panel members must be reimbursed for all expenses incurred due to panel participation. It is also desirable to pay panelists a small honorarium to compensate them for their efforts. For individuals in the private sector, any restrictions on bidding established to avoid conflicts-of-interest should be clearly explained.

An easy response mechanism (e.g. phone, fax, return mail form and envelope) should be provided for the potential panelists to indicate both willingness to participate and dates of availability.

Establishing Panel Meeting Dates

The individuals selected as panelists will likely have busy schedules related to the work of their respective agencies or institutions. Finding meeting dates convenient for both sponsoring agency and the panelists can be difficult. It will probably be necessary to query panelists about their availability with a lead time of at least six to eight weeks. In many cases, panelists have expressed a preference for either Monday-Tuesday or Thursday-Friday meetings. This permits panelists who must travel long distances to reduce their time away from regular work by traveling on a weekend day. It may also encourage some panelists to include a Saturday night stay, thereby reducing travel costs.

Preparing and Distributing Pre-Meeting Materials

Panel meetings are generally only a day or two in duration. To make most effective use of the time of both panelists and agency staff or consultants, it is essential to provide appropriate materials prior to the meeting. These would generally include a "charge to the panel" indicating what is expected to be accomplished at the meeting, background information describing the metropolitan area and/or the corridor under study (e.g. demographic and economic profiles, description of transportation systems, description of the current project; summaries of previous related studies and, for travel forecasting reviews, a detailed description of current and proposed methods). Depending on the status of available documentation, preparing or assembling these materials may take several weeks.

Panelists should also be provided with the name, location and phone number of the agency, the hotel at which they will be staying, the hotel reservation confirmation number, the meeting location, instructions for travel from the airport to the hotel and meeting location, and any necessary information on purchasing tickets, paying for the hotel and similar information.

These materials should be sent to panelists at least two weeks prior to the scheduled meeting.

Arranging Meeting Logistics

The smooth functioning of a Peer Review Panel requires attention to many details. These include:

  • Making hotel reservations for panelists — If possible, these should be at a hotel within walking distance of the meeting site. The sponsoring agency should seek to obtain the hotel's government rate for panelists. Reservations should be guaranteed for late arrival. Panels should be informed of hotel arrangements and their reservation confirmation number. The sponsoring agency should decide whether it wishes to be billed directly by the hotel or if it wishes panelists to pay hotel charges and obtain reimbursement from the agency.

  • Obtaining a meeting room — A comfortable well lit and well ventilated meeting room must be found with adequate seating for the expected attendance and appropriate audio-visual materials. Overhead projection, equipment, a screen and chalkboards are appropriate. If portions of the panel meetings are to be recorded, arrangements for equipment and a person to work the equipment must be made.

  • Designation of Attendees — Those attending and/or participating in panel meetings will include panelists and agency staff having responsibilities for work related to the panel's topic. If a consultant has been engaged by the agency, consultant staff will also be involved.

Depending on the nature of the panel's activities, the sponsoring agency may wish to invite representatives of other agencies with an interest in the project. These might include representatives of the MPO; transit planning, management or operating agencies; city planning or engineering agencies; and state transportation agencies. Invitations may also be issued to FTA and FHWA.

The sponsoring agency may also wish to consider permitting attendance by members of the "interested public." This may be particularly useful if the topic is related to a "high visibility" project.

Establishing the Meeting Agenda

To make best use of staff and panel time during the meetings, it is recommended that an agenda be prepared establishing the topics to be discussed, the order of presentation and the approximate time to be allotted to each item. The agenda will depend on the nature of the review effort and whether this is the only panel meeting, the first of several panel meetings or a meeting of a continuing panel activity.

A first or only panel meeting would generally include:

  • Introductions and Purpose
  • Presentation of Existing Travel Forecasting Techniques
  • Results of Validation Activities
  • Data Sources for Proposed New Efforts
  • Panel Caucus
  • Panel Report

For an MIS or corridor study, the topics would include:

  • Description of Corridor
  • Current Conditions
  • Needs Identification
  • Proposed Travel Forecasting Methods
  • Plan for Model Development and Validation
  • Panel Caucus
  • Panel Report

A brief tour of the corridor to be studied should be planned so that panel members can better understand the planning issues. It may also be appropriate to seek the advice of the panel on the definition of alternatives to be studied.

For subsequent meetings of a continuing panel, the topics should include:

  • Work Plan Progress since previous meeting
  • Actions taken in response to previous panel comments

The schedule for all panel meetings should include adequate time for the panelists to caucus to discuss issues raised and to come to a consensus on concerns and recommended actions. It is suggested that the panel caucus be private to encourage the most unfettered discussion but, at the option of the panel, agency staff or others may be permitted to observe the caucus sessions. A panel report should be presented at the close of the meetings.

Post-meeting Activities

A brief written report on the panel meetings and the panel's findings or recommendations should be prepared. This may be done either by agency staff or by a designated panel member. If the meeting is one of a series, the topics for the next meetings and the expected meeting dates should be established.

Panel members should be provided with instructions for filing claims for reimbursement of expenses. As claims are filed, expense payments and honoraria should be processed.

A few agencies have considered asking the panel to prepare a definitive statement regarding the validity of the results of the travel forecasts prepared for a project. Such a statement might be thought of as "certification" of the results. Requesting such a statement should be avoided. The panel's participation, even if continued throughout a project, is relatively minor. The members of the panel will not have been directly involved in project activities and will lack knowledge of details of the technical work. Providing a formal certification would require judgments that a panelist may not be prepared to make and could even expose them to legal liability should the forecasts not be achieved. Statements as to the validity of forecast results should be made by the agency or firm responsible for the work.

Panel Meetings

To achieve maximum productivity, meetings of the Panel should be structured with an established agenda and allocations of time for each topic. In practice, some flexibility for dynamic schedule adjustment is required. A meeting chair or facilitator should be designated to guide the discussion and assure adherence to the agenda. This could be one of the panelists or a representative of the sponsoring agency.

The first portion of a meeting will likely be devoted to presentations by agency staff providing background on previous work or a status report on work since the prior panel meeting. Adequate time should be allocated to each topic to permit panelists to ask questions and to pursue lines of inquiry. After the first portion of the meeting is completed, the panel should be given time to review the presented information and to identify topics for additional discussion.

The second portion of the meeting should focus on the topics identified by the panel. At the conclusion of this portion, the panel should be allotted about two hours to caucus to prepare an initial set of panel comments and recommendations. These should be briefly presented to all participants and observers before the close of the meeting.

Panel Report

A report for the record of the panel meeting and the panel's findings and recommendations should be prepared. Preparation of these materials will likely be the responsibility of the meeting chair or facilitator. Draft materials must be circulated to panel members for review and comment before final copies are prepared.