Draft Public Participation Playbook v3

This document is the unedited draft collection of initial ideas for articulating best practices and performance metrics for public participation in government. It is not intended for distribution as a final product, but instead an in-process collaborative space to contribute ideas and refine approaches before further opening it for wider participation.

This is the third draft version publicly posted in the development of this new resource in one month, and we hope will help set a standard in transparency and responsiveness.

To see past comments, please view the first draft of the U.S. Public Participation Playbook . We will continue to incorporate previous comments as the playbook continues to take shape. This will include pairing it down to a more concise size.

WHAT IS A PLAYBOOK? We asked 30 federal managers how we could best package best practices and metrics for them to use to better build and evaluate their public participation programs. Those target users identified the U.S. Digital Services Playbook as an effective format, which this playbook will adopt.

To read more about the playbook, please read the initial announcement from the White House OpenGov blog , or the most recent announcement of this second draft from the Digitalgov blog .

WHAT IS NEEDED IN THIS DRAFT?

1. More " Case Studies and Resources " that reflect the practical application of each play, especially offline and non-federal case studies. The draft is currently heavy on digital perspectives due to the large number of digital managers volunteering for the project, but we hope to open it up more through our citizen contributors.
2. Suggestions on better structuring the plays. We understand that after the initial open brainstorming period, it makes sense to better categorize the checklists and restructure the flow of the recommendations. We look forward to working with you more on this before our initial formal release in early January.

Introduction

Public participation has always been at the center of government -- *we the people. *Public participation includes all the activities by which people's concerns, needs, interests, and values are incorporated into decisions and actions on public issues. The American people have many ways to participate in their government's decision-making. Whether they are commenting on a new regulation, helping to pick infrastructure projects in their neighborhood, advocating for the environment, participating in open meetings, or engaging in online dialogues, or in other ways, they make their voices heard. Public participation allows each of us to voice our perspective.

All of our agencies have the mandate to engage with the public in a meaningful manner, and there are numerous benefits for government agencies that directly engage with those they serve. This playbook provides examples from federal agencies that have been leaders in public participation. Each agency pursues public participation differently, depending on the program, resources, audience, missions and goals. Similarly, the outcomes and impacts of public participation will also be measured differently, with each agency establishing metrics and reporting capabilities that align with their definition of success.

Regardless of which participatory technique, strategy or measure was employed, we wanted to share valuable lessons from a variety of agencies. We hope it will inspire you to examine how you are already engaging with the public and give you ideas on how you might improve public participation within your agency. We have also included checklists (steps and questions to consider) and suggested metrics.

As you explore new ways of fostering public participation, we invite you to use this playbook. Use the parts which work for you, and if we've left something out, let us know so we can include it the next time around. We need your help to craft a resource that captures the best ways for the government to engage and collaborate with the public.

1. Clearly define and communicate your objectives

a. Introduction

Agencies understand the importance of meaningful public engagement, but the form that engagement takes will likely be different for each organization, depending on resources, audience and mission. Each agency must determine goals, and establish metrics and reporting capabilities that align with the definition of success for that organization. Simply stated, we want to empower the public to engage with agencies in order to influence government priorities.

b. Checklist

1. Identify what goal(s) you aim to achieve.
2. Develop a plan that includes a timeline, ramp up and draw down strategy.
3. Evaluate the organizational capacity you have for engaging and managing public participation.
4. Identify internal stakeholders and partners.
5. Identify external participant personas, partners and audiences.
6. Identify the platforms, channels, or paths potential participants can use.
7. Consider how will you manage unintended participation that can distract from objectives.
8. Identify initial metrics to evaluate success.
9. Create a process for reporting and sharing information.
10. Develop a communications plan for internal reporting and engagement with the public.
11. Evaluate and change strategy based on feedback and performance data.

c. Case Studies and Resources

1. The Environmental Protection Agency Public Participation Guide has an excellent discussion of situation assessments that can guide you through this play.
2. The Transit Cooperative Research Program has a synthesis of effective public participation strategies for transit . Chapter 3 discusses goal-setting, information exchange, and identifying "the public."
3. The Department of Transportation's Federal Highway Administration provides a Transportation Professional Capacity Building site with many resources and case studies that may help you.
4. ePolicyWorks is an initiative by the Department of Labor's Office of Disability Employment Policy. The tool is designed around a clear policy-making objective and addresses specific challenges in collaborative policymaking.
5. The National Archives process for developing regulations uses the Regulations.gov portal during the commenting period to consider public input and make relevant, appropriate revisions. Regulations.gov is a portal that simplifies finding, reviewing, and submitting comments on Rules and Proposed Rules that appear in the Federal Register.

d. Metrics to Clearly Define and Communicate Objectives

Metrics will vary according to the platform you use. Some broad examples include:

1. Surveys – responses should drive learning and improvement.
2. Web – Google Analytics or heat-mapping can show user engagement.
3. Email – click rates, bounce rates, unsubscribes, participation in a call-to-action.
4. Social Media – user engagement analytics, reach, changing user behavior, hashtag use.
5. Apps (native and mobile) – use statistics, are "transactions" being completed, is there input or feedback from the field (ie uploading multiple photos in citizen science app), are people actively using a native app (i.e., logging in multiple times).
6. Online Video Chats (i.e., hangouts) – participation rate, views
7. Discussion Threads (i.e., chat rooms) – number of participants, input from the public, up votes / down votes
8. Blog – viewership, comments, action audience takes next
9. Transactional – hard numbers related to online transaction completion
10. Rates of completion for public participation opportunities (e.g., Comment form submissions to federal agencies)
11. Rates of API participation and usage e.g., number of API implementations, queries received via API, successful submissions received via API)

2. Select appropriate design format for public participation

a. Introduction

Broad participation is fostered by offering responsive, accessible, intuitive, mobile- and user-friendly tools that can be rendered in multiple formats on a wide range of platforms.

b. Checklist

1. Identify critical and secondary requirements for program success.
2. Identify time, resources and organizational capacity for public participation.
3. Determine paths for participation and their requirements, such as accessibility, commenting and annotation.
4. Design for flexibility using current coding techniques such as HTML5 and CSS.
5. Incorporate accessible graphics, color and visual techniques when appropriate to improve engagement.
6. Use analytics and feedback to check the need to diversify delivery methods, such as mobile.
7. Consider mobile-friendly capabilities such as responsive web design and link to mobile friendly pages.
8. Consider communication channels like SMS, Interactive Voice Responses and Unstructured Supplementary Service Data for participants without wireless connectivity or access to the Internet.

c. Case Studies and Resources

1. The White House Engage website allows citizens to submit questions and comments, join online events, and engage with government via social media.
2. Streamlined Design and New Features - USDA.gov underwent a major redesign to improve user experience and usability. They used analytics and lessons learned from prior redesigns to determine popular content and user preferences.
3. NOAA Release Mako Mobile App was created for fisherman to report their releases of Shortfin Mako sharks while in the water. The app uses a device’s built-in GPS, when available, to fill in exact location coordinates.
4. The Victoria (Australia) Toolkit for Public Participation has a great selection of different PP formats.
5. The Center for Land Use Education provides a framework for tailoring a plan that fits local needs and capacities. In addition to the framework, there are tips, examples, and worksheets to help prepare a sufficiently detailed, yet practical plan for public participation.
6. Social Media Hub for USFWS encourages accessible social interaction to increase participation.

d. Metrics

1. Increased readership/visits to sites/blog/etc.
2. Increased awareness about the federal agency's goals, mission, purpose, or resources.
3. Measurement of public participation (survey responses, event attendance, other contributions i.e., time, ideas, etc.)
4. Measurement of time spent on site, bounce rate, clicking patterns and high-, low traffic areas

3. Understand and communicate the benefit of participation

a. Introduction

Participants must understand how they can contribute to a program and why it is important they participate. A clearly defined and communicated value exchange helps craft effective messages and inclusive engagement. Understanding what successful participation looks like is critical to knowing whether your program succeeds.

b. Checklist

1. Outline the entire participatory process and specify the current stage.
2. Define what participants need to know before they engage.
3. Provide inclusive access to orientation materials so all participants start off with the same information.
4. Analyze participant expectations, motivations and objectives.
5. Define how participants benefit from the process.
6. Define what participants contribute for participation, including information, permission and time.
7. Outline the results of participation from the participant's perspective.
8. Communicate to participants about how you used their input and how participation will impact the outcome.7
9. Inform participants when they should expect to hear follow-up from their engagement.
10. Follow up and show how you applied participation.
11. Build a communications strategy that considers expectations of participants and the value of participation.
12. Ensure involvement of senior leadership and understanding of participation's importance.
13. Include thanks to participants for their contributions.

c. Case Studies and Resources

1. National Day of Civic Hacking organizers submit problem statements to explain an issue, explain how the public can address it, and provide resources to help orient people to the problem. The event organizers for the National Day of Civic Hacking outline time commitments and skills needed to take part.
2. Regulation Room is a pilot project sponsored by the Federal government and operated by the Cornell e-Rulemaking Initiative. Organizers provide participants resources to better understand the rulemaking process. They also instruct how to make an effective comment, and opportunities to browse the issues most relevant to them. Organizers encourage discussions among participants. Participants comment on a report after discussion ends to ensure the program captures ideas before the government receives a final version.
3. Give a Minute is a micro-participation/crowdsourcing event with clear expectations for participants. Moderators collect and share ideas both digitally and in-person.
4. NASA Socials are events where NASA's social media followers can learn and share information about NASA's missions, people, and programs. Through these behind-the-scenes experiences, participants become committed and empowered to participate in and advocate for the agency's mission.

d. What metrics indicate if you understand and communicate the benefit of participation

1. Rates of conversion from contacts (electronic or other) to visitors to signups to contributions
2. Rates of returning attendees/visitors
3. Volume of sharing & promotion rates - recruitment activity from participants or other digital promotion.
4. Quality of participation - level of discussion (comments on ideas) and relevance of feedback.
5. Percentage of participants satisfied with the "customer experience."
6. Volume of new participants meaningfully engaged in the process

4. Understand your participants and stakeholder groups

a. Introduction

Audience understanding is key to running a successful engagement effort. Once you determine who you're trying to reach, you can refine your outreach efforts to effectively communicate with participants and stakeholders.

b. Checklist

1. Identify who you are trying to engage with though public participation.
2. Listen carefully to the experiences, beliefs and values of potential participants.
3. Create personas of your target participants to help understand and relate to their needs.
4. Identify personas of all who may participate, including why they engage.
5. Conduct sampling and initial outreach to better define and understand participants.
6. Use groupwork methods to draw representations based on observation and not assumption.
7. Identify obstacles you must overcome to reach participants, including connectivity and schedules.
8. Build into your strategy how you will work around obstacles to reach and engage your target audience.
9. Consider how will you ensure you’re reaching a diverse group of people with broad perspectives.
10. Consider whether there are other stakeholders who might be interested if presented with the information.
11. Identify gatekeepers including community groups and societies who can help reach potential participants.
12. Customize your engagement strategies for each type of stakeholder.
13. Plan for participation that evolves based on the life cycle of your program.
14. Meet your audience where they are, not where you want them to be.

c. Case Studies and Resources

1. Peace Corps Application Process Redesign : The Peace Corps redesigned their application process to meet the expectations of their target audiences. The application was shortened and applicants can now choose which countries they want to serve in. This receptivity to the audience's needs and experiences has resulted in record-breaking application numbers for Peace Corps.
2. The National Geospatial Intelligence Agency opened unclassified geospatial intelligence information to the general public through their Ebola Relief Website . They recognized an unprecedented need for a large number of stakeholders to quickly access information that can help NGO's and other workers battle the virus.
3. The CDC: Gateway to Health Communications & Social Marketing Practice provides resources to build health communication or social marketing programs. It provides tips for analyzing and segmenting an audience, choosing appropriate channels and tools, and evaluating the success of messages or campaigns.

d. Metrics for understanding participants and stakeholder groups

1. Total number of users who participated
2. Did you reach the users who are most likely to be the actual end user of the program or product?
3. Number of users in sub-groups identified
4. Compare findings among subgroups. Analyze for similarities and differences
5. Identify quantity of users needed to be a representative sample size (total needed broken out by number needed within subgroups)
6. Baseline testing could help track impact of project upon completion.

5. Design for inclusiveness

a. Introduction

Whenever you're running any sort of public engagement effort, make sure the design and setup don't exclude anyone from participating. Consider persons with disabilities who use screen readers, people with limited English proficiency or low literacy skills, people who have suffered a stroke and many others who may experience difficulty reading. This play will help ensure that your public participation effort is designed to include as many people as possible.

b. Checklist

1. Provide accessibility options for persons with disabilities, the aging population and others.
2. Evaluate the need for multilingual support, including Spanish language services.
3. Ensure Plain Language communication throughout the life cycle of the program.
4. Consider both online and offline support, including a physical version and digital package
5. Analyze if participants can engage without a facilitator.
6. Analyze if participants can contribute with 'group input' rather than individual sign-up.
7. Consider if participants can submit input from face-to-face workshops and other sources.
8. Provide cultural competency support that values diversity reflected in the participant personas.
9. Talk to underrepresented members of communities.
10. Recognize common dynamics when cultures interact and develop adaptations to service delivery.
11. Consider multiple learning styles.
12. Recruit volunteers for visual notetaking, or graphic facilitation online or in meetings.
13. Ensure you provide contact information to official support and accessibility teams.
14. Test your content for accessibility at all stages of development.

c. Case Studies and Resources

1. Food and Drug Administration: Using Medicines Wisely: Increasing Access to Information for Women with Intellectual Disabilities
2. FDA's Office of Women's Health teamed up with the Administration for Community Living, the Association of University Centers on Disability, and the Administration on Intellectual and Developmental Disabilities to improve their safe medication materials for women. They conducted a series of focus groups with women in the disability community to learn how word choices, font, layout, and graphics could be improved.
3. The Digital Communications Division (DCD) in HHS has a 508 Program that leads the development and review of HHS Web content, social media, and supporting technologies. They also assembled a 508 Accessibility Lab with a collection of assistive and evaluative technologies.
4. Disability.Gov is the U.S. federal government website for information on disability programs and services nationwide and is a model for accessibility on the web.
5. Guide to Section 508 Standards . The U.S. Access Board’s Section 508 Standards apply to electronic and information technology procured by the federal government, including computer hardware and software, websites, phone systems, and copiers. They were issued under section 508 of the Rehabilitation Act which requires access for both members of the public and federal employees to such technologies when developed, procured, maintained, or used by federal agencies.
6. Usability.Gov contains resources for agencies to evaluate their websites for both usability and accessibility.
7. Improving the Accessibility of Social Media in Government toolkit curates and share best practices to help agencies ensure their social media content is accessible everyone, including users with disabilities. Efforts are also being made to work with social media platform and tool developers, citizens and partners to encourage greater accessibility.

d. Metrics

1. View the W3C Web Accessibility Metrics , a detailed report on useful metrics to assess the accessibility level of websites, including the accessibility level of individual websites, or even large-scale surveys of the accessibility of many websites, including:
2. Rate of pictures without an Alt attribute.
3. Rate of Level A and AA success criteria violations.
4. The number of possible failure points where accessibility issues can potentially happen (such as the number of images in a page).
5. Severity of an accessibility barrier.
6. Time taken to conduct a task.
7. View Accessibility in Practice: A Process-driven Approach to Accessibility . The best approach to accessible user experience is to integrate accessibility into the design and development process.
8. Grade your program with the Section 508 Checklist . Pass/Fail criteria for each of the Section 508 standards (e.g., a text equivalent for every non-test element, functional electronic forms).

6. Provide effective and timely notifications

a. Introduction

Effective communication with participants and stakeholders throughout the life of an engagement effort is key. This play offers tips and advice for letting your audience know about opportunities to participate, and how to keep them engaged and active for the duration of the event.

b. Checklist

1. Develop a comprehensive outreach plan that includes timelines to reach different potential participants.
2. Consider a variety of channels such as the Federal Register, SMS, postal, email, flyers, and social media.
3. Provide a toolkit of materials, including an outline of timelines.
4. Deliver consistent updates and repeat them across multiple platforms.
5. Thank participants for their contributions throughout the program's timeline.

c. Case Studies and Resources

1. USGS and Employee Use of Social Media : The Union of Concerned Scientists released a scorecard on how agencies allow employees to communicate with the public. USGS received a B+, but updated its public documentation to address suggested improvements within four hours .
2. FAFSA's #AskFAFSA campaign sets up monthly office hours for stakeholders to ask questions about student aid via Twitter. This is often faster than asking questions via telephone, and anyone following the hashtag can benefit from the answers.
3. DigitalGov Search seeks to provide immediate answers to the public’s search questions. In addition to Federal Register documents, it also incorporates results from other specialized government websites and social media accounts.
4. U.S. Immigration and Customs Enforcement’s Operation Predator App allows anyone to submit anonymous tips about suspected predators to the ICE Tip Line 24 hours a day, seven days a week.

d. Metrics

1. Measure overall participation in the event compared to previous efforts
2. Measure online engagement trends (e.g., comments and shares) over the entire length of the event. This should show your notifications are improving.
3. Measure the positive and negative feedback, post-event, on how participation discussions will be used.
4. Measure engagement across each channel, independently, to determine the level of participation on each.

7. Encourage community development and outreach

a. Introduction

1. Community development and outreach involves communication amongst community stakeholders, organizations, research centers, government agencies, etc. Goals include communicating reliable information about federal programs and policies, including addressing complaints; obtaining public feedback about the impact of government activities; fostering conversations and incorporating feedback into the policymaking process; and strengthening the channels of communication between communities, the government, and public/ private sectors.

b. Checklist

1. Moderate and facilitate conversations to encourage connections and discussion of other peoples’ ideas.
2. Develop a plan to catalyze the discussion if engagement stalls.
3. "Share out" and let participants see how else they can contribute.
4. Consider regular reports to summarize contributions so people can see what others are saying.
5. Identify your programs' points of contacts for communities to access.
6. Ensure that individual employees, not just the listed points of contact, understand how they may engage with communities.
7. Determine and share the specific value the community organization will derive from participating.
8. Consider shared decision-making responsibilities for shared success, problem solving, and accountability.
9. Develop a dialogue process with clearly defined guidelines.
10. Prepare for variations in commitment to participation.

c. Case Studies and Resources

1. The Commodity Futures Trading Commission hosted a smart investing webinar in partnership with AARP . Both organizations reached out to their respective communities and encouraged people to sign up and participate.
2. HHS Office on Women's Health encouraged community organizations around the country to organize events in celebration of National Women's Health Week 2014. They used the platform meetup.com so organizations could add their own events, and participants could search for other events in their area.
3. FDA's Office of Women's Health launched the Pink Ribbon Sunday Mammography Awareness Program to educate African American and Hispanic women about early detection of breast cancer through mammography. By partnering with churches and community organizations, they reached over 100,000 women throughout the country.
4. US Department of State Bureau of Educational and Cultural Affairs Exchange Program fosters community engagement through the arts. The American Arts Incubator uses new media and/or mural arts as a means to engage youth, artists, and underserved communities in Asia. It works to address a local issues, such as women’s empowerment, HIV-AIDS prevention, social inclusion, conflict resolution, and the environment.
5. NASA's International Space Apps Challenge is an international collaboration focused on space exploration that takes place over 48 hours in cities around the world. Local communities create and lead events while the project is managed at the Agency level. City leads mobilize participation in their city, choosing challenges and models appropriate to local interests. This results in an unprecedented level of diversity in solutions.

d. Metrics: How we know we're encouraging community development and outreach

1. Number of community organizations that support this event compared to previous events
2. Size of community that your partners reach
3. Integration with the community

8. Empower participants through public/private partnerships

a. Introduction

Corporate and community partners can serve a powerful role in gaining support and participation for public service campaigns and events. Partners and sponsors can help reach new audiences or reinforce messages within existing ones. Public/private partnerships also legitimize both what your agency is trying to accomplish and how that information or program is delivered.

b. Checklist

1. Define what a partner is and how you create partnerships.
2. Use a Memorandum of Understanding (MOU), an Interagency Agency Agreement (IAA), or another type of contractual agreement to describe each partner's responsibilities. These agreements allow you to create an "authorized partners" list, which reduces legal issues and helps ensure a beneficial partnership.
3. Outline the terms and the specifications of the partnership, including messaging, logo usage, promotional opportunities and boundaries.
4. Reach out to nonprofit organizations and community partners who can amplify participation opportunities and connect your agency with partners.
5. Survey agency employees who may have knowledge of community and stakeholder groups.
6. Ensure that employees who want to interact with the public understand ethics rules and agency policies.
7. Recruit volunteers and find ways for organizations to help that will serve your program goals.

c. Case Studies and Resources

1. The Heart Truth ® campaign, created by the National Heart Lung and Blood Institute (NHLBI), part of the National Institutes of Health (NIH), has partnered with dozens of corporate and community sponsors since its launch in 2002. The campaign raises awareness about heart disease and its effect on women through both online and in-person fitness events.
2. In 2014 LabTV launched its vision of creating a free, scientist-to-student web/video platform aimed at inspiring the next generation of researchers. The National Institutes of Health (NIH) worked with LabTV to produce hundreds of videos of NIH researchers with varying backgrounds and interests, and matched students with scientists who can share information about life as a medical researcher.
3. STOP.THINK.CONNECT. ™ was created by a coalition of private companies, non-profit organizations, and the Department of Homeland Security (DHS), with leadership from the National Cybersecurity Alliance and the Anti-Phishing Working Group. By partnering with industry leaders, DHS attracts a broader audience and garners significant public participation for online events such as Twitter chats to help spread its message. In October 2014, DHS teamed with dozens of public and private organizations including Microsoft, McAfee, Norton, Intel and Visa to host a series of Twitter chats highlighting National Cybersecurity Awareness Month.
4. The U.S. Department of Transportation (DOT) Distraction.gov website is dedicated to stamping out distracted driving. DOT enlisted the help of several organizations to spread its message. One partner, the FOX television show "Glee", helped produce public service announcements that feature characters from the show. Citizens can also sign a pledge to be a safe and distraction-free driver. DOT also uses Distraction.gov to encourage individuals to use social media tools and provides instructions on how to host events dedicated to stopping distracted driving.

d. Metrics

1. Value and measure in-kind services donated by private partners, such as air-time, printing, advertising, or prizes.
2. Use clearly defined web metrics to identify participation levels when utilizing online tools such as social media or websites.
3. Compare participation from past events and activities that did not include partnerships to help measure the value of the partnerships.

9. Provide multi-tiered paths to participation

a. Introduction

1. Good government is responsive and engaging. To ensure broad participation and involve more citizens in the decision-making process, use a variety of channels to communicate. It's important for agencies to be inclusive, and make maximum use of available resources to receive feedback from a wide variety of sources.

b. Checklist

1. Determine the level of participation your program needs to succeed.
2. Consider if you are empowering stakeholders as decision makers and sharing information.
3. Determine if you have the resources to sustain a multi-tiered, two-way engagement strategy.
4. Define what input you are looking for, whose you need, and what channels they are most likely to use.
5. Ensure all employees are aware of opportunities for participation and know how to respond if approached.

c. Case Studies and Resources

1. Numerous agencies have used Thunderclap as an easy way to for the public to show support and join conversations around timely issues. Supporters sign up to share the same message, at the same time, using social media. The Department of Health and Human Services (HHS) used Thunderclap to support Open Enrollment Season for Healthcare.gov. The Environmental Protection Agency (EPA) promoted its campaigns for Earth Day, #ActOnClimate ! and Be SunWise on Don’t Fry Day .
2. The Army's Assembled Chemical Weapons Alternatives Program has established two citizens advisory boards which are empowered to make project recommendations. The program holds quarterly public meetings with program leadership and maintains an outreach office that's open to the public and fully staffed to receive in-person feedback. They also maintain social media accounts, direct email, and a website for maximum interaction.
3. T he U.S. Trade Representative has a traveling roadshow that gathers feedback on trade agreement negotiations from stakeholders: industry, small business, academia, labor unions, environmental groups, and consumer advocacy organizations.
4. The Food and Drug Administration has multiple levels of participation through its Patient Network site. Participants can comment on proposed regulations and guidance, attend or speak at public meetings , and join an advisory committee to influence decisions like product approvals.
5. The International Association for Public Participation publishes an IAP2 standards document that describes the various levels of public participation and examples of techniques.

d. Metrics

1. Number of public meetings held
2. Rate of attendance at meetings.
3. Number of comments received per channel.
4. Number of avenues available for feedback.
5. Subjective metric: Value of feedback, credibility of source.

10. Maintain responsive, accountable feedback loops

a. Introduction

If you ask your audience to take time and effort to share thoughts and feedback with you, make sure you let them know that their voice is being heard and how you will use their feedback. Acknowledge the good and the bad openly and transparently.

b. Checklist

1. Include time in your program to pause, evaluate and take stock to evolve with the program.
2. Celebrate effectiveness with participants when feedback is constructive.
3. Address critical feedback in a transparent and informative manner.
4. Engage in continuous conversations and outreach efforts with civic groups and communities.
5. Build relationships with interactions such as hosting on and offline 'meet an expert' sessions.
6. Foster interactions to help reveal unexpected factors and influences.
7. Don't let the conversation end. You've just invested a tremendous amount of time and energy into building relationships. Use the participation framework you created to develop future uses for the network.

c. Case Studies and Resources

1. The National Parks Service reversed an earlier decision and decided to excavate the James Dexter site due to engagement with Philadelphia's African American community.
2. New Bedford Whaling National Historical Park used ongoing " Partners in the Park " roundtable dialogues with the New Bedford community to implement the park's General Management Plan.
3. National Park Service Tribal and Public Consultation for Historic Preservation Toolkit
4. Also reference: Using Social Media to Enhance Civic Engagement in Federal Agencies

d. Metrics

1. Rate of response to individual and group feedback.
2. Volume of response cycles.
3. Rate of contributions evaluated for implementation.
4. Rate of contributions implemented.

11. Use data to inform decisions

a. Introduction

Data can help you decide the best format for your engagement effort, the best tools to use, the most receptive audiences and more. Data will also help you get the desired results and adapt your engagement effort along the way.

b. Checklist

1. Define your goals and establish key performance indicators.
2. Develop a hypothesis to measure.
3. Continuously reconsider these metrics to make sure they are still the best ones to use.
4. Revise the process if a program or technique does not add value.
5. Recruit employees with a thorough understanding of the data that your agency provides.
6. Ensure employees are aware of the agency's open data policies and publicly available datasets.
7. Encourage data-savvy employees to attend and assist with public events.

c. Case Studies and Resources

1. Govcode is a website that features government open source projects.
2. Public Participation in Govt Web Design : The Data.gov team integrated feedback from virtual, online, face to face testing, and social media platforms to dramatically change the site design in response to customer needs.
3. Web Design Changes? Let the Metrics be Your Guide , where NASA metrics showed that mobile use of NASA.gov was outpacing desktop use, confirming their decision to change to a mobile-first design.
4. Government CX: Finding the Metrics That Matter , a study in how the Export-Import Bank measures transaction processing times and ease of use to create the best experience for their stakeholders.
5. Data.gov is a rich resource for civic hackers, tech entrepreneurs, data scientists, and developers of all stripes. It includes information about APIs, open source projects, and relevant developer resources across government.
6. Enigma is amassing the largest collection of public data produced by governments, universities, companies, and organizations. Concentrating all of this data provides new insights into economies, companies, places and individuals.
7. API Release Kit describes the elements agencies should include in federal API releases.

d. Metrics

(currently in development, but please feel free and encouraged to propose more)

12. Protect citizen privacy

a. Introduction

Public participation is crucial to our democracy, and agencies must engage in ways that protect privacy while maintaining compliance with federal laws such as the Federal Records Act (FRA) and the Privacy Act.

b. Checklist

1. Document privacy policies for all engagement channels so the public can see what information you collect, if any, and how you agency may use it.
2. Document terms of use detailing record-keeping practices and reasons why you may remove comments including profanity, sharing personal information and other rules to ensure protection of privacy.
3. Consult with your agency's Records Officer to determine how to manage interactions and public comments as records.
4. Invite your agency's Privacy Officer and/or legal experts to participate and provide input on your agency's privacy policies, terms of service, or other charter documents. Discuss how to handle personally identifiable information (PII).
5. Provide a medium for people to provide input anonymously.
6. Use indirect means like hashtags to join conversations online instead of targeting individual users.
7. Include provisions that allow community managers to delete content and comments that contain PII.
8. Include the Privacy Policy and terms of use on your website and social media platforms.
9. Identify what information your agency stores or maintains.
10. Identify whether your agency maintains records of conversations, and for how long.
11. Does your agency extend the same data privacy to U.S. citizens, U.S. residents, and to foreign nationals? Per a May 2014 report on Big Data , the White House may encourage agencies to extend universal Privacy Act protection.

c. Case Studies and Resources

1. CIO Council's Privacy Best Practices for Social Media : This comprehensive guide released in 2013 provides resources and best practices for standing up a Privacy and Records Keeping Act compliant social media policy.
2. White House Expands Data Protection : How can your agency use big data? There is no formal guidance yet, but this May 2014 report and recommendation from the White House provides insight on how and when agencies may be able to use big data from the Internet of things, the web, mobile, and more; and whether agencies will be required to extend universal Privacy Act Protections.
3. The FEMA Privacy & Comment Policy comprehensively addresses how social media, web, mobile, and SMS data is used, stored, and managed. The policy is plain-language and concisely explains and justifies the retention and management of different data sets.
4. The Peace Corps Privacy Act Notice is a plain language example of how to formulate a comprehensive, brief, and easy-to-understand privacy policy for the web.

d. Metrics

1. Percentage of websites and social media sites that contain provisions for protecting PII and maintaining Privacy Act compliance?
2. Frequency community managers monitor comments for PII.
3. Number of comments community managers delete for PII on a monthly or bi-monthly basis.
4. Rate of screenshots or other records that sufficiently protect PII captured by blurring, cropping, or blacking-out names, photos, or other PII.
5. Rate your agency "audits" its records to ensure it is not maintaining broader records than necessary to comply with federal records keeping requirements.
6. Rate of third party websites that collect data or PII in a way that is inconsistent with the Privacy Act.

13. Transparently report outcomes and performance of participation

a. Introduction

Transparency builds and maintains trust and respect with participants. When government takes an active role in allowing public participation in the decision-making process, accountability is key to maintaining trust of participants, by transparently reporting outcomes and performance of participation on a regular basis.

b. Checklist

1. Establish and communicate where, when and how you will report outcomes and performance.
2. Decide what type of open and accessible formats will be most useful for your audience.
3. Report outcomes on time while participants are still engaged.
4. Explain the importance of transparently reporting outcomes and performance of participation.
5. Encourage others to share findings and outcomes from your program.
6. Encourage continued community dialogue and awareness of lessons learned and outcomes.

c. Case Studies and Resources

1. The National Archives and Records Administration (NARA) used public feedback to develop the agency's third Open Government Plan. It also reported outcomes of NARA's previous Open Government Plans.
2. NASA - Disk Detective is a crowdsourcing project whose primary goal is to produce publishable scientific results. It uses citizen science to help astronomers discover embryonic planetary systems, reporting outcomes on time while participants are still engaged.
3. Ford's Theatre: Remembering Lincoln on HistoryPin - The HistoryPin platform encourages participants to contribute content, share findings and continue community dialogue. Ford's Theatre is working with a range of partner institutions to collect, digitize and share local responses from the 13 months after former U.S. President Abraham Lincoln's assassination.

d. Metrics

1. Frequency of performance evaluations.
2. Frequency of performance reports distributed publicly.
3. Volume of channels performance reports distributed on.
4. Rate of feedback on performance reports.