Needed Math (NM) is a three-year full-scale research and development project submitted to the Advanced Technological Education (ATE) program as Targeted Research on Technician Education. The aims of the project are (1) to identify contextual examples of important mathematical concepts, skills, and processes (mathematical competencies) that technicians in selected manufacturing sectors are expected to be able to apply; and (2) to develop and evaluate a replicable mechanism for industry to communicate with community college (CC) faculty about the mathematics needed.
The NM Project brings together researchers, employers, and technical and mathematics educators with exceptional credentials, NSF project management success, and a history of collaboration.
Dr. Marilyn Barger, email@example.com, (site visit and interview coordinator) is executive director of the Florida Advanced Technological Education Center. She has been co-PI and PI on NSF ATE project and center grants since 2000, focusing on curriculum development and credential alignment for CC and HS manufacturing programs across the country. She has worked with manufacturers to help define their needs for soft skills integrated into workforce education and participated in research on student motivation to enter ATE programs. She will help validate the NM survey, conduct interviews, and assist in organizing industry site visits.
Rosemary Brester, Rosemary@hobartmachined.com, (research team member and industry liaison) is president and CEO of Hobart Machined Products in WA. CC interns work with her employees to learn how math is applied in the workplace. She has a deep understanding of the tools and math processes used by technicians. She has worked on many projects in WA including the Transition Math Project, Association of Washington Business Education Committee, and the Governor’s Committee on Aerospace and Advanced Manufacturing. She will help organize industrial site visits, assist in validating the NM survey, and help coordinate the Governor’s Committee CWG in WA.
Dr. Sol Garfunkel, firstname.lastname@example.org, (research team member and lead math consultant) is executive director of COMAP in MA. He has been PI/co-PI on 17 NSF projects since the 1970s including ATE projects supporting students in technical fields, such as DevMap (NSF #9950036) and TechMap (NSF # 9818961). His efforts on the WorkMap Project included site visits to 15 IT workplaces to document the math being used by graduates of two-year degree programs. NM will draw upon the protocols he used. He was the lead author on the NCEE report (see p. 2) on the math needed to succeed in associate degree programs. He will help establish site visit and interview protocols, serve as a math expert on industry site visits, validate survey items, help write journal articles, and participate as a member of the CWG in MA.
Dr. Bernard S. Gorman, email@example.com, (co-PI and research lead) See Research Team, below.
Dr. Michael Hacker, firstname.lastname@example.org, (PI) has co-directed the Hofstra CSR since 1999, and was the Needed Math Conference PI. During 57 years in education, he was a secondary school teacher and administrator, teacher educator, and NYSED Supervisor for Technology Education. His research interests are investigating the mathematics needed to be successful in the workplace, broadening participation in STEM, and design-based technology and engineering education curriculum reform. He will manage the NM Project workflow and liaise with the CWG coordinators.
Dr. Deborah Hecht, DHecht@gc.cuny.edu, (external evaluator) is director of CUNY/CASE. She has over 30 years of experience evaluating large-scale NSF and other STEM education projects including the Needed Math Conference. She will lead the evaluation team that will include CASE doctoral students.
Dr. Paul Horwitz, email@example.com, (co-PI) is a senior scientist at the Concord Consortium in MA. A physicist by training, his expertise is in helping students use mental modeling to learn and apply scientific principles. He was co-PI on the Needed Math Conference Project. He will help coordinate the project, develop protocols, organize site visits, conduct interviews, and write journal articles and reports.
Lois Miceli, firstname.lastname@example.org, (Needed Math project administrator) had provided administrative support to numerous NSF projects conducted by the Hofstra Center for STEM Research. She will liaise with various Hofstra offices (grants administration, Webworks, fiscal management) to ensure smooth project flow.
Professor Rodney Null, Null.R@rhodesstate.edu, (co-PI) is professor emeritus of mathematics at Rhodes State College. He has been a secondary/postsecondary mathematics educator for over 30 years, an active member of the American Mathematical Association of Two-Year Colleges AMATYC since 1994, and a past president of the Ohio MATYC. He was the PI on the Mathematics Transitions in STEM Education project (DUE 1204849, 2012–2015) and was a program developer for numerous professional development (PD) initiatives to improve math instruction. He will help originate/review/validate survey scenarios, organize site visits, conduct interviews, serve as a research team math expert, write journal articles/reports, and coordinate the OH CWG.
Dr. Gerhard Salinger, email@example.com, (co-PI) became an NSF program officer in 1989 in a program to fund instructional materials development for K-12 science and math. His interests have been in materials and PD that support students in developing sophisticated understandings of STEM concepts and practices. In 1992, he was asked to establish and co-lead the ATE program where he set policy, wrote program solicitations, and recommended funding of regional and national centers. He retired in May 2014. Prior to his work at NSF, he was a professor and chair of the Physics Department at Rensselaer Polytechnic Institute. He will help coordinate the project, develop protocols and survey scenarios, organize site visits, conduct interviews, and write journal articles and reports.
Professor Gordon F. Snyder, Jr., firstname.lastname@example.org, (research team member and AB chair) is professor of engineering at Holyoke CC in MA. He brings 32 years of diverse academic program and curriculum development, college teaching, administrative, and leadership experience working directly with colleges, students, faculty, administrators, and industrialists at the local, regional, and national levels. He was the executive director of the National ATE Center for Information and Communications Technologies at Springfield Technical CC in MA and the former associate director of the National Center for Optics and Photonics Education. He will attend industrial site visits, chair the AB, organize and manage the co-PI, and coordinate the MA CWG.
Rita Wang, email@example.com, (GRA). See Research Team, below.
A prestigious AB representing all stakeholder groups, including experts on diffusing innovation, will provide ongoing conceptual direction to the project team. The AB will meet as a group to engage in discussion about the math required in the scenarios; reflect on progress, research, and evaluation; frame recommendations to the NM team; and help plan and facilitate dissemination. The AB will also meet in small focus groups of experts to address specific issues (e.g., scenario items, engaging persons with disabilities) as needed. Advisors include:
Dr. Kirk Adams, President and CEO, American Foundation for the Blind
Catherine Balas, Director of Employee Relations, Clark County Board of Commissioners and CEO, Balas Consulting Service, OH
Dr. Diane Briars, Chair, Conference Board of the Mathematical Sciences; past president, NCTM; former math supervisor, Pittsburgh, PA schools
Dr. James Dearing, Brandt Endowed Professor, Department of Communication, MSU
Amy Getz, Manager, Systems Implementation Team, Charles A. Dana Center, UT Austin
Jo-Anne Hongo, President, JS Hongo Consulting (former Genentech Senior Scientist), CA
Eric Isbister, CEO, GenMet Corporation, WI
Dr. Elaine Johnson, former Director of Bio-Link, the ATE National Center for Biotechnology
James Kerlin, CEO, Beyond Vision, WI
Kathryn Kozak, President, American Mathematical Association of Two-Year Colleges
Chris Shannon, Deputy Director, The Federal Reserve Bank of Boston, MA
Michael Tamasi, President and CEO, AccuRounds Corporation, MA
Michelle Younker, Chair, Department of Mathematics, Owens CC, OH
Employers, instructors of technical subjects, and mathematics educators who participated in a three-day conference on January 12-15, 2018, concluded that students’ mathematical competence should be strengthened by enhancing their ability to solve problems found in real-world contexts. That conference provided the impetus for the Needed Math Project. To view the conference proceedings, see: hofstra.edu/stem-research/needed-math-2018.html.
Dr. Bernard S. Gorman, firstname.lastname@example.org, (co-PI and research lead) is SUNY Distinguished Professor Emeritus of Psychology at Nassau CC and a professor in Hofstra University's graduate psychology program, where he teaches courses in multivariate statistics, qualitative analysis, statistical applications in psychology, and psychometric theory. He is an American Psychological Association Fellow and has written numerous articles and presented many conference papers in the areas of psychotherapy process, personality assessment, multivariate analysis, and relationships between cognition and affect. He is on the editorial boards of several refereed journals and a site visitor for the APA Commission on Accreditation. He will supervise our GRA, participate in site visits, liaise with SRI, oversee data analysis, and help write journal articles and annual reports.
Rita Wang, email@example.com, (graduate research assistant) is a doctoral candidate in Applied Organizational Psychology at Hofstra focusing on helping organizations succeed in change initiatives. Her research interests include organizational interventions and industrial psychology. She has worked to improve consulting firms’ organizational performance with data science and has been teaching statistics and research methods courses. She will help to develop interview and site visit protocols and will assist Dr. Gorman in analyzing survey results.
Our intended research outcome is to support industry partners of CC technical faculty to assume a more prominent role in determining the mathematical competencies required of manufacturing students. This is actionable research that will contribute to mathematics education reform. We will examine the following research questions:
RQ 1: What mathematical competencies do technicians in specific sectors of the manufacturing industry deem important to understand and use to be successful in the workplace?
RQ 2: What mathematical competencies do technical faculty in CC manufacturing programs deem important for students studying to be technicians in these sectors to understand and use to be successful in the workplace?
RQ 3: Where is there consensus and where are there significant differences between CC manufacturing educators and manufacturing technicians relative to the perceived importance of the mathematics competencies needed by technicians in specific sectors of the manufacturing industry?
RQ 4: Based upon NM research findings and use of industry-vetted math-rich manufacturing scenarios, what models for CWGs comprising CC technical educators, industrialists, and other stakeholders are effective in reformulating mathematics instruction in technical programs?
The project will conduct surveys of manufacturing industrialists and educators. Survey development and implementation will be managed by the Cornell Survey Research Institute which has expertise in all aspects of survey work: establishing objectives, identifying the study population and key concepts in operational terms, selecting methods of sampling, instrument design and testing, data collection, and analysis. The SRI will provide feedback; help with formatting, grouping, and wording; and pilot test the survey with industry technicians and CC technical faculty. Follow-up telephone interviews will be made to discuss items that are or are not working well.
Instead of using end-of-chapter textbook-type math problems as survey items, math competencies will be introduced in the context of industry-vetted scenarios. Survey scenarios will emphasize the math used in complex situations that technicians encounter. CC students may have been exposed to much of the math required by such scenarios, but as discrete topics, frequently in contrived contexts. Technicians need to choose and proficiently apply a variety of mathematical concepts and to evaluate results based on real-world constraints, standards, and experience. The scenarios will present contexts in which numerous mathematical concepts, often not explicitly spelled out, are required to perform a task. This is a skill set industry expects of technicians, but one which is not prevalent in their CC education.
We have selected survey validation panel members to reflect the expertise needed to review and refine survey scenarios and to establish its content validity. Selection criteria include having experience as either: a) a manufacturing industrialist; b) a manufacturing educator; c) a mathematics educator; or d) a survey research expert. Seven people have voluntarily agreed to assist and included the following individuals:
- Dr. Marilyn Barger, Executive Director, Florida Advanced Technological Education Center
- Dr. Solomon Garfunkel, Executive Director, Consortium on Mathematics and its Applications, MA
- John M. Mason, Manufacturing instructor, Ivy Tech Community College, IN
- Nathan Monroe, Toray Composite Materials America, Inc., WA
- Rod Null, Project co-PI, Mathematics Professor Emeritus, Rhodes State College (OH)
- Dr. Matthew Thompson, Toray Composite Materials America, Inc., AL
- Libby Simpson, Director of Education at FIRST, NH; former supervisor for Career and Technical Education, Hillsborough County Public Schools.
- Stephanie R. Slate, Cornell University Survey Research Institute, NY
Three sample scenarios (see below) were collaboratively produced by our industrialist partners and math educators. These are intended to illustrate what we will provide to survey respondents. We are not expecting the respondents to solve these problems; rather, on a five-point Likert scale, technicians will be asked, “How often do you encounter math-rich tasks similar to what is presented within the scenario?” Technical educators will be asked, “How often will your students encounter math-rich tasks similar to what is presented in this scenario once they are in the workplace?” Scale choices will include (1) never, (2) rarely, (3) occasionally, (4) often, (5) very often.
Sample Survey Scenarios
1. Produce 50 12-foot lengths of pipe from 6-inch-long sections of 2-inch NPT 316 SS schedule 40 pipe, threaded on both ends. The 6-inch sections must be joined by flanges which will be fastened by nuts and bolts. Using 6-inch Dia. 316 SS round bar stock, produce enough flanges with bored and threaded holes and gaskets to join the pipe sections. Machine to 5.750 OD, bore 2.188, and part off the blank. Consider the waste generated by the parting off tool. Leave provision for finishing to a final thickness of 0.625. Thread the 2.188 bore to 2-inch in the flange for a tight fit, with the 6-foot pipe not to extend beyond the end of flange face. Machine four through holes for 5/8-inch-18 UNF bolts on 4.750 PCD (pitch circle diameter). Deburr the parts, pass for inspection. The tolerance on all operations is +/-0.010 inches.
Needed math includes arithmetic, 2D and 3D geometry, measurement, number sense, percentages, polar coordinates, and fraction-to-decimal conversion.
2. Produce 25 one-liter containers of yeast extract peptone dextrose (YEPD) containing 1% yeast extract, 2% peptone, 2% dextrose. You are given a 20% solution of sterile filtered dextrose; and yeast extract and peptone as solids. Mix the yeast extract and peptone and bring the volume of each container to 900 mL with milli-Q water, autoclave, cool, and fill to the one-liter mark with the dextrose solution. How much yeast extract, peptone, and dextrose are needed in total?
Needed math includes algebra, arithmetic, percentages, and ratio and proportion.
3. Configure and install a wireless network that monitors machine performance in a manufacturing company. Wireless signal and noise strength are measured such that the network provides a S/N ratio that is at least 25 dB at each machine location. Signal and noise strength are measured in watts or volts using hand-held multimeters.
Needed math includes arithmetic, algebra and geometry (i.e., ratios, antenna position and location), logarithms, number sense (metric prefixes), units and conversions.
Industry Site Visits
NM team members trained in protocols by our research team will visit companies in different manufacturing sectors. Through these visits, we will acquire and validate survey items (scenarios) through interviews and observations. The site visits will enable our math educators to translate observations of technician activities into math competencies and will provide data to help generate survey items based upon actual workplace tasks. The visits will also enable us to create annotated examples of manufacturing technicians applying mathematics. We intend to meet with, observe, and interview supervisors, technicians with various levels of experience, recent graduates of technical programs, and interns. The translation of technician activities in highly contextualized and fluid workplaces into mathematics competencies is expected to be illuminating.
The following sample interview questions reflect some key questions that might be asked of each of the two NM groups. They will be revised, refined, and expanded during the conduct of the project. We will also include demographic questions to obtain background information about the respondents.
Examples of interview questions that might be asked of industrial employers or supervisors:
- In what manufacturing subsector do you work? (We will provide a list.)
- How many technicians do you supervise?
- What is the educational level of technicians that you hire? (We will provide categories.)
- As you see it, what kind(s) of mathematics preparation do you believe that technicians in your employ need?
- Which mathematical competencies that you require of your technicians have they already learned and which need further development?
Examples of interview questions that might be asked of technicians:
- In what manufacturing sector do you work? (We will provide a list.)
- What is your highest level of education? (We will provide categories.)
- How long have you been working as a technician?
- To what degree was your mathematics preparation sufficient for to do your job? Provide examples of the mathematics that you learned in school that are useful to you.
- What other mathematics competencies have you learned while working as a technician?
Examples of interview questions that might be asked of technician educators?
- In what manufacturing sector do you teach? (We will provide a list.)
- To what degree do you believe that your students are adequately prepared in terms of their mathematical background to be successful in the course(s) you teach?
- To what degree do you feel that your graduates generally possess the mathematics skills that they will need to do their jobs well?
- What kind of dialogue do you engage in with mathematics faculty at your institution?
- What kind of dialogue do you engage in with manufacturing industrialists?
Collaborative Working Groups (CWGs)
Many of our discussions have affirmed the Needed Math Conference summary finding that collaboration among employers, technical educators, math educators, and other stakeholders must be deliberately structured and ongoing. Thus, we will establish eight Collaborative Working Groups (CWGs) and study their functioning, effectiveness, and sustainability. The eight CWGs represent broad geographic, disciplinary, racial and ethnic diversity. A NM management team member will serve as a liaison to each group. Teleconferences will be held between CWG coordinators and NM liaisons to plan meeting agendas and discuss research findings. We will share data and provide guidance (but not straitjackets) for these groups, recognizing that a one-size-fits-all template would constrain their work. Groups that do not sunset prior to NM’s conclusion have agreed to continue to meet after the project ends as an ongoing mechanism to further industry/education collaboration.
As many technical educators have industry experience and often have math-intensive degrees in science, technology, or engineering, their backgrounds will spark meaningful dialogue with industrialists and mathematics educators. The CWGs will help to originate and review survey scenarios and serve as models of how industrialists can work with educators and other stakeholders to ensure that technical education produces technicians with the math skills and capabilities that industry requires.
We will facilitate interactions among three types of new or existing Collaborative Working Groups: (1) those established by this project; (2) those that build upon the efforts of a state-chartered committee; and (3) those involving existing ATE manufacturing centers and projects.
CWGs Established by the Project
Two CWGs will be established by NM—one in the Midwest (OH) and one in the Northeast (MA). These locations have been chosen because they have a strong advanced manufacturing base, have recently developed technical math curriculum, and have resident NM Project expertise. They will be coordinated by Rodney Null in OH and Gordon Snyder in MA. Null and Snyder were members of the Needed Math Conference Steering Committee and have established close working relationships with technical and math educators, CC administrators, manufacturing industry leaders, and labor representatives in their regions over many years of ATE work. Null is Professor Emeritus of mathematics at Rhodes State College. Snyder is a professor of engineering at Holyoke CC. These two CWGs will have the advantage of project management leadership and support for meetings.
CWG Building on a State-Chartered Committee
The Washington State Governor’s Aerospace and Advanced Manufacturing Pipeline Committee includes aerospace and advanced manufacturing industrialists, technical educators, subject matter experts, and leaders of professional associations, government, and organized labor. The committee was established by statute in 2012 to study the skills gap in the aerospace industry. Committee co-chairs were Needed Math Conference attendees and have agreed to bring industry’s perspective to bear on the math taught at the CC level in Washington, and to suggest and vet survey scenarios. This CWG will model how a committee established at a state executive level might operationalize NM research by issuing recommendations and by having its members carry them forward by serving on related committees and CC ABs.
CWGs Involving Existing ATE Centers and Projects
Five ATE manufacturing center and project PIs (from Central Virginia CC, VA; Gadsden State CC, AL; Ohlone College, CA; the National Center for Supply Chain Automation, CA; and the Support Center for Microsystems Education, NM) have agreed to engage their National Visiting Committees, project ABs, and/or departmental ABs in ongoing discussions to suggest how NM research findings can be applied in local settings to strengthen the alignment of mathematics in CC curriculum with industry needs.
Dissemination will take place throughout the project with emphasis on research findings and on recommendations from Collaborative Working Groups (CWGs). Established relationships with AB members, ATE centers and projects, and partner organizations including AMATYC, COMAP, the Dana Center, and the National Council of Teachers of Mathematics (NCTM) will enable dissemination to a broad range of stakeholders. AMATYC has suggested using NM results as the basis for themed sessions at annual conferences and for position papers.
All materials, including products, research findings, instrumentation, and reports, will be accessible on the Needed Math website, and through links on partner websites. Websites will be optimized for mobile access and socialization, including creation of an integrated blog. We will also establish accounts on Facebook, Instagram, LinkedIn, Reddit (r/matheducation, r/education), and Twitter; social media management tools will be used to monitor metrics, streamline communication, and maximize dissemination.
We will invite the CWGs to publicize their findings, and we will disseminate what we learn from them to the STEM community through presentations—developed jointly—at ATE-PI, HI-TEC, and mathematics conferences.
We will establish accounts on Facebook, Instagram, LinkedIn, Reddit (r/matheducation, r/education), and Twitter; social media management tools will be used to monitor metrics, streamline communication, and maximize dissemination.
Findings will be disseminated through submission of jointly written articles to refereed journals. Research results will be sent to all survey participants, salient professional organizations, ATE center and manufacturing project PIs, and state education departments.
We firmly believe that the issue we are proposing to research has far-ranging implications for the way math is taught to a broad segment of the student population; thus, interest in our findings is likely to extend beyond the world of mathematics education to encompass the public as well. Accordingly, toward the end of the project, we will distribute press releases to the news media to reach this broader audience.
A grantee-approved no-cost extension will enable us to convene a preconference NDW at the October 2024 ATE PI conference. Together with CWG member attendees, we will share research findings with 50 invited ATE leaders and request that they consider implementing CWG mathematics reform recommendations within their communities and spheres of influence.
Journal Articles and Papers
Other Tangible Results (e.g., legislative initiatives, training programs, curricular plans)
External evaluation will be led by Deborah Hecht, director of CASE at CUNY (see Biosketch in Section J). Hecht evaluated the Needed Math Conference and many of the PI’s projects. Evaluation efforts will build upon collaborative and trusting relationships, allowing the evaluation to have an impact on the work immediately.
Formative and summative evaluation will document and assess to what extent project goals and objectives are achieved as proposed and within the anticipated timeframe. Evaluation will also examine whether research is being carried out according to protocols and whether the conclusions are reasonable given the evidence collected. During the initial project period, the evaluation will be highly formative, providing the management team with information to help guide and enhance its work.
Formative evaluation questions: To what extent is the project meeting its outcomes according to the proposed timeline? How does project leadership interact with the Cornell SRI? To what extent are the different sources and partners helpful in developing scenarios? What is the project team learning from site visits and interviews, and how is that information being used? To what extent do the scenarios relate to the sectors of manufacturing and reflect different and/or similar math competencies? Are respondents able to look at specific problems and generalize to similar problems? What efforts were made to ensure an adequate response rate, and how successful were those efforts? Do the survey responses indicate that respondents understand the questions and tasks? Are the response rates adequate to do the analysis? How are the researchers using the survey responses? How does the team use feedback from the researchers, evaluator, and AB?
Summative evaluation questions: How are survey and interview results being shared with and used by the Collaborative Working Groups (CWGs)? What is the evidence that the scenarios are facilitating productive discussions between industrialists and educators during CWG or AB meetings? What changes have occurred or been suggested in reframing mathematics curriculum or instruction? What CWG models are effective in reformulating mathematics instruction in technical programs? How are the scenarios used to present mathematics in technical courses?
Data Collection and Results. Data to answer each of the evaluation questions will come from multiple sources: internal data collected by the research team (e.g., survey responses, case studies, open-ended questions), artifacts collected during project activities (e.g., work of the CWGs, dissemination activities), and data collected directly by the evaluator. A data plan has been drafted which identifies multiple data sources that will help answer each evaluation question. For example, technician and industrialist feedback surveys, document reviews, and interviews will be used to assess the degree to which respondents are able to look at specific problems and generalize to similar problems.
Yearly project team surveys, observations during team meetings, and formal/informal interviews will help assess how well the team works together and with the SRI. Surveys, interviews, and notes from meetings with research and management team members will be used to evaluate how the scenarios are developed, the usefulness of different sources, and the extent to which various manufacturing sectors and mathematics competencies are included. As the researchers use and interpret the survey data, the evaluator will provide feedback to help understand what was done and how conclusions were reached. Responses/response rates to surveys and participation in the site visits and CWGs will be examined and disaggregated to help assess adequacy, how representative the respondents are relative to those sampled, whether underrepresented groups are included as proposed, how groups’ results differ, and what factors or activities may be related to participation.
To assess how NM survey results are shared with and used by the CWGs and to compare differences across CWGs, the evaluator will collect feedback surveys from participants and coordinators, review materials produced, and conduct interviews. The evaluator will observe some CWGs and document what occurs. The ways the scenarios facilitate discussion and are used or planned to be used in math courses will be examined through surveys, interviews, and document reviews (meeting agendas and materials, suggested new course outlines, or syllabi, etc.).
Evaluation surveys will be collected from participants after project events (e.g., AB meetings, national dissemination workshop). Additionally, the evaluator will collect feedback from surveys and interviews of the project team to help assess progress, identify challenges, and track changes that might not otherwise be evident.
Results will be descriptive and empirical, with the goal of learning as much as possible and helping the project maximize what it can accomplish. When appropriate, qualitative evaluation data will be coded for evidence of themes related to the outcomes while also allowing new themes to emerge for further study. Qualitative data will also provide contextual information that can help during dissemination. Quantitative data will be aggregated and analyzed as appropriate. Tables will be created to depict and help interpret the data.
As the amount of data increases, the evaluation will synthesize evidence from different sources to draw conclusions about overall progress and to answer the evaluation questions. Reoccurring themes found within and across data sources will be identified. While one data source will provide evidence of specific impacts, confidence in the evaluation conclusions will increase with consistent findings across different sources.
Just-in-time formative feedback will be an essential part of the evaluation. The evaluation findings will be shared through both formal and informal updates, not only according to the feedback schedule, but also as data is collected and analyzed. Formal reports will be written annually as well as cumulatively. Throughout, timely feedback and recommendations will assure that formative findings are maximally useful for project enhancement and for guiding needed changes.