"Much that is taught in high school is not needed, much that is taught in middle school is not learned, and some topics that are needed are neither taught nor learned." — What Does It Really Mean to be College and Work Ready? A report from the National Center on Education and the Economy (NCEE), 2013
"When am I ever going to use this?"" — Common question from high school math students
NEEDED MATH CONFERENCE ABSTRACT
Employers, instructors of technical subjects, and mathematics educators who participated in a three-day conference on January 12-15, 2018, concluded that students’ mathematical competence should be strengthened by enhancing their ability to solve problems found in real-world contexts. The Needed Math Conference, supported by the Advanced Technological Education (ATE) program of the National Science Foundation (Grant number: 1737946), focused on bringing to light how mathematics education might better reflect the concepts and skills that are prerequisites for postsecondary education technical programs and for successful employment as entry-level technicians in STEM-related fields, as well as in many other occupations.
The mission of the National Science Foundation's ATE program is stated in its program solicitation:
With an emphasis on two-year colleges, the Advanced Technological Education (ATE) program focuses on the education of technicians for the high-technology fields that drive our nation's economy. The program involves partnerships between academic institutions and industry to promote improvement in the education of science and engineering technicians at the undergraduate and secondary school levels (NSF, 2018).
This goal is of vital importance nationally if the United States is to remain globally.
Adding contextual relevance can demonstrate the utility and beauty of the mathematics students learn. Students who have this experience of studying mathematics in real-world contexts will have a firmer basis on which to build more advanced concepts.
Many students are disenfranchised from mathematics because they do not see how the subject as they study it in school applies to their everyday lives and future careers. Although we might like all students to be drawn to study mathematics by its elegance, to appreciate its patterns, rigor, and structure, and to learn through its study to think abstractly and reason logically, a curriculum aimed at such lofty goals can result in convincing many students that they lack some innate ability and are thus doomed forever to be "bad at math." The insecurity that this engenders then becomes a self-fulfilling prophecy that endures into adulthood, and mathematics instruction becomes a barrier not only to further education but to career goals as well.
The current K-12 math curriculum and pedagogy all too often produce students who are ill-prepared for employment in the technical workplace. Employers report that, despite studying mathematics in every grade from kindergarten through high school and beyond, the graduates they hire often don’t know how to tackle the kinds of problems that typically arise in the workplace. For example, while the "renaissance" in manufacturing is a positive development for the economy, employers surveyed for the Manufacturing Institute’s 2015 Skills Gap study reported an obstacle: a "sizeable gap" between the talent they need and what is available in the job market. The respondents – over 450 manufacturing executives – listed math skills among the most serious deficiencies, along with technical and computer skills and problem-solving ability. An analysis of skills gaps in four industries including manufacturing found significant "foundational skills gaps" particularly in applied mathematics (Manufacturing Institute and Deloitte, 2015).
The Needed Math Conference, held on January 12 to 15, 2018, brought together employers in three STEM fields (biotechnology, manufacturing technology, and information and communication technology), postsecondary instructors of technical subjects related to those fields, and mathematics educators. There was consensus among them that there is a significant gap between the math that students are taught, tested on, and retain beyond school and the mathematical skills and abilities they need to solve problems commonly found in real-world contexts and that this likely contributes to the undersupply of skilled workers prepared for successful careers in STEM-related fields. Employers felt there will be a continuing need to revisit math curricula to ensure that they align with the needs of technicians in the 21st century workplace. Anecdotal evidence suggests that the inadequate mathematical preparation among high school graduates is reflected in a gap between the number of applicants for postsecondary training in STEM fields and the number of academic openings available.
The problem is complex, and the solutions to it are correspondingly varied, but the basic recommendation from the Conference is that the mathematics standards, assessments, and curriculum be revisited and revised so as to place greater emphasis on the skills needed to solve the kinds of problems that arise in the real world. Although the Conference participants represented only STEM fields, several of them noted that the kind of mathematical skills useful in those areas – e.g., mathematical modeling, statistical reasoning, and systems thinking – are valuable in many other career paths.
We recognize that implementing the recommended changes will be a complex and challenging undertaking, one that is likely to take years to accomplish and require engaging a great many diverse communities with varying interests and constituencies. Therefore, to support the effort, we recommend placing greater emphasis on contextualized math instruction. We urge an increased focus on topics that support teaching math in context and a corresponding shift in emphasis in the instruments that are used to assess learning. We argue that the current mathematics-for-all curriculum and assessment framework should be augmented by establishing a separate mathematically rigorous pathway based on realistic problems representative of those that many students will encounter after they leave school.
Continuing the dialogue among groups that appear disparate but are united in their greater goals is essential. In an ever-more connected world, there is little excuse for not creating a platform for such communication. This conference could be the seed that grows into a full, easy-to-use communication line among the various groups concerned with math curriculum, teaching/assessing, and utilization.
Needed Math Conference Overview
The Needed Math Conference brought together 50 individuals representing three groups: employers (from 10 states), community college and secondary school instructors of technical programs (from seven states), and math educators (from seven states). Three STEM domains were represented at the Conference: biotechnology, information and communication technology, and manufacturing technology. The Keynote Address by Dr. Solomon Garfunkel can be found in Appendix A.
The Conference was organized by its Steering Committee, comprised of ATE Center directors/co-PIs, a former NSF ATE program officer, industrialists, secondary school and community college teachers, mathematics educators, researchers, and education administrators. An external evaluator helped design the Conference logic model and examined the outcomes in relation to stated goals. Domain-based Steering Committee members extended personal invitations to employers of STEM technicians and to instructors who are actively engaged in improving education within their technical fields. The mathematics educators on the Steering Committee invited math educator colleagues who are active and respected members of their professional communities. Some companies even fully supported the attendance of their representatives financially, enabling the Conference to exceed its attendance limit. Appendix B lists the participants and Steering Committee members.
The Conference focus on needed math provoked great interest from all constituent groups. Small- and large-group discussions and debates were lively, given the diverse backgrounds of the participants, but the atmosphere remained entirely collegial, mutually respectful, and focused on outcomes. The Conference agenda and the structures used to “mix and match” the various groups — a "horizontal" grouping of math specialists, content instructors, and employers and a "vertical" grouping around disciplines — are described in Appendix C. This matrix strategy may well account for the congeniality and productiveness of the Conference, which is evident in the Findings section and leads directly to the recommendation on following up by maintaining the cross-disciplinary contacts.
Prior to the Conference, the employers invited were asked to submit examples of workplace problems involving math that they might use to test prospective employees. The 29 needed math problems submitted, which can be found at the end of these Proceedings, were distributed to all participants at the start of the Conference. The participants were then divided into three groups, one for each of the three domains, consisting of employers, mathematics educators, and STEM instructors. Each group reviewed all the problems for its domain using a data collection worksheet, presented in Appendix D, to guide them. The worksheet asked for specification of the problem to solve; the math, process skills, and competencies needed to solve the problem; and suggestions to assist teachers in conveying the needed math concepts in context. Using this common instrument, each group analyzed the examples relevant to its domain and used one or more of them to create a single scenario. The scenarios, which are described in Appendix E, are representative of realistic workplace math problems that are likely to arise in that domain. They are valuable in that they single out math skills that even students who have passed the required high school exams often fail to apply successfully. This can happen because the math that the students were taught did not cover some required skills or because the students have difficulty applying skills that they acquired in school when confronted with applying them in a work setting. A Conference subgroup of secondary school STEM educators suggested that Career and Technical Education (CTE)/STEM pathways should be designed to maximize student engagement using "real-world/industry" mathematics. Those recommendations are included in Appendix F.
The Missing Math Skills Problem
The missing math skills problem is not new, nor has it gone unnoticed. In the United Kingdom, for instance, the highly regarded Cockcroft “Mathematics Counts” study (Cockcroft, 1982) set out to identify the mathematical needs of various types of employment and found some important differences between the ways in which math is used in STEM employment and the ways in which the same math is encountered in the classroom. The Cockcroft report suggested that tests of arithmetical skill play a dominant role in schools, whereas conceptual skills such as spatial awareness, understanding of orders of magnitude, approximation, and optimization are of equal or greater importance in industry and receive far less attention in school.
In the U.S., the Curriculum Renewal Across the First Two Years (CRAFTY) project sought to clarify the rationale for the math requirements at two-year colleges and has published mathematics curriculum and instruction (MAA, (n.d.). Research done by the National Center on Education and the Economy (NCEE, 2013) examined the math needed for popular community college programs in multiple colleges in seven states and found that middle school math topics were most important to preparation for these programs. The study also found a misalignment between what was being taught in community college math departments and the math needed by the students to be successful in the applied programs they were pursuing.
Additionally, prior ATE-funded efforts have attempted to identify needed math skills (several members of the Steering Committee were leaders and/or participants in those projects). Among these efforts were WorkMap, developed by the Consortium for Mathematics and its Applications (COMAP), which identified the mathematical reasoning and skills employed in various work settings (Garfunkel, 2009) and the National Center for Optics and Photonics Education (OP-TEC) ATE Center’s review and study guides to help aspiring technicians begin their college studies with adequate math skills (OP-TEC, 2015). The Carnegie Foundation has funded two Pathways projects (Statway® and Quantway®) intended to "accelerate students’ progress through developmental mathematics by presenting engaging, relevant, and useful mathematics concepts that students can use in their daily lives" (Carnegie, 2010). The National Research Council’s The Mathematical Sciences in 2025 examined "the current state of the mathematical sciences and explores the changes needed for the discipline to be in a strong position and able to maximize its contribution to the nation in 2025" (NRC, 2013).
Barriers to Change
A clarion call was heard at the Needed Math Conference, especially from the employers, for teaching math in context using an approach that emphasizes its utility and stresses the application of mathematical models of real-world phenomena and processes from the mixing of solutions to the optimization of a wireless communication system. The question arises then, "Why are we not doing this already?" To understand the answer, we need a little history.
In 1989, the National Council of Teachers of Mathematics (NCTM) produced a set of standards that called for, among other things, the teaching of mathematics through practical applications (NCTM, 2000). Following this initiative, the National Science Foundation funded a series of comprehensive curriculum projects at the elementary, middle, and high school levels to create curricula that embodied these standards. Several of these curricula took a contextual approach. Their publication, however, created a reaction from some members of the mathematical research community, who were concerned, among other things, that talented math students would be disadvantaged, or held back, by the mathematics-for-all approach of the new texts and argued for more traditional approaches.
Despite some opposition, the new curricula enjoyed significant market penetration at the elementary and middle school levels, though much less at the high school level, and the more traditional textbooks continue to dominate in high schools to this day. Following the passage of the No Child Left Behind Act of 2001 (USDOE, 2001), every state was required to have a set of state math standards and to test every child every year in grades 3 through 8 and at least once in high school. This created a plethora of tests, curricula, and evaluation standards under which the same student might very well test "adequate" in one state but "deficient" in another.
The Common Core State Standards
The goal of the Common Core State Standards was to create one set of standards for all students across the nation, together with tests to measure student performance. An important concern was to get as many states as possible to accept the standards. While mostly traditional in content, they did mention modeling as one of eight overarching mathematical practices. Despite the fact that it was up to individual states to adopt them, the Common Core standards came be regarded as a "national math curriculum" and this engendered a negative perception in some quarters. However, for the foreseeable future, many states will probably retain the skeleton of the Common Core, and we optimistically believe that the importance of modeling and context-based problem solving will increasingly be recognized.
It should be noted that a fundamental difference between this round of reform and the last one, is that this one has not produced many new artifacts. To date, few new textbooks have been written that fully embody the Common Core, whereas in the 1990s, every grade level could avail itself of several newly reformed texts. The likely result is that when all is said and done, we will still mostly be teaching to new tests that look a lot like the old ones unless there is a concerted effort on behalf of educators, employers, and policy makers to reform mathematics instruction to include experiences for students in modeling solutions to real-world, work-specific scenarios.
Conference Findings and Recommendations
Even in geographic regions of significant unemployment, some STEM employers at the Conference stated that they often found it difficult to recruit employees due to their lack of mathematical skills. In addition, technical programs at many community colleges find that the supply of qualified entrants is inadequate to fill the available openings. There have been previous efforts (NCEE, 2013) to elucidate the reasons for these imbalances. The Needed Math Conference was unusual both in its focus on mathematics education and in its efforts to solicit input not only from math and content educators, but also from employers in technology-rich industrial sectors.
In this context, the most important takeaway from the Conference may be the extraordinary level of agreement among the participants. The mismatch between the mathematics that students learn in school and the mathematical requirements of an increasingly technological workplace can only be appreciated by a multidisciplinary community, hence the rationale for the Needed Math Conference. However, the decision to include three disparate groups – employers, program instructors, and math educators – carried with it the potential for discord. It was possible, for instance, that each group might blame the others for the perceived problem. Remarkably, this did not happen. Individuals who under normal circumstances might never have met each other engaged in spirited but respectful discussions over the two and a half days, and although their experiences and starting points were quite distinct, they reached consensus on several key issues. Most encouraging, there was general acceptance of a clear need for continuing cooperation between these groups to address the problem and lively interest in facilitating and enlarging the ongoing dialogue around the value and utility of mathematics in the 21st century workplace.
Finding 1. There is a gap between the typical textbook problem and the problems that arise on the job, even though the underlying mathematics may be the same.
The mathematics problems presented in textbooks and on tests have certain features in common. Being intended to teach and assess specific math skills, they usually involve only one such skill at a time and often make that skill explicit. All the problems presented at the end of a chapter dealing with multiplying fractions, for example, will call on that particular skill. In this circumstance, obtaining a passing grade certifies that a student can solve well-defined problems that evoke the application of specific algorithms. This does not prepare students to tackle the more open-ended and ill-defined problems they are likely to confront on the job. After much in-depth discussion, the conferees agreed that the discrepancy can be traced to the following characteristics of workplace problems:
- Workplace problems often involve unspecified constraints or tradeoffs.
- The problem as presented to the employee may include irrelevant information and may not include all the necessary information.
- The statement of the problem does not typically identify the mathematical concepts or skills required for its solution.
- Mathematically equivalent problems are often presented quite differently in industrial contexts, obscuring the underlying skills needed to solve them.
- The math skills or concepts needed may not have been taught to the employee or may have been taught a long time ago (e.g., in middle school) and never revisited.
- The problem may have more than one solution, and solving it may require choosing among them.
And to make matters worse:
- Arriving at an incorrectly thought-through solution to a workplace problem can have serious consequences, both financial and in terms of safety, for customers or for the workers themselves.
Finding 2: There is a gap between the mathematical preparation many students receive and the mathematical requirements of an increasingly technological workplace.
As Conference employers and other studies have corroborated, it is often difficult for businesses to find math-competent local employees. Some of the mathematics that is required for success in a STEM career is either not taught at all or not taught in a way that helps students retain it to be recalled for later use. These topics include the use of mathematics to model real-world phenomena and processes, statistical reasoning, and data analysis so that entry-level technicians are able to define a problem, find the supporting evidence needed to solve the problem, and define the expected goals and outcomes of the solution within constraints.
In addition, some employers expressed concern that since the traditional mathematics curriculum does not address topics relevant to their industry, it constrains the availability of qualified employees by erecting an artificial barrier to students' quest for the appropriate technical degree or certificate.
Finding 3. Too many students cannot make effective use of technology commonly found in the workplace.
Employers gave the use of spreadsheets as an example of a common technology, found in a broad array of enterprises, which entry-level technicians often cannot use. The mathematics they learn in school should prepare students to apply this ubiquitous tool to organize, display, analyze, and present data and to explore "what if?" questions using numerical models. A related skill, identified by many Conference participants is the ability to estimate mathematical results prior to performing a calculation. A common lament was that many job applicants "can't use a calculator," meaning not that they don't understand how to key in a calculation but that they fail to compare the answer to an estimation-based reality check.
Finding 4. As currently taught and assessed, math education has become a barrier to success for many students rather than the pathway to it.
While many students succeed under the present system and acquire a deep understanding of, and appreciation for, mathematics, the Needed Math Conference recognized the existence of a large number of students for whom math has decidedly negative associations.
The issues raised by the Conference are not new. Significant initiatives have been undertaken to address the problems, yet they persist. Ultimately, ensuring the mathematics preparedness and competency of STEM technicians is the shared responsibility of all stakeholders. To effect real change and improvement, each community has an important role to play, and collaboration among all stakeholders: employers, instructors, mathematics educators, and parents must be ongoing.
Recommendation 1. Place greater emphasis on contextualized math instruction at all grade levels and in all mathematics courses.
The rationale for teaching mathematics in context is two-fold: such instruction is vital for preparing students to solve practical real-world problems and it improves retention of the very mathematical concepts that constitute the learning goals of the current system of instruction. A voluminous body of research in mathematics education (COMAP, 2017; NCTM, 2000; Johnson, 2002; Larson, 2017; Olkun, 2003; Winkel, 1997, 2016) confirms that many students struggle to learn new abstract concepts because they seem unrelated to other bodies of knowledge with which they are acquainted. For those students, mathematics becomes understandable only when its applications are made explicit.
Recommendation 2. Make a shift in emphasis in the assessments that students must take.
This recommendation follows the one above, which calls for modest but important changes in the mathematics curriculum. Such changes cannot take place without corresponding changes in how mathematics achievement is assessed. Assessments should be driven by the open-ended, flexible, creative problem-solving that is encountered in the real world. They should include items that call for expertise in the areas of mathematics described in Recommendation 3. Assessments not grounded in authentic problems lead to an undervaluing of skills relevant to the workplace. It is not easy to score, much less to create, items that test complex problem-solving ability, but rising to this challenge is seen as central to effecting the changes that are needed.
Recommendation 3. Increase the focus on topics, approaches, and pedagogy that better reflect the demands of the contemporary workplace in all mathematics courses.
Due to the growing availability and importance of data, the conferees agreed that more attention should be paid to teaching topics, approaches, and pedagogy in mathematics that are central to analysis of data – e.g., probability, graphical representations, mathematical modeling, the ability to effectively communicate mathematically, and descriptive statistics.
Although the Conference focused on technical content, the mathematics content we are advocating for is useful across a broad variety of occupations. Data analysis, for instance, is an increasingly important skill for the social sciences, and errors in policy occur because policy makers do not properly apply statistical reasoning. Thus, teaching mathematics in the context of practical problem solving can not only prepare many more students for STEM careers, but also improve mathematics instructions for everyone.
Recommendation 4. Augment the current mathematics-for-all curriculum and assessment framework by establishing a separate rigorous pathway from the traditional trajectory, based on solving real-world problems representative of those that students may encounter after they leave school.
The current one-size-fits-all math curriculum fails students in two ways:
- By erecting artificial barriers to admission to college, it deprives many students of the opportunity to pursue higher education or entry into technical careers.
- By failing to emphasize those aspects of mathematics most relevant for the solving of real-world problems, it fails to prepare many students to succeed in an increasingly complex technological world.
The separate pathway would not be any less challenging than the traditional one, and we do not recommend "tracking" students from an early age. Instead, we envision a more democratic approach wherein a wider variety of math courses are available to secondary school students, who would then be free to make choices based on their interests and intended career paths. We might also envision offering a middle school math course that exposes students to both pathways, so they can use that experience to inform subsequent choices.
The new pathway would contextualize real-world applications of a more restricted body of mathematical knowledge. Thus, a modeling course would almost certainly include models of change over time but would not include a detailed exposition of derivatives and integrals as measures of such change. The calculus course, in contrast, would cover the chain rule for differentiation, but might stop short of implementing and exploring numerical models of change.
In support of this recommendation, a study undertaken by one of the employers at the Conference (Wallace, 2018) indicates that school personnel in New Mexico are very interested in finding an alternative math pathway for many students that does not require Algebra II and the Algebra II PARCC for high school graduation and would like to see project-based learning integrated into high school mathematics courses utilizing many real-world applications pertaining to business and industry.
Recommendation 5. Create a Needed Math Center charged with expanding the reach of the Conference findings to additional stakeholders by holding follow-on conferences on relevant topics and publishing articles aimed at a wide variety of audiences including policy makers at the state and federal levels.
An important aspect of the Conference was the creation of a multidisciplinary group of educators and employers focused on addressing what was universally perceived as a significant mismatch between the mathematical concepts that most students learn in school and the mathematical skills required for successful performance in the STEM workplace. However, the momentum and energy generated by a single event are unavoidably temporary and easily dissipated. This highlights the need for continuing to support the dialog between mathematics educators, instructors of technical subjects, and employers across a wide range of industrial sectors.
Accordingly, we recommend that the National Science Foundation, in concert with professional organizations such as the American Mathematical Association of Two-Year Colleges, establish a Needed Math Center focused on supporting and sustaining an active community of interest including business and industry representatives, state and local educational agencies, professional organizations, and representation from secondary and postsecondary mathematics programs to provide a forum for improving the mathematics preparedness of STEM technicians entering the workforce.
Among the issues that the center might address are the following:
- The center could examine the existing mathematics standards to determine the degree of overlap between the math that is taught and the math that is needed.
- The center could advocate for changes in the math that is taught, how it is taught, and how learning is assessed.
- Changes in content might include, for instance, an increased focus on common themes like mathematical modeling and reasoning under uncertainty.
- Changes in pedagogy could place more emphasis on contextual learning and instruction customized to students' needs and interests, rather than treating mathematics as a one-size-fits-all general education prerequisite to programmatic courses.
- The center could provide support for collaborations, perhaps in the form of periodic conferences between mathematics and programmatic faculties and in consultation with business and industries that hire graduates from their programs. to provide timely, relevant mathematics content in the degree programs.
- The preparation of a skilled 21st century workforce is of interest to groups beyond the education and business communities. The center could engage labor unions as well as federal, state, and regional entities concerned with planning for the changes initiated by technological innovation.
- In order to prepare mathematics teachers to offer appropriate contextualized instruction, the center might sponsor summer internships with selected local firms. This experience would not only inform their teaching, but also provide them with a persuasive counter to students' perception that they will never use the math they are learning.
- Through its connections with employers, unions, and professional associations, the center would be responsible for tracking changes in technology of relevance to the workplace and would work to ensure that the mathematics curriculum and professional development keep pace with such changes.
According to a report from the Policy Analysis for California Education:
Evolutions in various disciplines and in learning sciences are calling into question the relevance and utility of this [mathematics for all] trajectory as a requirement for all students. The emerging movement is toward differentiated "math pathways" with distinct trajectories tied to students’ goals. Alternatives emphasizing statistics, modeling, computer science, and quantitative reasoning that are cropping up in high schools and colleges are beginning to challenge the dominance of the familiar math sequence (Burdman, 2015).
We recognize that much excellent and well-intentioned work has been done over many years and has resulted in heartfelt and innovative approaches to mathematics education. We are hopeful that the work done by the Needed Math Conference participants can inform further reform efforts, as too many students continue to be disenfranchised by mathematics instruction that does not facilitate the development of a growth mindset but rather often discourages them from pursuing further mathematics study.
By emphasizing mathematical modeling in contexts that learners see as authentic, we, as a community, can inspire and assist schools, states, and organizations to implement math curricula that speak to all learners. Our hope is that all students will be engaged in interesting context-based mathematical problem solving that instills a disposition toward appreciating and valuing the richness of mathematics.
Needed Math Conferences Proceedings: References
Carnegie Foundation for the Advancement of Teaching. (2017, March). Carnegie Math Pathways 2015-2016 Impact Report: A Five-Year Review. Retrieved July 27, 2018 from https://www.carnegiefoundation.org/in-action/carnegie-math-pathways
Cockcroft Report. (1982). Mathematics Counts. Report of the Committee of Inquiry into the Teaching of Mathematics in Schools under the Chairmanship of Dr. WH Cockcroft. London. Her Majesty's Stationery Office. ISMB 0 11 270522 7
Consortium for Mathematics and Its Applications (COMAC). (2018). Products and Free Materials. http://www.comap.com.
Garfunkel, S. (2009). Math to Work. Paper prepared for the Carnegie-IAS Commission on Mathematics and Science Education. Retrieved February 2, 2017 from http://www.mathismore.net/articles/pdf/Sol-Math_to_Work.pdf
Johnson, E. (2002). Contextual Teaching and Learning: What It Is and Why It's Here to Stay. Thousand Oaks, CA: Corwin Press
Larson, M. (2017, December 20). Mathematics Learning: A Journey, Not a Sprint. Retrieved March 27, 2018 from https://my.nctm.org/blogs/matthew-larson/2017/12/20/mathematics-learning-a-journey-not-a-sprint
Manufacturing Institute and Deloitte. The Skills Gap in U.S. Manufacturing: 2015 and Beyond. (2015). Retrieved March 27, 2018 from http://www.themanufacturinginstitute.org/~/media/827DBC76533942679A15EF7067A704CD.ashx
Mathematical Association of America (MAA). (N.D.) Curriculum Renewal Across the First Two Years (CRAFTY). Retrieved March 12, 2018 from http://www.maa.org/programs/faculty-and-departments/curriculum-department-guidelines-recommendations/crafty
National Center on Education and the Economy (NCEE). (2013). What Does It Really Mean to Be College and Work Ready? A Report from the National Center on Education and the Economy. Retrieved March 1, 2018 from http://ncee.org/wp-content/uploads/2013/05/NCEE_MathReport_May20131.pdf
National Council of Teachers of Mathematics (NCTE). (2000). Principles and Standards for School Mathematics. Reston, VA: NCTM, p. 64
National Research Council. (2013). The Mathematical Sciences in 2025. Washington, DC: National Academies Press. https://doi.org/10.17226/15269
National Science Foundation. (2018). Advanced Technological Education Program Solicitation. NSF 17-568. Retrieved March 27, 2018 from https://www.nsf.gov/pubs/2017/nsf17568/nsf17568.htm#pgm_desc_txt
Olkun, S. (2003). Making connections: Improving spatial abilities with engineering drawing activities. International Journal of Mathematics Teaching and Learning (1): 1-10
OP-TEC. (2015). Essential Mathematics for Engineering Technicians. Retrieved February 1, 2018 from https://www.optecstore.org/products/essential-mathematics-for-engineering-technicians-pdf-etextbook
US Department of Education (USDOE). (2001). No Child Left Behind Act of 2001. 20 U.S.C. § 6319 https://www2.ed.gov/nclb/landing.jhtml
Winkel, B. (1997). In plane view: An exercise in visualization. International Journal of Mathematical Education in Science and Technology 28(4): 599-607
Winkel, B. (2016). Learning Mathematics in Context with Modeling and Technology. Retrieved May 1, 2018 from https://blogs.ams.org/matheducation/2016/06/27/learning-mathematics-in-context-with-modeling-and-technology
Appendix A. Needed Math Conference Keynote Address
Dr. Solomon Garfunkel, Director, Consortium for Mathematics and its Applications
In preparation for giving this keynote I did quite a bit of research. First I called on the NSF Historian (yes, there is such a person). He sent me to the NSF advanced search database, which has reliable information going back to about 1975. There one sees thousands upon thousands of grants with the key words mathematics, education, industry, technicians. There are conferences, reports, industry specific as well as more general addressing the main themes of this meeting. I found articles in the MATH MONTHLY dating to the late 19th century discussing how to present the mathematics needed by 'modern' industry. This is not a new problem. And this is far from the first conference formed to deal with it. But if we do our job right, it should be the last!
What do I mean by that? Look, we must be aware that essentially, we have two systems that by their nature move at different rates, so that even if we 'catch-up' we will soon be out of sync. Imagine that school mathematics curricula delivered exactly what companies needed tomorrow. Where would we be the day after tomorrow? After all, industry needs what it needs and is used to the notion of having to adapt quickly in order to thrive and survive. Curriculum reform in particular and educational reform in general, move at geologic speed (as anyone who ever tried to change a college course catalog description knows all too well). In many ways the situation is even worse for two-year colleges.
A large part of a two-year college's reputation is built on how successful its students are in transferring to an appropriate four-year school, often a large state university. That university has specific criteria for giving transfer credit, which in a mathematics culture often means standard texts for standard courses. So, required math courses are inevitably taught in mathematics departments in two-year schools trying to clone the 4-year school's offerings. And the student going for an associate degree in IT or Biotech or Manufacturing all too often is sitting in those math classrooms alongside the students hoping to get transfer credit, being taught by teachers who know little of their future employment plans and nothing about IT or Biotech or Manufacturing. Moreover, as numerous reports have documented, the mathematics in those courses is often not what they will need in the workplace – and it is almost always taught out of any real context.
An anecdote. As part of my preparation for this meeting I called a close friend and mentor Henry Pollak. Henry was the director of mathematical research at Bell Labs and then Bellcore for over 28 years beginning in the 60s. He told me an interesting story. For a while during the mid-60s Henry was involved with hiring personnel at the Labs, including technicians. For the senior research personnel, Henry added a criterion. He would ask the applicant to tell him of one instance where he/she used mathematics in their daily life. And, independent of credentials, if the person couldn't name such an instance Henry would not hire them. True story.
But the story even more pertinent to this meeting refers to other hires. Evidently for many years Bell Labs hired technicians who received two-year degrees from DeVry University. And according to Henry these graduates were extremely successful at their jobs. In fact, some were so successful that Bell Labs paid for them to go on to four-year schools to get engineering degrees and thus advance their careers. Well, Devry got wind of this and saw a potential revenue stream. They instituted a full four-year engineering degree program themselves in order to take advantage of this new cohort. But two things happened. First the engineering graduates of Devry were not as well trained as those that Bell had been sending to local universities and second the students who graduated as technicians after the new program had been instituted were nowhere near as good hires as in previous years. And so, Bell Labs stopped hiring Devry graduates in either program. I find this an important cautionary tale.
One more story I heard from Henry that also has something to say about our work this weekend. Some of you may be old enough to remember when there were things called telephone coin boxes. For the youth among us you put your dime (or nickel if you're old enough) in the coin slot in order to get a dial tone and make a call, which likely would cost additional coins. Once upon a time it seemed like there was almost one box on every corner. AT&T sent out its drivers on a regular schedule to pick up the coins in the boxes. But there was a problem. Sometimes a box had very few coins in it and hence not worth the driver's time and other times a box might be so full that it couldn't take any more money.
So the math modelers at Bell Labs were given the problem – was it worth it to put into every coin box an electronic probe which would report when a box had enough coins to be emptied and not be too full. It was estimated that these probes would cost 15 cents each to install. And lo and behold after almost a year's worth of work it was determined that yes in fact it would be cost effective to install the probes and adjust pick up routes accordingly. However, this was the real world and the drivers refused to change their routes and their schedules and the probes were never introduced. Another reminder and reality check on our work.
But we need to get back to why we are here and what we hope to accomplish. Our job is not simply to write yet another report to sit unread in math department and government offices or even to write words that find their way into future NSF RFPs. After all we already know a lot of the answers – we know that a great deal of the mathematics technicians need on the job was presented in grades 6-8 – ratio and proportion, formulas, elementary data analysis. We understand that while arithmetic and geometric visualization are presented in the early grades, in the world of work they are used in complex schema and diagrams. We understand that if mathematics is taught and not used it is forgotten. We know that math taught in context works. We understand the importance of mathematical modeling—moving from the real world to mathematical constructs and back—is fundamental to being able to address new situations. We are a knowledgeable group of practitioners. To quote Joan Crawford (cleaned up a bit) this is not our first time at the rodeo.
Yes, we will go through real examples from our industry colleagues and we will identify the mathematics embedded therein. We will spend some time learning each other's language. Perhaps quite a bit of time. We will talk about what we teach and how we try to teach it. We will learn from each other by identifying deficiencies and how we are trying to overcome them both on the job and in the classroom. We will discuss local programs that have been effective and talk about how we might scale them up. If we do this well—and I'm sure we will—we will get a useful snapshot of where we are with hints as to how we might get better.
But I hope we can do more! Because as I said at the beginning we have two systems that are out of alignment and are likely to keep being so unless we can set in place a self-correcting mechanism that can adjust for needed change in a timely way. Figuring out what that mechanism should be and how to implement it seems to me to be a challenge worth working on. And so I look forward to a wonderful, productive, and LAST conference.
APPENDIX B. Steering Committee and Conference Participants
Steering Committee Members
- Marilyn Barger, Director, Florida ATE Center, Hillsborough Community College, Tampa, FL
- Rosemary Brester, President and CEO, Hobart Machined Products, Hobart, WA
- Sol Garfunkel, Director, Consortium for Mathematics and Its Applications (COMAP), Bedford, MA
- Michael Hacker, Co-director, Center for STEM Research at Hofstra University, Hempstead, NY (Project PI)
- Katherine Hughes, Principal Researcher, American Institutes for Research, Washington, DC (Research Lead)
- Paul Horwitz, Senior Scientist, Concord Consortium, Concord, MA (Project C0-PI)
- Jennifer Lazare, Biotech Instructor, Anderson High School and Austin Community College, TX
- Rodney Null, Professor, James A. Rhodes State College, Lima, OH
- Gerhard Salinger, retired National Science Foundation program officer and originator of the ATE program, Albuquerque, NM
- Lisa Seidman, former Program Director in Biotechnology, Madison Area Technical College, Madison, WI
- Gordon Snyder, Co-principal Investigator, National ATE Center for Optics and Photonics Education, Waco, TX
- Deborah Hecht, Director, Center for Advanced Study in Education, CUNY Graduate School, NYC
Steering Committee Biographies
Marilyn Barger is the principal investigator and executive director of the Florida Advanced Technological Education Center of Excellence (FLATE) housed at Hillsborough Community College in Tampa. FLATE is funded by the National Science Foundation (NSF) and serves technical education pathways and programs in Florida that support the manufacturing industry. Marilyn has over 20 years of experience developing and delivering STEM curriculum for K-20 students and educators and has initiated long-term, systemic education reforms. Her PhD is in environmental engineering. Marilyn holds a licensed patent in membrane technology and is a registered professional engineer in Florida.
Rosemary Brester is president and chief executive officer of Hobart Machined Products, Inc. She has served as a member of the Governor’s Pipeline Committee, Transition Math Project, and numerous industry and community organizations. Rosemary is passionate about convening industry, community, and governmental groups to make the most of small businesses - and in turn deliver benefits to all of those groups. Rosemary holds an AA degree in Business Management from Green River Community College and a Certificate in Business Collaboration from UW Foster Business School, along with numerous manufacturing-related certificates from South Seattle Community. She has a passion for aerospace, space and education.
Sol Garfunkel received his PhD from the University of Wisconsin. He taught at Cornell University and the University of Connecticut before founding COMAP (Consortium for Mathematics and its Applications), where he has been Executive Director since its inception. He has directed NSF-funded curriculum development projects, including UMAP, HiMap, and ARISE. Sol was host for the Annenberg/CPB telecourse, For All Practical Purposes and founded and administers the Mathematical Contest in Modeling. Sol received the Glen Gilbert Award from the National Council of Supervisors of Mathematics (NCSM) and a lifetime achievement award from the International Society for Design and Development in Education (ISDDE). He served on the Mathematics Expert Panel of PISA.
Michael Hacker, PhD, Needed Math Project PI, is co-director of the Center for STEM Research (CSR) at Hofstra University. He has conceived, written, and directed 14 large-scale NSF projects focused on K-14 STEM education. During over 50 years in education, Michael was a teacher, department supervisor, and university teacher educator. As New York State Education Department Supervisor for Technology Education, he co-managed the development of New York’s Standards for Mathematics, Science, and Technology. Michael has co-authored six textbooks and numerous journal articles, scholarly compendia, and conference proceedings. He was on the writing team for the National Standards for Technological Literacy and is a member of the ITEEA Academy of Fellows.
Paul Horwitz, PhD, holds the position of senior scientist at the Concord Consortium, a nonprofit organization dedicated to creating innovative educational technology for STEM learning. Following research in theoretical physics in universities and industry, Paul has been exploring the application of technology to teaching and assessment. He has worked in a wide variety of discipline areas and with various age ranges, including running projects that taught evolution to fourth graders, classical mechanics and genetics to middle schoolers, and statistics, chaos theory, and relativity in high school. Currently, he directs projects aimed at assessing collaborative problem-solving skills in students enrolled in electronics courses at two- and four-year colleges.
Katherine Hughes, PhD, is a principal researcher at the American Institutes for Research, and also the project director for Technical Assistance for Advancing Equity in Career and Technical Education, an initiative funded by the U.S. Department of Education and awarded to Manhattan Strategy Group. Katherine also works as an independent consultant for the Community College Research Center (CCRC) and other organizations. For many years she was the assistant director of CCRC, conducting and overseeing research on a range of topics addressing the transition from high school to college and careers. She has numerous publications, many on dual enrollment programs, and she co-authored the book, Working Knowledge: Work-Based Learning and Education Reform. She holds a doctorate in sociology from Columbia University.
Jennifer Lazare has taught biotechnology as a dual credit high school instructor and an adjunct faculty member at Austin Community College for 10 years. Jennifer also serves as the high school outreach coordinator for biotechnology education as senior personnel on the ATE Regional AC2 Bio-Link grant. In this role, she manages a statewide mentor network for high school biotechnology teachers, develops curriculum and assessments, and provides hands-on training to teachers and administrators. Jennifer received her master’s degree in molecular genetics and K-12 science teaching certificate from the University of Texas. She holds principal certification and is enrolled in the PhD program for school improvement at Texas State University.
Rodney Null is a mathematics professor at Rhodes State College with 30+ years of teaching experience, ranging from middle school through university. He is a recipient of the American Mathematical Association of Two-Year Colleges’ Teaching Excellence Award and former president of the Ohio Mathematics Association of Two-Year Colleges. He was a steering committee member for the Ohio Mathematics Initiative. Rod has participated in three NSF-sponsored projects, most recently as PI for Mathematics Transitions in STEM Education and has been a program developer and instructor for numerous initiatives focusing on professional development programs for teachers and improving mathematics instruction.
Gerhard Salinger, PhD, retired from the National Science Foundation (NSF) in 2014, where he co-founded the Advanced Technological Education (ATE) Program. He continues to be interested in STEM education K-12 and in the impact of ATE on technician education 7-14. Prior to coming to the NSF, he was on the faculty of the Physics Department at Rensselaer Polytechnic Institute. He now lives in Albuquerque.
Lisa Seidman is a co-PI for Bio-Link, an NSF-funded consortium of biotechnology programs. She received her PhD in biology from the University of Wisconsin and was a post-doctoral fellow at Yale University and the University of Wisconsin before becoming an instructor in biotechnology at Madison Area Technical College in 1987. Lisa developed a contextual biotechnology math program that was named “exemplary” by the American Mathematical Association of Two-Year Colleges in 2001. She is the lead author of three textbooks: Basic Laboratory Methods for Biotechnology: Textbook and Laboratory Reference, Basic Laboratory Calculations for Biotechnology, and Laboratory Manual for Biotechnology and Laboratory Science: The Basics.
Gordon F. Snyder Jr. is past director of the National Center for Information and Communications Technologies and currently a consultant and associate director at the National Center for Optics and Photonics Education. He has authored four engineering/technology textbooks and has over 33 years of industry consulting experience. He served as the Verizon Next Step program telecommunications curriculum co-leader and on several technology boards around the United States, including the Microsoft Community College Advisory Council. In 2001, he was selected as one of the top 15 faculty in the country by Microsoft and the American Association of Community Colleges.
External Evaluator Biography
Deborah Hecht, PhD, is director of the Center for Advanced Study in Education, City University of New York (CUNY) Graduate Center. She has been the co-PI or lead evaluator on dozens of NSF and other funded projects, ranging from small programs that focus on a regional group to statewide and national projects designed to impact practice and policy. With over 30 years of experience, she has studied initiatives in a wide variety of areas including youth development, school reform, STEM education, and faculty development. Deborah has a background in applied psychology and uses multi-methods approaches to study innovations within the context of the implementation environment.
Conference Participants (Alphabetically by Domain)
|Last Name||First Name||Domain
(Biotech, ICT, Mfg)
|Hongo||Jo-Anne||Biotechnology||Employer||Genentech and JS Hongo Consulting|
|Jochem||Adam||Biotechnology||Employer||Morgridge Institute for Research|
|Kirkpatrick||Bridgette||Biotechnology||Instructor, CC||Colin College|
|Simpson||Kevin||Biotechnology||Employer, Instructor, CC||Delgado CC/Autoimmune Technologies LLC|
|Balas||Cathryn||ICT||Employer||Balas Consulting Services|
|Beheler||Ann||ICT||Instructor, CC||Collin College|
|Stephenson||Scott||ICT||Instructor, Secondary||Education Service Center Region 11 Cisco Academy|
|Adams||Kirk||Manufacturing||Employer||American Foundation for the Blind|
|Horine||Dan||Manufacturing||Instructor, CC||Virginia Western CC|
|Mason||John||Manufacturing||Instructor, CC||Ivy Tech Community College|
|Monroe||Nathan||Manufacturing||Employer||Toray Composite Materials America|
|Simpson||Elizabeth||Manufacturing||Instructor, Secondary||Polk County Public Schools|
|Sweatman||Shannon||Manufacturing||Employer||Southern Manufacturing Technologies|
|Thompson||Matthew||Manufacturing||Employer||Toray Composite Materials America|
|Wallace||Blake||Manufacturing||Employer||Process Equipment and Service Co.|
|Wallace||Jana||Manufacturing||Employer||Process Equipment and Service Co.|
|Altose||Aaron||Mathematics||Instructor, CC||Cuyahoga Community College|
|Baratto||Stefan||Mathematics||Instructor, CC||Clackamas Community College|
|Math Educator||SUNY Stony Brook|
|Getz||Amy||Mathematics||Math Educator||Dana Center|
|Kimball||Rob||Mathematics||Math Educator||Wake Technical CC (Retired)|
|Lewis||Matthew||Mathematics||Math Educator||San Jacinto CC, AACC|
|Robinson||Eric||Mathematics||Math Educator||Professor Emeritus, Ithaca College|
|Robinson||Stacy||Mathematics||Instructor, Secondary||Trotwood Madison High School|
|Younker||Michelle||Mathematics||Instructor, CC||Owens Community College|
|Instructor||Center for Occupational Research and Development (CORD)|
|Instructor||National Research Center for CTE Southern Regional Education Board|
APPENDIX C: Agenda and Conference Organization
Needed Math Conference Agenda
Friday, January 12
Registration opens in Friendship One Room
Greetings and Introductions
Welcome: Dr. Gerhard Salinger, Founder, NSF Advanced Technological Education (ATE) Program
Intended conference outcomes
Overview of agenda, discussion groups, data collection, research and evaluation plans
6:15 p.m. Keynote Address: A Conference to End All Conferences
Dr. Solomon Garfunkel, Executive Director, Consortium for Mathematics and its Applications (COMAP)
7 p.m.: Buffet Dinner
Saturday, January 13
7:30 - 8:15 a.m.: Continental Breakfast in Bistro 1 Restaurant
8:15 - 9:45 a.m.
Employer Panel: Discussion of needed math with Jo-Anne Hongo, Biotech; Mark Taylor, ICT; Jana Wallace, Manufacturing; and Kirk Adams, Using Math on the Job as an Employee Who Is Blind or Visually Impaired. Dr. Lisa Seidman, Moderator. Q&A.
9:45 a.m.: Refreshment Break
Session 1. Small-group discussions (same domain). Begin to develop ideas to be refined during the conference and presented at Session 7 on Sunday 1:15 pm
Noon: Buffet Lunch
Session 2. Domains reconvene in larger groups to refine emerging recommendations on needed math. Each domain produces three or more Data Collection flipcharts. Begin to consider these questions:
- What commonalities did you find in the needed math?
- With what math do employees have difficulty, and why?
- Given that technicians have difficulty with these problems in the workplace and given that they studied math for years, what can we do about it?
3:15 p.m.: Refreshment Break
3:30 – 4:30 p.m.
Session 3. Groups present Session 2 results on flipcharts. Dr. Gerhard Salinger, Moderator
4:30 – 5 p.m.
Session 3, continued. Synthesis. Group discussion of responses to questions considered in Session 2.
5 – 6 p.m.: Break
6 p.m.: Buffet Dinner
Concluding Discussion: What went well. Sunday agenda review. Dr. Deborah Hecht, Moderator
Sunday, January 14
7:30 - 8:30 a.m.: Continental Breakfast in Bistro 1 Restaurant
8:30 – 8:35 am: Reimbursement procedures – Lois Miceli
8:35 - 9:45 a.m.
Session 4. Affinity groups address focus questions:
- Employers: With what math do employees have difficulty and why? What guidance can you give to educators? How is the field evolving, and what are the implications for math requirements?
- Instructors: Given the constraints of the present system, what would you change with respect to mathematics in secondary school and community college programs?
- Math educators: What needed math commonalities and differences (between domains) have emerged? Given that students have difficulty with math after years of study, what can and should we do?
9:45 - 10:30 a.m.
Session 5. Affinity groups report out
10:30 a.m.: Refreshment Break
10:45 a.m. - 12:15 p.m.
Session 6. Domain-based groups finalize the work started on Saturday and prepare flipchart presentations
12:15 p.m.: Buffet Lunch
1:15 – 2:15 pm
Session 7. Presentations from each domain
2:15 -3 p.m.
Concluding Session. Large-group discussion and reflective comments. Potential areas for funding new proposals: Dr. Gerhard Salinger. Framing recommendations, further initiatives, next steps. Paul Horwitz, Moderator
3 p.m.: Farewell
Needed Math Conference Organization
The Needed Math Conference was organized to encourage active participation from all attendees. Presentations to stimulate discussion and frame the conference outcomes were made by NSF Advanced Technological Education program founder, Dr. Gerhard Salinger, and Dr. Solomon Garfunkel, Executive Director, Consortium for Mathematics and its Applications (COMAP).
A panel comprised of employers from the three Conference domains — biotechnology, information and communication technology, and manufacturing technology — described examples of the mathematics that their STEM technicians typically applied in their work. This occurred on the first morning of the conference and provided a backdrop to group deliberations.
Three types of groupings were facilitated by Steering Committee members.
- To begin, small groups of people from the same domain (employers, community college, and secondary school instructors) introduced themselves, discussed intended conference outcomes, and raised questions. Also at each table were math educators. The facilitators clarified objectives and guided data gathering procedures. The small groups reconvened several times to capture needed math ideas.
- All participants in the same domain met to discuss and refine emerging recommendations. Each of these three larger groups was facilitated by a domain-based Steering Committee member. The groups developed flipcharts using a consistent template to begin to generalize ideas.
- Affinity groups of all employers, instructors, and math educators were convened to add their suggestions regarding conference recommendations. The community college and secondary instructors addressed specific issues related to implementation of needed math at their school level.
Large group discussions focused on synthesizing findings and recommendations from domain-based and affinity groups.
This interactive framework resulted in spirited discussion and the development of a collective perspective.
APPENDIX F: Secondary School Implications
A Conference subgroup of secondary school STEM educators suggested that Career and Technical Education (CTE)/STEM pathways should be designed to maximize student engagement using "real- world/industry" mathematics. Such coursework should satisfy core graduation requirements. STEM and math instructors should work as a team to reproduce the workplace environment with scenarios and context-based assessments.
- Provide a complete inventory and alignment of math courses to all STEM courses/pathways by functions (e.g., addition, division, extrapolating, graphing, best fit line, linear regression, finding unknown values, order of operations, interpolating).
- Create a marketing document for initial engagement that targets policymakers and other specific groups.
- Share the math misalignment with administrators, non-math people, and policymakers highlighting what is taught vs. what is needed in the workplace.
- Begin the gap analysis at the middle school level to ensure alignment with workplace needs.
- Have math teachers, STEM instructors, and industry personnel collaborate to create scenarios based on authentic industry problems/tasks; then use them in curriculum development and PD incorporating the gap analysis at regional or national centers.
- Create a bank of scenarios aligned with STEM courses, fields, and standards, with online access and editable features. Provide detailed answer keys that also highlight common misconceptions and possible errors. This is vital in order for teachers to have confidence in putting new materials into their courses. Videos of teacher explanations of thought processes and problem solving on a whiteboard would be more helpful than text-based materials.
- Require math lab for a certain percentage of class time.
- Allow the STEM and math teachers to be the same person double certified
- Have STEM and math teachers collaborate on campus and meet with industrialists regularly.
- Update curriculum periodically to reflect changes in the workplace.
- Require state-level training to teach STEM math courses (and include a teacher externship with scenario-based training).
- Allow more flexibility for alternative math courses, as well as math-rich STEM courses, to count for math credit toward graduation requirements.
- Create math classes that are specific to a pathway and team taught by a math teacher and subject-matter expert.
- Require every CTE and math course to have objectives (students should be able to, SSBAT). These objectives should include scenario-based learning in both courses.
Teacher Professional Development
- Require working with an advisory board that includes industry representatives.
- Partner with national centers to integrate the scenarios and the contextualized math into existing teacher PD/training.
- Include scaffolding: Use whiteboard samples to show how to explain the concept to students (like Khan Academy videos but focused on scenario-based math).
- Form a cohort from the same school consisting of STEM and CTE teachers with disciplinary and mathematics expertise to collaborate with industry personnel to provide PD. As in-person PD may not be possible for everyone, offer online instruction as well.
- Provide model programs and best practice examples.
- Develop and provide rubrics so that programs can "self-grade," and use them to assess student progress and provide formative feedback.
- Provide a roadmap for implementation at the school and district levels.
- Provide teachers with industry-based experiences.
- Establish regional or national ATE/CTE centers focused on contextualized mathematics that is aligned with the workplace needs.
- Establish a Master Teacher program: STEM and math teachers would work together in collaboration with an industrialist and/or a CC technical program that has an operational advisory board.
- Create and support an online community to share best practices.
- Adopt scenario-based teaching and learning as standard expectations in course knowledge and skills (policy).