By the third quarter of the year-long statistics sequences, SS10C and ECON10C students already have a substantial background in statistics, but limited experience in dealing with open-ended problems that are not straightforward applications of a single statistical tool in a way that derives directly from formulae and approaches specified in the textbook. To define a team assignment that exemplifies PBL principles for these statistics courses, and to provide a touchstone to demonstrate to students the importance of applied statistical analysis, coming as the culmination of their year long course, we will assign a previously published research paper or report by a reputable scholar or organization that (a) involves data analysis at a reasonable level of sophistication (but not so complex that the students could not understand it), (b) draws strong public policy conclusions from its analyses, (c) provides the actual data set(s) so that it (they) can be used for both straightforward reanalysis and further analyses by the students, and (d) is on a topic where additional rearach material and data sets on WEB or in the library are available that can be used for supplementary analyses of the topic. To further motivate students, we also intend to make sure that (e) the topic is interesting and (f) the claims made by the author(s) are either controversial or counterintuitive.

     For SS10C/ECON10C Spring 2001, during the last six weeks of the class (with their final report, an in-class Powerpoint presentation to be given during finals week in lieu of a traditional final), students will be formed into groups of three to look at the American Legislative Exchange Council "Report Card on American Education: A State-By-State Analysis, 1976-1998," a 106 page statistics-laden document issued in 1998 by a conservative think tank. This report argues that public money on public primary and secondary education is not well spent, and claims that there is no link between the amount a state spends on education and the educational performance of its students. In particular, the Report asserts that:

     Much of the analysis in the report involves bivariate or multivariate regression of indicators of student performance on spending data, with states as the unit of analysis. The next slides show illustrative bivariate analyses from the ALEC Report.

The student project is in five parts. The first three parts, and the fifth part are each to be completed within one week, while the fourth part (the heart of the project) is given three weeks.

(1) Part I is to simply redo the main part of the original analysis using the raw data provided to them on disk by the authors of the report. They must replicate at least three of the analyses that support the specific findings that "[T]here is no evident correlation between teacher-student ratios, spending on school infrastructure, and teacher salaries on the one hand, and educational achievement as measured by various standardized test on the other hand." The main purpose of this part is simply to make sure that student's understand the analyses in the report and can replicate them.

(2) Part II is to critique the research looking for poor operationalization of variables, missing variables, inadequate research design that renders causal conclusion inappropriate or suspect, inadequate documentation of methods or findings, etc. Here, students are asked to draw on the discussions earlier in the course of issues of varaibel definition and proper research design. These first two parts of the assignment are to make sure that all the students are "on the same page" and are ready to cope with the really challenging later aspects of the assignment.

(3) Part III is to reanalyze the data in a fashion that improves on the original analysis using the data that is made available in the article. Here, there are various ways that students could redo the analyses, but there still are some particular points that we will be looking for as a check on student's subject mastery (e.g., their ability to make use of multivariate regression incorporating key variables).

(4) Part IV really is the heart of the PBL aspect of the assignment. This part will be given the most time, three weeks, to complete, but students will be asked to begin the research for this part of the project as of the fifth week of the quarter. Now the student is asked to reanalyze the evidence in a fashion that improves on the original analysis by going beyond the data that was originally made available in the article. To do this, students will need to find additional sources of theory and data via MELVYL and the WEB and to interview people (e.g., teachers, principals, members of the education department at UCI and elsewhere) who might help them gain insights into the problem. Here, there are no clearly right answers, and the TA and I expect to learn, ourselves, from the analyses the students will do. Of course, the TA and I will be available to give advice. For example, we will suggest that they look at the accuracy of measures of educational performance, try to figure out where state education money actually goes, and examine factors that affect student test performance like SES and English language proficiency.

(5) Part V is to prepare an in-class Powerpoint slide presentation (approximately 15 minutes) that highlights the critique, the new analyses, and the policy conclusions in a clear and graphic form. This presentation will be made during the time normally scheduled for finals (although perhaps not all groups will have time to do public presentations; the remainder will submit their slides to the instructor.)

While the statistical tools used will be ones previously taught, this full-scale real-world application of multiple tools and use of multiple data sources will be far more challenging than anything done previously, and will be the culmination of the course -- a signal to students that they are competent to use statistics, and evidence that using statistics is relevant to real world policy analysis of important issues.

 

List of PBL Questions and Answers

PBL Project for Hewlett Foundation Grant

"Does More Money Buy Better Education?"
(For the third quarter, SS10C/ECON10C, of the social sciences/economics statistics sequences)

Instructor: Bernard Grofman, Department of Political Science,
School of Social Sciences
TA: Anthony Crider, Department of Linguistics, School of Social Sciences

 

Q. How many students per problem-solving group?

A. 3

 

Q. How will students be assigned to PBL groups?

A. Randomly, by kth digit of student ID.

 

Q. How much credit to give for the PBL project?

A. The PBL project will serve in lieu of a final. We will give it the same weight normally given a final exam (In my statistics courses, grade weights are variable so that a final may count for anywhere from 10 to 45% of the grade: 10% if the final is the student's worst grade (on homeworks, midterm, and final), 45% if it is the student's best grade on these items.) It is seen as the culmination of a year of statistics, demonstrating to students that they have sufficient mastery of the material to tackle important topics.

 

Q. How much of the class time to allocate to this PBL project?

A. The last six weeks of the class will be devoted to the project, as well as finals week. We expect that about 1/3rd of the class time each of these weeks will be used for discussion related to the project. Student in-class presentations on the project will take place during what would otherwise have been the time scheduled for finals.

 

Q. How to handle sequencing?

A. The project will be divided into five parts. Students will be required to complete one part per week for each of the first three weeks. Weeks 8-10 will be devoted to Part IV, the heart of the project. The parts build on each other and grow more intellecturally challenging, and the fourth component requires students to find additional information and data sources on the WEB and elsewhere. The fifth part requires each group to present a 15 minute powerpoint slide show to demonstrate their main findings.

 

Q. How to assign and define student roles?

A. Because the task is divided into parts, we will be assigning one student in the group the role of coordinator on each of the first three parts. Initially this assignment will be done randomly, but students who wish to may trade assignments (but must notify the instructor of the "trade"). Because Parts I and II are the easiest in work load, the coordinator on Part I will also be the coordinator on Part V and the coordinator on Part II will also be the (informal) overall coordinator on Part IV. Part IV is the heart of the project. It is the most complex, and is given three weeks to do, rather than the one week allowed to complete each of the other parts. Part IV will be divided into three subparts, with each student assigned a coordinating role for one of the subparts, with the coordinator on Part III of the project also taking the overall coordinating role for this part. In Part IV, some students would be assigned to search the WEB for info, some to check for info on MELVYL, and others to interview teachers and administrators in (local) schools for their insights.

We will not attempt any very detailed description of the coordinator's role other than to indicate that s/he must arrange for the group to get together either in person or otherwise to get the necessary work done, and to indicate that the rough division of expected labor for each of the parts (or subparts) is 50% for the coordinator and 25% for each of the two others.

  

Q. How much of the assignment credit is given for individual contributions? How much credit is assigned to the group as a whole?

A. Each student will be given 50% of the credit for the component(s) or subcomponents that s/he coordinated and 50% of the grade based on the other three major components. That should guarantee incentives for coordinators to engage in special efforts to put together a good team effort, but also incentives for those not in the coordinator's role to still feel the need to contibute.

 

Q. How to handle feedback?

A. After each of the first three parts of the project are handed in, we will go over their answers and provide feedback. Also we offer a "checklist" of key points that should have been addressed to make sure that everybody is "on the same page" when they start the next part of the project.