Research
Methods & Design

INTEG 340, Winter 2018
Tuesdays and Thursdays, 10:00 AM - 11:20 AM in EV2 2069
Class Slack Group: integ340-winter2018.slack.com

Professor: Dr. John McLevey
Office hours: Thursdays, 3 pm - 4:30 pm or by appointment


Course Description

This course provides an introduction to empirical research design and methods with a focus on applications in the social sciences and related fields. You will learn about core issues in research design (e.g. sampling) that transcend specific approaches, and about a variety of techniques for collecting and analyzing quantitative and qualitative data. The course will cover both abstract and practical issues related to methodology and decision making in empirical research. You will have the opportunity to develop and workshop proposals that you may wish to pursue in your senior honors thesis. By the end of the course you will be a more informed consumer and have a basic set of skills for designing and implementing your own empirical research projects. Most importantly, you will have a foundation for future learning about research design and methods.

Learning Objectives

The Schedule section of the syllabus identifies the core learning objectives for every scheduled topic in the course.

Books

There are 2 books that we will read cover to cover. They are available on reserve at the university library (Porter), in the university bookstore, and online.

Earl Babbie and Lance Roberts. 2018. Fundamentals of Social Research. Nelson.

This book is used in other research methods classes on campus, so you should be able to find used copies without too much difficulty. If for some reason you can't get this edition, it's OK to use a previous one. However, you must talk to me about how you can compensate for differences in the content across editions. If at all possible, I would prefer for you to use the 4th edition (i.e. the 2017 edition).

Matthew Salganik. 2017. Bit by Bit: Social Research in the Digital Age. Princeton University Press.

You can currently read Salganik's book for free online at http://www.bitbybitbook.com/en/preface/. I am not sure if / when that will change.

Any other readings will be available on the course website, on reserve, or through the library database.

Deliverables



Assignment Dates %
Comprehension Quizzes      Due at 9 pm on every Mon and Wed between Jan 8 and Mar 14      40
Class Facilitation      Sign up for a slot on March 22 or 27      10
4 Mini Research Proposals      All due at 9 pm on Friday April 13      40
Engagement / Participation      Ongoing      10


Comprehension Quizzes     

There are no exams in this course, but there are many short quizzes to test your comprehension of the readings and other class material. You can do them on your own time, wherever you want, with or without your textbook, but you must complete them independently. No collaboration of any kind. The quizzes will be due at 9 pm on every Monday and Wednesday between January 8 and March 14th, except reading week. LEARN will not let you submit a late quiz. If you need to submit late, please email me a medical note and request that I re-open the quiz for you.

Each quiz will consist exclusively of multiple choice and, in some cases, true / false questions. You should expect each quiz to have a lot of questions (usually between 30 and 60), but if you have done the readings carefully you should be able to complete each quiz in less than 20-30 minutes.

I suggest that you approach each quiz the following way:

  1. Do the assigned readings carefully
  2. Answer as many questions as you can without consulting your book
  3. When you get to the end of the quiz, go back to your book and re-read the sections that explain content you were unsure about
  4. Go back to your quiz and answer the remaining questions
  5. Submit!
Obviously it will take you longer than 20 minutes to do each quiz if you use this strategy, but the extra time you invest will pay off. This approach will help you learn better and more efficiently. You will retain more information from each chapter, which will help you avoid problems later in the term.

Each quiz will cover content from the readings that are scheduled for the upcoming class. In other words, the quiz covering the readings for January 9th's class on "theory-driven empirical research" will cover Chapter 1 "Human Inquiry and Science" from Babbie & Roberts (2018) Fundamentals of Social Research. The quiz will become available on LEARN after class on Thursday January 4th, and will close on Monday January 8th.

Why have I decided to quiz you on readings before we cover them in class? And why are there so many questions on each quiz? Reasonable questions... The main reason why I have scheduled the quizzes this way is because it enables me to tailor our class meetings to meet your intellectual needs as they evolve throughout the semester. Every Tuesday and Thursday morning, I will look at the results of the quiz you submitted the night before. If necessary, I will revise my notes for the class meeting to clarify things you seem to be struggling with. Similarly, the reason I have included so many questions on each quiz is because I need to know exactly what you don't understand. If I ask you a few questions about each of the key concepts from each chapter, then there end up being a lot of questions on each quiz. I think that's fair, since all the questions are multiple choice and true / false.

I have enabled auto-grading for the quizzes on LEARN. That means that you will get your evaluation as soon as you finish the quiz, and you will be able to see the correct answers for all questions. I set it up this way to help you learn as efficiently as possible.

Class Facilitation     

You will collaborate with 4-5 students (approximately 1/4 of the class as of January 3rd) on a 30 minute class covering one chapter from Matthew Salganik's book Bit by Bit. Eligible chapters are: "Observing Behavior," "Asking Questions," "Running Experiments," and "Mass Collaboration." We will pick collaborators and chapters in class on February 15th. The student-led classes will happen on March 22 and 27.

You and your collaborators should begin with an overview of the core arguments from the chapter. You should explicitly relate those arguments to relevant material covered in the Babbie and Roberts' textbook or in previous class discussions. For example, if you are covering the Bit by Bit chapter "Asking Questions," you should connect the key arguments to content on survey methodology and qualitative interviews covered in Babbie and Roberts. Once you and your collaborators have reviewed the key material from your Bit by Bit chapter, you should pose questions for class discussion.

Each chapter in Bit by Bit includes a technical appendix, further commentary, and a list of potential activities. It is not necessary for you to cover any of this material in your 30 minute class, but you should pick a couple of activities that you think are especially interesting or useful. If both groups stick to 30 minutes, there will be 20 minutes of remaining class time on March 22 and 27th. We will use that time to start (and maybe finish) one of the activities.

4 Mini Research Proposals     

In addition to your final research proposal, you will collaborate (in groups of 3-4) on four different mini-proposals: (1) one for a quantitative project, (2) one for a qualitative project, (3) one for an experimental / audit study, and (4) one for a content analysis. You may work with different students on each proposal if you like. It is not necessary to have a completely new topic for each mini-proposal. Each mini-proposal is limited to 650 words. You must edit your work very carefully. There are no words to waste!

Each proposal must identify a general problem, pose a few good research questions, operationalize key concepts, propose reasonable hypotheses, propose and justify a sampling strategy, explain how data will be collected, and then explain what the expected contribution of the work will be. Finally, each proposal must include a paragraph, approved by all authors, that clearly describes what each person contributed (2-4 sentences per person). Unless there is a problem, everyone will receive the same grade.

Engagement / Participation     

The quality of this course -- like any other -- depends on you being engaged. Your participation grade will be based on (a) contributions to class discussion, (b) small group discussion, (c) your involvement in any online discussions, and (d) attendance. If you really don't like speaking up in class, you can participate more online, but you must speak with me about this. Although I will not be assigning a participation grade until the end of the semester, I am happy to provide qualitative feedback on your participation throughout the semester.

If you arrive more than 10 minutes late, you will lose 50% of the credit for attending class. In other words, arriving late twice is equivalent to missing a class. There is no penalty for excused absences, which always require advance notice and generally require a note from a doctor.

COURSE SCHEDULE



Date Topic
Thursday Jan 04     Introduction + How to study for and succeed in this course
Tuesday Jan 09      Theory-driven Empirical Research
Thursday Jan 11     Paradigms, Theory, and Research
Tuesday Jan 16      Research Ethics
Thursday Jan 18     Research Design and the Logic of Causation
Tuesday Jan 23      Conceptualization, Operationalization, and Measurement
Thursday Jan 25     Indexes and Scales
Tuesday Jan 30      The Logic of Sampling, Part 1
Thursday Feb 01     The Logic of Sampling, Part 2
Tuesday Feb 06      Experiments
Thursday Feb 08     Survey Research
Tuesday Feb 13      Non-Reactive Research
Thursday Feb 15     Field Research
Tuesday Feb 20      Reading Week, No Classes
Thursday Feb 22     Reading Week, No Classes
Tuesday Feb 27      Qualitative Interviewing
Thursday Mar 01     Evaluation and Action Research
Tuesday Mar 06      Qualitative Data Analysis
Thursday Mar 08     Quantitative Data Analysis
Tuesday Mar 13      The Logic of Multivariate Analysis
Thursday Mar 15     Social Statistics
Tuesday Mar 20      Reading and Writing Social Research + Digital Data
Thursday Mar 22     Digital Data + Observing Behavior + Asking Questions
Tuesday Mar 27      Digital Data + Running Experiments + Mass Collaboration
Thursday Mar 29     Revisiting Ethics + The Future of Social Research

Detailed Schedule and Learning Objectives


thursday jan 04 INTRODUCTION + HOW TO STUDY FOR AND SUCCEED IN THIS COURSE

This class meeting will introduce the core themes and learning objectives for the term. By the end of the class, you should (1) know what the key themes of the course are, (2) be able to clearly explain what I expect from you in the course, and (3) understand what you need to do to succeed.

  • This syllabus!

tuesday jan 09 THEORY-DRIVEN EMPIRICAL RESEARCH

By the end of this class, you should be able to (1) understand how and why social scientists integrate theory and empirical research, (2) understand the importance of studying aggregates rather than solely individual cases, (3) start speaking in "variable" language, and (4) differentiate between inductive and deductive reasoning.

  • Babbie & Roberts (2018) Chapter "Human Inquiry and Science"

thursday jan 11 PARADIGMS, THEORY, AND RESEARCH

By the end of this class, you should be able to (1) distinguish between general paradigms and specific theories at both micro- and macro-levels, (2) understand how to derive testable hypotheses from theoretical propositions, and (3) understand how hypotheses can be tested empirically. Finally, you should be able to (4) contrast theory construction and hypothesis testing in inductive and deductive research.

  • Babbie & Roberts (2018) Chapter "Paradigms, Theory, and Research"

tuesday jan 16 RESEARCH ETHICS

By the end of this class, you should be able to to (1) explain the ethical precepts that researchers must consider before undertaking research with human participants, in particular ethical precepts about confidentiality, disclosure, potential harm, and accurate reporting of results. In addition, (2) you should have some basic familiarity with the Tri-Council’s ethical guidelines on respect for persons, concern for welfare, and justice.

  • Babbie & Roberts (2018) Chapter "Ethical Issues for Social Researchers"

thursday jan 18 RESEARCH DESIGN AND THE LOGIC OF CAUSATION

By the end of this class, you should be able to (1) compare different purposes for a research project, (2) describe the three criteria for causation (non-spuriousness, temporal ordering, and correlation), (3) clearly specify the units of analysis for research, and (4) distinguish between cross-sectional and longitudinal designs.

  • Babbie & Roberts (2018) Chapter "Research Design and the Logic of Causation"

tuesday jan 23 CONCEPTUALIZATION, OPERATIONALIZATION, AND MEASUREMENT

By the end of this class, you should be able to (1) define conceptualization, operationalization, and measurement, (2) explain how to conceptualize and measure variables, and (3) explain how to ensure that measures are valid.

  • Babbie & Roberts (2018) Chapter "Conceptualization, Operationalization, and Measurement"

thursday jan 25 INDEXES AND SCALES

By the end of this class, you should be able to explain what is involved in developing good (1) indexes and (2) scales.

  • Babbie & Roberts (2018) Online Chapter (See LEARN) "Indexes and Scales"

tuesday jan 30 THE LOGIC OF SAMPLING, PART 1

By the end of this class, you should be able to (1) explain how samples are used to represent broader populations, (2) describe a variety of sources of potential error and bias in sampling processes, and (3) explain when and why to use probability sampling vs. non-probability sampling strategies.

  • Babbie & Roberts (2018) Chapter "The Logic of Sampling"

thursday feb 01 THE LOGIC OF SAMPLING, PART 2

By the end of this class, you should be able to (1) articulate strategies for drawing samples systematically.

  • Babbie & Roberts (2018) Chapter "The Logic of Sampling"

tuesday feb 06 EXPERIMENTS

By the end of this class, you should be able to (1) explain how experimental and quasi-experimental methods are used in the social sciences, such as in studies of discrimination in hiring processes, (2) describe the power of experimental and quasi-experimental methods for understanding causal processes, and (3) describe the value of "natural" experiments in the social sciences.

  • Babbie & Roberts (2018) "Chapter "Experiments"

thursday feb 08 SURVEY RESEARCH

By the end of this class, you should be able to (1) summarize the total survey error framework, and (2) distinguish between well-written and poorly-written survey questions.

  • Babbie & Roberts (2018) Chapter "Survey Research"

tuesday feb 13 NON-REACTIVE RESEARCH

This class cover the basics of analyzing secondary statistics, content analysis, and comparative-historical research. There is a lot to cover, and our conversation will necessarily be a bit superficial. By the end of this class, you should be able to (1) explain the value of analyzing already-existing statistics, (2) explain how existing statistics can be used to motivate and provide context for non-quantitative research, and (3) recognize and avoid the ecological fallacy. For content analysis, you should be able to (4) describe the process of selecting content for analysis and developing a code book, (5) differentiate between manifest and latent content, and (6) discuss ways of ensuring reliability and validity in content analysis research. Finally, (7) you should be able to characterize the basic process of conducting theory-driven comparative historical research.

  • Babbie & Roberts (2018) Chapter "Non-Reactive Research"

thursday feb 15 FIELD RESEARCH

By the end of this class, you should be able to (1) describe the advantages and disadvantages of ethnographic field methods and participatory action research, (2) demonstrate an understanding of how to address practical matters such as gaining entry to field sites, taking field notes, producing codes, evaluating the reliability and validity of ethnographic observation, and finally, (3) explain what is involved in conducting field research ethically.

In class, we will pick collaborators and chapters for the "class facilitation" deliverable, due on March 22 and 27.

  • Babbie & Roberts (2018) Chapter "Field Research"

tuesday feb 20 READING WEEK, NO CLASSES

  • None!

thursday feb 22 READING WEEK, NO CLASSES

  • None!

tuesday feb 27 QUALITATIVE INTERVIEWING

By the end of this class, you should be able to (1) explain how social scientists use qualitative interviews to produce qualitative data, including how they develop rapport with participants, (2) describe the typical structure and development of an interview-based research project, (3) explain the logic of "sequential" interviews, and (4) compare the strengths and limitations of focus groups over one-on-one interviews.

  • Babbie & Roberts (2018) Chapter "Qualitative Interviewing"

thursday mar 01 EVALUATION AND ACTION RESEARCH

By the end of this class, you should be able to (1) explain how social science methods can be used to evaluate the effectiveness of social programs, policies, or other interventions, (2) describe unique challenges of evaluation research (e.g. ethical and political barriers), and (3) compare qualitative and comparative approaches to evaluation research. Finally, (4) you will be able to differentiate between evaluation research and action research.

  • Babbie & Roberts (2018) Chapter "Evaluation Research"

tuesday mar 06 QUALITATIVE DATA ANALYSIS

By the end of this class, you should be able to (1) characterize different approaches to qualitative data analysis, (2) describe the iterative (and generally inductive) nature of qualitative data analysis, (3) correctly define and describe the grounded theory approach to qualitative data analysis.

  • Babbie & Roberts (2018) Chapter "Qualitative Data Analysis"

thursday mar 08 QUANTITATIVE DATA ANALYSIS

By the end of this class, you should be able to (1) define key concepts in basic quantitative data analysis (including concepts related to frequency distributions, central tendency and dispersion), (2) differentiate between types of quantitative variables, and (3) construct simple bivariate tables.

  • Babbie & Roberts (2018) Chapter "Quantitative Data Analysis"

tuesday mar 13 THE LOGIC OF MULTIVARIATE ANALYSIS

By the end of this class, you should be able to (1) describe the basic logic of multivariate analysis, including explaining what it means to "control" for a variable when analyzing the relationship between two other variables, (2) define and describe the elaboration model, and (3) differentiate between replication, explanation, interpretation, and specification.

  • Babbie & Roberts (2018) Chapter "The Logic of Multivariate Analysis"

thursday mar 15 SOCIAL STATISTICS

By the end of this class, you should be able to (1) understand and describe the difference between descriptive statistics and inferential statistics, (2) read and interpret a correlation matrix, (3) understand the logic of regression analysis, (4) understand the meaning of statistical significance as it relates to the likelihood that an observed relationship is due to sampling error, and (5) evaluate whether an association shown in a cross-tabulation is statistically significant using the Chi-square statistic.

  • Babbie & Roberts (2018) Chapter "Social Statistics"

tuesday mar 20 READING AND WRITING SOCIAL RESEARCH + DIGITAL DATA

By the end of this class, you should be able to (1) plan and conduct a literature review, (2) effectively read, assess, and communicate the key findings in a journal article without getting lost in details, and (3) outline and write a report presenting the results of empirical social research. (4) You will also be able to characterize the general opportunities and challenges for social research in the digital age.

  • Babbie & Roberts (2018) Chapter "Reading and Writing Social Research"
  • Salganik (2018) Chapter 1

thursday mar 22 DIGITAL DATA + OBSERVING BEHAVIOR + ASKING QUESTIONS

This class meeting will be facilitated by student groups!

By the end of this class, you should be able to (1) explain how digital trace data and administrative data are changing observational methods, and (3) describe changes in the logic of survey research in the digital age.

  • Salganik (2018) Chapters 2 - 3

tuesday mar 27 DIGITAL DATA + RUNNING EXPERIMENTS + MASS COLLABORATION

This class meeting will be facilitated by student groups!

By the end of this class, you should be able to (1) explain how social scientists use digital platforms to run randomized control experiments that get at complex causal processes, (2) describe how to create mass collaborations -- such as citizen science projects -- to address extremely challenging problems, and (3) be able to article the ethical challenges that are unique to, or amplified in, the context of research with digital data or using digital platforms.

  • Salganik (2018) Chapters 4 - 5

thursday mar 29 REVISITING ETHICS + THE FUTURE OF SOCIAL RESEARCH

By the end of this class, you should be able to (1) describe clear strategies for reasoning about ethics and making ethical research decisions in the digital age, and (2) clearly summarize how social research is and is not changing -- or should and should not change -- in the digital age.

  • Salganik (2018) Chapter 6 and 7

Course Policies

Submitting Work & Grading Process

I will only grade work that you submit electronically.

Text matching software will be used to screen assignments in this course. This is being done to verify that use of all materials and sources in assignments is documented. Students will be given an option if they do not want to have their assignment screened by Turnitin. If you do not wish to submit your work to Turnitin, you can schedule a meeting with me to discuss your submissions in person.

Late Policy

I will deduct 5 points a day for every day, or part of a day, that your work is late, including weekends. I will not make exceptions without a medical note.

Laptops and the Facebook Penalty

Laptops may be used in the classroom on the honors system, but you must sit in the designated laptop section. I reserve the right to modify this policy if laptops appear to be interfering with student learning. If I see Facebook, email, an IM client other than #slack, a newspaper story, a blog, or any other content not related to the class, I will remove 1 point from your participation grade on the spot.

Communication

We will be using the collaboration tool #slack for regular class communication. I use the do not disturb settings on #slack, so I will not see any messages you send me outside of normal working hours. You are free to email me, of course, but I tend to respond to slack messages from students faster than I respond to emails. There are free #slack apps for Mac OS X, Linux (beta), Windows (beta), iOS, and Android.

Preferred Chat System, by XKCD

Feedback

I will solicit brief, informal, and confidential course evaluations throughout the semester. These will only take a few minutes of your time. The purpose is to make sure that we are moving at a comfortable pace, that you feel you understand the material, and that my teaching style is meeting your needs. I will use this ongoing feedback to make adjustments as the course progresses. Although you are not obligated to do so, please fill out the evaluations so that I can make this the best learning experience for you, and the best teaching experience for me.

On Campus Resources

Access Ability Services

The AccessAbility Office, located in Needles Hall, Room 1132, collaborates with all academic departments to arrange appropriate accommodations for students with disabilities without compromising the academic integrity of the curriculum. If you require academic accommodations to lessen the impact of your disability, please register with the AccessAbility Office at the beginning of each academic term.

Mental Health

The University of Waterloo, the Faculty of Environment, and our Departments consider students' well-being to be extremely important. We recognize that throughout the term students may face health challenges -- physical and / or emotional. Please note that help is available. Mental health is a serious issue for everyone and can affect your ability to do your best work. Counselling Services is an inclusive, non-judgmental, and confidential space for anyone to seek support. They offer confidential counselling for a variety of areas including anxiety, stress management, depression, grief, substance use, sexuality, relationship issues, and much more.

The Writing Centre

Although I will be giving you feedback on your work throughout the term, I encourage you to make appointments with people at the writing centre. Their services are available to all UW students.

University Policies

Academic Integrity

In order to maintain a culture of academic integrity, members of the University of Waterloo community are expected to promote honesty, trust, fairness, respect and responsibility.

We will all uphold academic integrity policies at University of Waterloo, which include but are not limited to promoting academic freedom and a community free from discrimination and harassment. You can educate yourself on these policies -- and the disciplinary processes in place to deal with violations -- on the Office of Academic Integrity website.

A student is expected to know what constitutes academic integrity, to avoid committing academic offense, and to take responsibility for his/her actions. A student who is unsure whether an action constitutes an offense, or who needs help in learning how to avoid offenses (e.g., plagiarism, cheating) or about 'rules' for group work / collaboration should seek guidance from the course professor, academic advisor, or the Undergraduate Associate Dean. For information on categories of offences and types of penalties, students should refer to Policy 71, Student Discipline. For typical penalties, check Guidelines for Assessment of Penalties.

Grievances and Appeals

A student who believes that a decision affecting some aspect of his / her university life has been unfair or unreasonable may have grounds for initiating a grievance. Read Policy 70: Student Petitions and Grievances, Section 4. When in doubt please contact your Undergraduate Advisor for details.

A decision made or penalty imposed under Policy 70 -- Student Petitions and Grievances (other than a petition) or Policy 71 -- (Student Discipline) may be appealed if there is a ground. A student who believes he/she has a ground for an appeal should refer to Policy 72 (Student Appeals).

Religious Observances

Student needs to inform the instructor at the beginning of term if special accommodation needs to be made for religious observances that are not otherwise accounted for in the scheduling of classes and deliverables.