Research Process Guide

Quantitative research methods have a few designs to choose from, mostly rooted in the postpositivist worldview. The experimental design, quasi-experimental design and single subject experimental design (Bloomfield & Fisher, 2019; Creswell & Creswell, 2018). Single- subject or applied behavioral analysis consists of administering an experimental treatment to a person or small group of people over an extended period of time. Of the quasi experimental designs, subcategories; causal-comparative design and correlational design. Causal-comparative research allows for the investigator to compare two or more groups in terms of a treatment that has already happened. For correlational design, the researcher is looking to examine the relationship between variables or set of scores (Bloomfield & Fisher, 2019; Creswell & Creswell, 2018).

Generally, these kinds of designs fall into two categories, Survey Research and Experimental Research. Survey research uses a quantitative (numerical) description of trends, attitudes, opinions of a population by examining a sample of that population through questionnaires or structured interviews for data collection (Fowler, 2008; Fowler, 2014; Bloomfield & Fisher, 2019; Creswell & Creswell, 2018). These studies can be cross-sectional and longitudinal. Ultimately, the goal is to analyze the data and have the finding be generalizable to the entire population.

Experimental research uses the scientific method to determine if a specific treatment influences an outcome. This design requires random assignment of treatment conditions, and the quasi-experimental and single subject  version of this uses nonrandomized assignment of treatment (Bloomfield & Fisher, 2019).


Survey Methods

Survey research methods are widely used and follow a standard format.  Examining survey research in scholarly journals would be a great way to familiarize yourself with the format and determine how to do it and, more importantly, if this method is right for your research.

How to prepare to do survey research? Creswell and Creswell (2018) as well as Fowler (2014), have provided basic framework for the rationale of survey research that you consider as you make the decision abou what kind of methods you will employ to conduct your inquiry.

1. Identify the purpose of your survey research- what variables interest you? This means start sketching out a purpose statement such as “ The primary purpose of this study is to empirically evaluate whether the number of overtime hours predicts subsequent burnout symptoms in emergency room nurses” (Creswell & Creswell, 2018, p. 149).
2. Write out why a survey method is the appropriate kind of approach for your study. It may be beneficial to discuss the advantages of survey research and the disadvantages of other methods.
3. Decide whether the survey will be cross-sectional or longitudinal. Meaning, will you gather the data at the same time  or collect it over time?
4. How will the data be collected, meaning how will the survey be filled out? Mail, phone, internet, structured interviews? Please provide the rationale for your choice.
5. Discuss your population and sampling - who is the target population? What is the size? Who are they in terms of demographic information? How do you plan to identify individuals in this population? Random sampling or systematic sampling and what is the rationale behind your choice? You should really aim for a particular fraction of the population that is typical based on past studies conducted on this topic.
6. Complete a power analysis to determine your possible sample size (n). There are many free online resources to conduct a power analysis (see Kraemer & Blasey, 2015). Ultimately, whether you are completing survey research or experimental design you need to:
   • Determine the estimated size of the correlation (r) . Using our above example, you might be looking at the relationship between hours worked and burnout symptoms. This might be difficult to determine if no other studies have been completed with these two variables involved.
   • Determine the two tailed value (a) This is called a Type I error and deals with the risk associated with a false positive. Typically, the accepted Type 1 alpha value is set at 0.5%, meaning there is a 5% probability that there is a significant (non-zero) relationship between the two variables (number of hours worked and burnout symptoms.
   • A beta value (b) is called a Type II error which refers to the risk we take saying there is no significant effect when there is a significant effect (false negative) Beta value is commonly set at .20.
   • By plugging in these numbers, r, alpha, and beta into a power analysis tool, you will be able to determine your sample size.

Survey Instrument

As you determine what instrument you will use, a survey you create or that has been used and created by someone else, you should consider the following (Fowler, 2008; Creswell & Creswell, 2018; Bloomfield & Fisher, 2019):

1. Name and give credit to the instrument and the researchers who developed it.  Or discuss your use of proprietary or free survey products online (Qualtrics, Survey Monkey).
2. Consider the validity of the instrument in three areas:
   • Content validity (did the survey measure what it was intended to measure?)
   • Predictive validity (do scores predict a criterion measure? Do the scores correlate with other results?)
   • Construct validity (does the survey measure hypothetical concepts?)
3. Consider the reliability of the instrument:
   • What is the internal consistency of the survey? Does it perform in the same way with each variable and each item on the survey behaves in the same way? You can use the test-retest reliability, whether the instrument is stable over time.

Experimental Design

There are three components to experimental design which also follows a standard form: participants and design, procedure and measurement. There are a few considerations that Bouma et al. (2012), Bloomfield and Fisher (2019), Creswell and Creswell (2018) suggest you determine early on in your design.

1. Decide how you will recruit participants and describe your sampling techniques. Essentially, you need to make sure that you state the inclusion and exclusion characteristics. It would be important to review quantitative sampling techniques. The sampling techniques below are appropriate for experimental design. Experimental design is largely different from survey research because there is a requirement of random sampling and randomization of assignment. According to Sharma (2017), there are several kinds of sampling methods that can be used in quantitative design.
   • Random Sampling - the sampling technique in which each sample has an equal probability of being chosen and is meant to be an unbiased representation of the total population.
   • Quota Sampling - is defined as a non-probability sampling method in which researchers create a sample involving individuals that represent a population.
   • Convenience Sampling - defined as a method adopted by researchers where they collect market research data from a conveniently available pool of participants.
   • Probability Sampling - refers to sampling techniques which  are aiming to identify a representative sample from which to collect data.
2. The idea of randomized assignment is a distinct feature of experimental design. When participants are randomly assigned to groups, the process is called a true experiment. If this is the case with your study, you should discuss how, when and why you are assigning participants to treatment groups. You need to describe in detail how each participant is placed to eliminate systematic bias in assigning participants. If your study design deals with more than one variable or treatment that cannot utilize random assignment (i.e. female school children benefit from a different teaching technique than male school children), this would change your design from true experimental design to a quasi-experimental design.
3. As with survey research, it would be essential to conduct a power analysis for sample size. The steps for power analysis are the same as survey design, however the focus for a power analysis of experimental design is about measuring effect size, meaning the estimated differences between groups of manipulated variables of interest. Please review steps  for power analysis in the survey research section.
4. Identify variables in the study, specifically the dependent and independent variables, as well as any other variables you intend to measure in the study. For example, you might want to think about participant demographic variables, variables that might impact your study design like time of day (i.e. energy levels might fluctuate during the day so that could impact measurement) and lastly, other variables that might impact your study’s outcomes.

Instrumentation

Just like with survey research, it is important to discuss how you are collecting your data, through what instrument or instruments are used, what scales are used, what their reliability and validity are based on past uses (Bouma et al., 2012; Creswell & Creswell, 2018; Bloomfield and Fisher, 2019). Ultimately, some quantitative experimental models may use  data sets that have already been collected like the National Center for Educational Statistics (NCES). In that case, you will be able to discuss the validity and reliability easily as it is well-established. However, if you are collecting your own data, you must discuss in detail what materials are used in the manipulation of variables. For example, you might want to pilot test the experiment so you have a detailed knowledge of the procedure (Bouma et al., 2012; Creswell & Creswell, 2018).

Also, often in experimental design, you don’t want the participants to know which variables are being manipulated or which group they are being assigned to. In order to be sure you are in line with IRB regulations (See IRB section), you want to draft a letter that will be used to explain the procedures and the study’s purpose to the participants (Creswell & Creswell, 2018). If there is any deception used in the study, be sure to check the IRB guidelines to ensure that you have all procedures and documents approved by Kean University’s IRB.


Measurement and Data Analysis for Quantitative Methods

It is important to reiterate that there are several kinds of ways to collect data for a quantitative study. The data is always numerical, as opposed to qualitative data, which is largely narrative. The most common data collection methods for quantitative research are:

1. Close-ended surveys
2. Polls
3. Close-ended questionnaires
4. Structured interviews

The data is collected across populations, using a large sample size and then is analyzed using statistical analysis. Then, the results would be generalizable across populations. However, before you collect the data, you need to determine what exactly you are proposing to measure as you choose your variables and you. There are several kinds of statistical measurements in quantitative research. Each has its own purpose and objective. Ultimately, you need to decide if you are going to describe, explain, predict, or control your numerical data.

Quantitative data collection typically means there are a lot of data. Once the data is gathered, it may seem to be messy and disorganized at first. Your job as the researcher is to organize and then make the significance of the data clear. You do this by cleaning your data through “measurements” or scales and then running statistical analysis tests through your statistical analysis software program.

There are several purposes to statistical analysis in a quantitative study, such as (Kumar, 2015):

1. Summarize your data by identifying what is typical and what is atypical within a group.
2. Identify the rank of an individual or entity within a group
3. Demonstrate the relationship between or among variables.
4. Show similarities and differences among groups.
5. Identify any error that is inherent in a sample.
6. Test for significance.
7. Can support you in making inferences about the population being studied.

It is important to know that in order to properly analyze your numerical data, you will need access to statistical analysis software such as SPSS. The OCIS Help Desk website provides information on how to access SPSS under the Remote Learning (Students) section.

Once you have collected your numerical data, you can run a series of statistical tests on your data, depending on your research questions.

There are four kinds of statistical measurements that you will be able to choose from in order to determine the best statistical tests to be utilized to explore your research inquiry. These measurements are also referred to as scales, and have very particular sets of statistical analysis tools that go along with each kind of scale (Bryman & Cramer, 2009).

1. Nominal
2. Ordinal
3. Interval
4. Ratio

Nominal measurements are labels (names, hence nominal) of  specific categories within mutually exclusive populations or treatment groups. These labels delineate non-numerical data such as gender, city of birth, race, ethnicity, or marital status (Bryman & Cramer, 2009; Ong & Puteh, 2017).

Ordinal measurements detail the order in which data is organized and ranked. These measures or scales deal with the greater than( >)compared to those less than (<) within a data set. Again, these are organized (named/ categorized)  and ranked (ordinal), such as class rank, ability level (beginner, intermediate, expert), or Likert scale answers (strongly agree, agree, undecided, disagree, strongly disagree) (Bryman & Cramer, 2009; Ong & Puteh, 2017).  

Interval measurements take data and order them (nominal), rank them (ordinal) and then evenly distribute them in equal intervals. There is also a zero point which is established. deal with equal units where a zero point is established. Interval measurements can be used for height, weight where there would be an absence of one of those variables (Bryman & Cramer, 2009; Ong & Puteh, 2017).

Ratio measurements allow for data to be measured by equal units (interval) and an absolute zero point is established. Here, in ratio measurements, the absolute zero value signifies the absence of the variable. For example, 0 lbs means the absence of weight. Height, weight, temperature are all examples of variables that can be measured through ratio scale (Bryman & Cramer, 2009; Ong & Puteh, 2017).

Bloomfield, J., & Fisher, M. J. (2019). Quantitative research design. Journal of the Australasian Rehabilitation Nurses Association, 22(2), 27-30. https://doi-org.kean.idm.oclc.org/10.33235/jarna.22.2.27-30

Bouma, G. D., Ling, R., & Wilkinson, L. (2012). The research process (2nd Canadian ed.). Oxford University Press.

Bryman, A., & Cramer, D. (2009). Quantitative data analysis with SPSS 14, 15 & 16: A guide for social scientists. Routledge/Taylor & Francis Group.

Creswell, J. W., & Creswell, J. D. (2018). Research design: Qualitative, quantitative, and mixed methods approaches. Sage.

Fowler, F. J., Jr. (2008). Survey research methods (4th ed.). Sage.

Fowler, F. J., Jr. (2014). The problem with survey research. Contemporary Sociology, 43(5): 660-662.

Kraemer, H. C., & Blasey, C. (2015). How many subjects?: Statistical power analysis in research. Sage.

Kumar, S. (2015). IRS introduction to research in special and inclusive education. [PowerPoint slides 4, 5, 37, 38, 39,43]. Informační systém Masarykovy univerzity. https://is.muni.cz/el/1441/podzim2015/SP_IRS/

Ong, M. H. A., & Puteh, F. (2017). Quantitative data analysis: Choosing between SPSS, PLS, and AMOS in social science research. International Interdisciplinary Journal of Scientific Research, 3(1), 14-25.

Sharma, G. (2017). Pros and cons of different sampling techniques. International Journal of Applied Research, 3(7), 749-752.