Study of APSA results in FHMS

This is a quantitative analysis of the academic / scholarship program of FHMS. In order to gauge the effectiveness of FHMS especially in delivering instruction, FHMS opted to employ the services of APSA (Asian Psychological Services and Assessment Corporation).

The 30 students of FHMS – Malolos were tested before they were accepted to the scholarship program. Chosen from more than 100 applicants, these students were accepted after submission of their card for grade 3, and subjected to interview by teachers. Their homes were also visited by our Social Worker, who rated their economic and social adequacy.

Before the classes started this June, these students were again tested on their academic aptitude in English (Language Arts), Science and Math.



Statement of the problem:

1. Are the increases in SAS (Scaled Ability Score) significant for a) English b) Science, and c) Math?
2. Are the increases in PR (Percentile Ranking) significant for a) English b) Science, and c) Math?
3. What among the following factors explain
i) high APSA SAS and
ii) high APSA SAS increase?

a. Gender
b. Age
c. Economic Adequacy
d. Social Adequacy
e. Teacher Factor/Grouping
f. Academic Performance in Grade 3
g. APSA Scores in Grade 3

4. Are there significant relationships among the following factors (taken two at a time or bivariate)?
a. English Grade 3 Grade
b. Math Grade 3 Grade
c. Science Grade 3 Grade
d. APSA English post Grade 3
e. APSA Math post Grade 3
f. ILA Grade for First Trimester
g. Math Grade for First Trimester
h. Science Grade for First Trimester
i. General Average for First Trimester
j. Percentile Rank – Star Reading (Renaissance Learning)
k. Percentile Rank – Star Math (Renaissance Learning)
l. APSA Grade 4 Pretest SAS in English
m. APSA Grade 4 Pretest SAS in Science
n. APSA Grade 4 Pretest SAS in Math
o. APSA Grade 4 Midterm SAS in English
p. APSA Grade 4 Midterm SAS in Science
q. APSA Grade 4 Midterm SAS in Math
5. Multiple Correlation – Predicting APSA Success
a. What predicts high SAS in English?
b. What predicts high SAS in Math?
c. What predicts high SAS in Science?
d. What predicts success in academics (high General Average)?
e. What predicts high conduct grade?
6. What is the path analysis of the success in APSA scoring, as demonstrated by high SAS scores?


Tools Used:

I used SPSS (Statistical Package for the Social Sciences) 12.0, and Microsoft Excel 2003 to perform the following statistical treatment and analysis:

For number 1 and 2, Paired-wise Sample T-test;
For number 3, One-way ANOVA and Chi-Square;
For number 4, Pearson correlation
For number 5, Multiple Correlation / Linear Regression (stepwise)
For number 6, Path Analysis


Results:


1. a. Are the increases in SAS (Scaled Ability Score) significant for English?

Yes. The mean SAS for ILA increased 9% from 69.17 to 75.59. Paired-wise T-test for the paired scores yielded a p-value of 9.78 E -09, an error below 0.01%. Statistically speaking, we are sure that the two scores are significantly different 99.99%.

The distribution of students based on proficiency level (as reported by APSA) will point out the differences in nominal scale:

Proficiency Level June 2006 October 2006
Proficient 0 6
Progressing Towards Standards 12 24
Standards Not Met 17 0

b) Are the increases in SAS (Scaled Ability Score) significant for Science?

Yes. The mean SAS for Science increased 10% from 67.48 to 74.21. Paired-wise t-test for the paired scores yielded a p-value of 3.16 E -07, an error below 0.01%. Statistically speaking, we are sure that the two scores are significantly different 99.99% of the time.

The distribution of students based on proficiency level (as reported by APSA) will point out the differences in nominal scale:

Proficiency Level June 2006 October 2006
Proficient 0 5
Progressing Towards Standards 8 22
Standards Not Met 21 3

b) Are the increases in SAS (Scaled Ability Score) significant for Math?

Yes. The mean SAS for Math increased 10% from 71.24 to 78.13. Paired-wise t-test for the paired scores yielded a p-value of 2.73 E -08, an error below 0.01%. Statistically speaking, we are sure that the two scores are significantly different 99.99%.

The distribution of students based on proficiency level (as reported by APSA) will point out the differences in nominal scale:

Proficiency Level June 2006 October 2006
Proficient 2 13
Progressing Towards Standards 19 16
Standards Not Met 8 1


2. Are the increases in PR (Percentile Ranking) significant for a) English b) Science,
and c) Math?

For the questions above, the answers are all YES. PR ended up for English 122% (from 15.17 to 33.69), 124% for Science (13.38 to 29.97), and 103% (21.48 to 43.55) for Math. T-test gives significant differences for the three paired variables, with p-values lesser than 0.000. Hence, as above, we are 99.99% sure that the paired scores are significantly different.

It is also considered that FHMS’ student scores represent the aptitude for a grade 4 student in his/her 5th month. The standards used by APSA are based on students who have finished grade 4 already.

In fact, in the attached Excel file, I studied each APSA skill and found out that most of the wrong items are those items that have not been tackled yet in the classroom, following scope and sequence. Amazingly, there are items where FHMS scored better against the standards, even for those which have not tackled by all the groups especially in Math. (Please see attached file Apsa analysis of skills mastered, an excel file)


3. What among the following factors explain
i. high APSA SAS and
ii. high APSA SAS increase?

a. Gender – Although the girls have outdone the boys in APSA English, APSA Math, APSA Science, General Average, Star Reading PR, and Star Math PR, the difference cannot be attributed apart from chance. We cannot reject the null hypothesis; hence, there is no significant difference between the scores of the boys and the girls.

b. Age – For the age groups, we have three age groups in FHMS – Malolos: the 9 year olds (N=7), the 10 year olds (N=17) and the 11 year olds (N=6). We see here that the 9 year olds have the highest scores in all tests and the 11 year olds, the lowest. The differences in scores for the three groups are significant for the following: APSA Science (p=0.016), Math (p=0.048), Star Reading (p=0.034, the 9-year olds are better reader than the 11 year olds, and this difference is significant), and the General Average for First Trimester (p=0.024).
The nine year olds are the students who are early schoolers. In contrast, the 11 year olds are those students who have not started school early, or those who have not been promoted quickly. For example, Angela, our first honor last trimester is only 9, and she said she was promoted from kinder to grade 1. On the contrary, most of our 11 year olds are those who are struggling in academics (Moran, Macasiray)

c. Economic Adequacy
The result is a non-significant verdict. Economic Adequacy cannot explain high SAS in Science, English and Math.

d. Social Adequacy
Not significant for all APSA subjects.

e. Teacher Factor/Grouping.

Every child was factored as to their teachers for the first five months. For example, Aguillon started ILA last June with Teacher G (high group). When transfers were made to other groups last August, he wasn’t transferred. Then, after the first trimester test, he was transferred to Teacher M (higher group) where he stayed for two months. Hence, for the teacher factor, Aguillon stayed with Teacher G 60% of the time, and with Teacher M 40% of the time. His SAS increase in ILA is 4 points. This will reflect with Teacher G as 2.4 (4 x 60%) and with Teacher M as 1.6 (4 x 40%). This is done for Math and ILA, computing only the increases in SAS and in Percentile Ranking.

Chi-square was used to verify whether there is consistency in the share of each teacher in the increase, taking into consideration the time each student stayed with them.

For ILA, total increase for all students amounted to 186 points. Teacher G’s share is 52.59 (there was a period of time when the high group numbered only 6), Teacher M 73.12 (she handled more students), and Teacher A 60.29. Factoring all the increases of students to the shares of each teacher who handled each of them, we came up with a 67 pt increase in G, 79.4 in M, and 39.6 in A. Dividing the total time the students were with them, we have an average increase in G of 8.17, M, 6.96 and A, 4.21.

Chi-square checked this against the theoretical probability scores for each teacher share and the result is a significant verdict with p = 0.003. Hence, for ILA, teacher is a factor.

For Math, the result is more or less the same. Average increase in Teacher G is 8.55, M 5.6 and A 5.76. Chi square analysis gives a verdict of significant difference, with p = 0.014. Hence, allowing 1.4% error, we are sure that teacher is a factor in Math APSA results.

I used only the SAS increase because considering SAS per se will all the more give higher score to G in Math, who handled the highest group. It will also skew Teacher A in ILA who handled the highest group 60% of the time, or to Teacher G, who handled them 40% of the time.

f. Academic Performance in Grade 3
Not significant enough for English (p=0.064), Science (p=0.869) and Math (p=0.425).

g. APSA Scores in Grade 3
Not significant enough, with p-value of 0.101 for English SAS. For Science SAS, it is significant (p=0.023). The highest performers in APSA Grade 3 scored an average of 78.83 as compared to the lowest performers from APSA Grade 3, whose average is at 71.5 only.

It is also significant for Math SAS, with the scores coming from the 3rd group (those who ranked 13th to 18th in APSA grade 3) outperforming even the 1st group (those who ranked 1st to 6th in APSA grade 3). Their average is 82, as compared to 78.03 by all the groups. The 1st group (those who ranked 1st to 6th in APSA Grade 3) has an average of 80. This may prove that FHMS is not catering only for the brightest. Indeed, those who perform very well in our midterm APSA test were not always the best from grade 3. Neither are they the ones who did very well in APSA grade 3.

4. Are there significant relationships among the following factors (taken two at a time or bivariate)?

Before I give the correlations, let me define the terms here. “Correlated” in this part of the study means that there is a significant (p < 0.05, or 95% certainty) relationship between the two variables. Now, “highly correlated” will mean that the relationship between the two variables is highly significant with p < 0.01, or 99% certainty.
a. English Grade 3 Grade. Highly correlated to Math grade 3, Science grade 3, APSA English post grade 3, Science grade for first trimester, General Average for first trimester, APSA Grade 4 English pretest. Correlated with ILA grade for first Trimester, Math grade for first Trimester, Percentile Rank for Star Math, APSA Math 4 Pretest, APSA English SAS Midterm and APSA Math SAS Midterm.

Here, it would appear that one of the best predictors of success in our students is their grade in English from grade 3. This is understandable since our class revolves around the Language class, in accordance to the SIOP background. Even Science success in class is presupposed by the student success in English grade 3. Now, we also see a consistency of grades in grade 3. The grades of Math, Science and English are all highly correlated against each other. Also, it has significant relationships with all the APSA test results for both pretest and midterm test.


b. Math Grade 3 Grade. Highly correlated with Science grade 3, Science grade for first trimester, General Average for first trimester, APSA Grade 4 ILA Pretest. Correlated with Math grade for first trimester, PR Rank – Star Math, APSA Grade 4 Math Pretest, and APSA Grade 4 Math Midterm SAS.

No new knowledge here. As in English 3 grade, Math 3 grades predict success in APSA and Academics.

c. Science Grade 3 Grade. Highly correlated with Science grade for First Trimester, and APSA Grade 4 English Pretest. Correlated with APSA English post grade 3 test, Math grade for first trimester, General Average for first trimester, APSA Grade 4 English Mid SAS.

Same as in Math 3 and English 3.

d. APSA English post Grade 3. Correlated with Science grade for first trimester. Highly correlated with APSA English midterm SAS. Both English APSA tests are correlated, which is understandable.

e. APSA Math post Grade 3. Highly correlated with ILA grade for first trimester, APSA Grade 4 Science Midterm SAS, and APSA Grade 4 Math Midterm SAS.

The result here points out that those who were good in math in grade 3 (evidenced by our APSA grade 3 posttest) are most likely to get high SAS for Science and Math in the midterm. Science and Math are of course related with each other, especially in the measurement part.

f. ILA Grade for First Trimester. Highly correlated with Math, Science and General Average for First Trimester, Percentile Ranking for Star Reading, APSA Midterm SAS for Math, APSA Midterm SAS for Science, and APSA Midterm SAS for English. It is also correlated with Percentile Rank in Star Math.

This will point out that our grading system is consistent with the true aptitude of the students, and their classroom performance predicts APSA success or failure.

g. Math Grade for First Trimester. Highly correlated with Science grade, General Average, PR in Star Math, PR in Star Reading, APSA Midterm Math and APSA Midterm Science. It is also correlated with APSA Midterm English (p=0.03).

Again, we have the same conclusion as above. Math grade for first trimester is representative of the students’ aptitude, not only in Math, but also in Science and English.

h. Science Grade for First Trimester. Highly correlated with General Average for First Trimester, PR in Star Math, PR in Star Reading, APSA Midterm SAS for Science, Math and English.

Science grade seems to be most representative of the student’s APSA success in academics and a good predictor of APSA success.

i. General Average for First Trimester. Highly correlated with Star Reading, Star Math, APSA Midterm SAS for Science, English and Math
j. Percentile Rank – Star Reading (Renaissance Learning). Correlated with Star Math, Math SAS Midterm APSA. Highly correlated with English and Science APSA Midterm SAS
k. Percentile Rank – Star Math (Renaissance Learning). Highly correlated with English and Math SAS APSA Midterm. Correlated with Science SAS APSA Midterm.
l. APSA Grade 4 Pretest SAS in English. Not correlated
m. APSA Grade 4 Pretest SAS in Science. Highly correlated with Science and Math Midterm SAS APSA.
n. APSA Grade 4 Pretest SAS in Math. Correlated with APSA English Midterm SAS.
o. APSA Grade 4 Midterm SAS in English. Highly Correlated with Science APSA Midterm SAS
p. APSA Grade 4 Midterm SAS in Science. Highly Correlated with Math APSA Midterm SAS
q. APSA Grade 4 Midterm SAS in Math. Follows from above.
5. Multiple Correlation – Predicting APSA Success

I used multiple correlation using linear regression analysis (SPSS-assisted, stepwise) in order to identify the best predictors for APSA high SAS. I considered many models, as will be clear in the attachment (Predicting APSA Success2.doc) and the model presented here is the best model.
a. What predicts high SAS in English?
The best model is the one with the following factors as predictors:
APSA English 3 post grade 3 and Percentile Rank in Star Reading. This model has an R of 0.719 and an R-square of 0.516. This means that this regression line accounts for the variance in the variables 51.60% of the time. It is significant with p=0.000

Coefficients are: Star Reading 0.782, APSA Grade 3 English, 0.321. The coefficient of the constant is 49.691. All are significant with p < 0.01

Hence, APSA SAS English = 0.782 Star Reading + 0.321 APSA Grade 3 English + 49.691

Another model to be considered uses the General Average for the First Trimester, although the coefficients given to Star Reading is higher (0.564), followed by General Average (0.291). Lastly, we have APSA Grade 3 English (0.268). This simply means that in predicting English SAS APSA, it is more important to note their score in Star Reading. Then, we must consider their First Trimester General Average. Lastly, we must take a look at their previous APSA English results from Grade 3.

b. What predicts high SAS in Math?

The only predictor given by the Linear Regression is the General Average for the First Trimester. This regression line has an R of 0.786, an R-square of 0.618, and is significant with p=0.000. This means that this regression line allows error of 0.000+ % only, and can account for the variance of the variable 61.8% of the time.

Coefficient of General Average is 1.360 (Significant with p=0.000). Constant is at -34.111 (significant, p=0.05). Hence, regression line is

APSA Science SAS = 1.36 General Average – 34.111

c. What predicts high SAS in Science?

Again, amazingly, the only predictor given by the Linear Regression is the General Average for the First Trimester. This regression line has an R of 0.760, an R-square of 0.577, and is significant with p=0.000. This means that this regression line allows error of 0.000+ % only, and can account for the variance of the variable 57.7% of the time.

Coefficient of General Average is 1.118 (Significant with p=0.000). Constant is at -17.981 (not significant, p=0.247). Hence, regression line is

APSA Science SAS = 1.118 General Average – 17.981

d. What predicts success in academics (high General Average)?
e. What predicts high conduct grade?
6. What is the path analysis of the success in APSA scoring, as demonstrated by high SAS scores?