Orig­i­nal source pub­li­ca­tion: Brito, M. A. and F. de Sá-Soares (2010). Com­puter Pro­gram­ming: Fail Fast to Learn Sooner. In Lytras, M. D., P. O. De Pab­los, D. Avi­son, J. Sip­ior, Q. Jin, W. Leal, L. Uden, M. Thomas, S. Cer­vai and D. Horner (Eds.), Pro­ceed­ings of the First Inter­na­tional Con­fer­ence TECH-EDU­CA­TION 2010—Tech­nol­ogy Enhanced Learn­ing: Qual­ity of Teach­ing and Edu­ca­tional Reform, 223–229. Athens (Greece). Com­mu­ni­ca­tions in Com­puter and Infor­ma­tion Sci­ence 73. Hei­del­berg: Springer-Ver­lag. ISBN: 978-3-642-13165-3.
The final pub­li­ca­tion is avail­able here.

Com­puter Pro­gram­ming: Fail Fast to Learn Sooner

Miguel A. Brito and Fil­ipe de Sá-Soares

Uni­ver­sity of Minho, Depart­ment of Infor­ma­tion Sys­tems, Guimarães, Por­tu­gal

1. Reasons for Failure

The lit­er­a­ture iden­ti­fies sev­eral rea­sons for stu­dents fail­ure when learn­ing to pro­gram [Robins et al. 2003], some address the cur­ricu­lum, oth­ers focus on method­olo­gies [Cook 2008], resources or the best first pro­gram­ming lan­guage to use [Duke et al. 2000]. The inher­ent dif­fi­culty of pro­gram­ming [Teague and Roe 2008; Winslow 1996] and con­trol­ling what kind of mis­takes stu­dents are more likely to make [Spohrer and Soloway 1986] are also addressed.

How­ever, a well known truth about teach­ing pro­gram­ming is that a moti­vated stu­dent needs lit­tle guid­ance so he will suc­ceed no mat­ter how bad are the over­all con­di­tions, teach­ers included; also a stu­dent not moti­vated to spend some weekly hours prac­tic­ing will fail no mat­ter what the teacher says or how well the teacher explains all about com­puter pro­gram­ming.

Another con­clu­sion our teach­ing expe­ri­ence lead to is that many stu­dents do not have a real­is­tic idea about their effec­tive study per­for­mance.

• Some think that under­stand­ing the solu­tions pre­sented by the teacher or book is enough. They will prob­a­bly try to solve their first prob­lem dur­ing the assess­ment itself.

• Oth­ers go a lit­tle fur­ther and do some exer­cises but stop train­ing as soon as they reach a solu­tion for a cer­tain kind of prob­lem. How­ever, being able to solve a ten-minute prob­lem in a cou­ple of hours is far from suf­fi­cient and prob­a­bly the stu­dent is not even sure how the solu­tion works.

• A more insid­i­ous kind of prob­lem emerges with stu­dents that always study in a (same) group. Fre­quently, what hap­pens is that although every­one fol­lows and con­trib­utes to the solu­tion, it is always the same stu­dent that per­forms the first step from the prob­lem state­ment. The oth­ers just fol­low the lead and gen­uinely believe they can alone solve the prob­lem from the very begin­ning.

Papers about dif­fi­cul­ties in com­puter pro­gram­ming learn­ing can be found in the lit­er­a­ture [Jenk­ins 2002]. In [Gomes and Mendes 2007], these dif­fi­cul­ties are divided in cat­e­gories. In this paper the main con­cern is to address the need to per­suade stu­dents to crit­i­cally analyse their study method­ol­ogy and progress over the semes­ter. So the focus is on fail­ure rea­sons that can be over­lapped by tun­ing com­puter pro­gram­ming ethod­ol­ogy of study.

2. Constructivism in Education

Con­struc­tivism is a learn­ing the­ory in which Jean Piaget argues that peo­ple (and chil­dren in par­tic­u­lar) build their knowl­edge from their own expe­ri­ence rather than on some kind of infor­ma­tion trans­mis­sion.

Later, based on Piaget and oth­ers’ work with the expe­ri­en­tial learn­ing par­a­digm, impor­tant works have been devel­oped, such as Kolb’s Expe­ri­en­tial Learn­ing Model [Kolb and Fry 1975], which rein­forces the role of per­sonal exper­i­ment in learn­ing and sys­tem­atizes iter­a­tions of reflec­tion, con­cep­tu­al­iza­tion, test­ing and back again to new expe­ri­ences. A rich set of works about con­struc­tivism in edu­ca­tion can be found in [Steffe and Gale 1995].

Mean­while the dis­cus­sion was brought to the com­puter sci­ence edu­ca­tion field claim­ing that real under­stand­ing demands active learn­ing on a lab envi­ron­ment with teacher’s guid­ance for ensur­ing reflec­tion on the expe­ri­ence obtained from prob­lem solv­ing exer­cises. Pas­sive com­puter pro­gram­ming learn­ing will likely be con­demned to fail­ure [Ben-Ari 1998; Had­jer­rouit 2005; Wulf 2005]. Indeed, con­struc­tivism can even be used to explain the prob­lem of weak stu­dents and be part of the solu­tion [Lui et al. 2004].

3. Why Weekly Assessment

An inter­est­ing study that crosses stu­dents’ indi­vid­ual cog­ni­tive level with the types of errors made [Ran­jeeth and Naidoo 2007] sug­gests the need to adopt inno­v­a­tive strate­gies in order to counter the seem­ingly per­pet­ual rate of fail­ure and at the same time increase the inten­sity for stu­dents with bet­ter cog­ni­tive level.

What­ever the rea­son, the ulti­mate truth is that a lot of things can go wrong when learn­ing com­puter pro­gram­ming, espe­cially in under­grad­u­ate courses.

So it is vir­tu­ally impos­si­ble to pre­vent them all, mainly because stu­dents tend to over­es­ti­mate their own under­stand­ing [Lahti­nen et al. 2005] and usu­ally they are not very open to fol­low your good advices, espe­cially if fol­low­ing those advices means more work.

This leads to the only win­ning strat­egy we have found so far: fail fast to learn sooner. If fre­quent assess­ment oppor­tu­ni­ties are given to the stu­dent, no mat­ter the rea­son why he is even­tu­ally per­form­ing badly, two impor­tant goals are imme­di­ately achieved:

• there is still time to change stu­dent’s study method­ol­ogy and

• the teacher finally gets some real atten­tion from the stu­dent to his good advices.

Even for stu­dents who do not need to fail to cor­rect even­tual study errors, we observe that weekly assess­ment is an extra moti­va­tion for not post­pon­ing study and con­se­quently they will also attend next class bet­ter pre­pared [Becker and Devine 2007].

Other medium term issues related with tun­ing the dis­ci­pline from year to year are also addressed by weekly assess­ment. The order­ing of dif­fer­ent con­cepts by dif­fi­culty [Milne and Rowe 2002] can be inferred from final exam­i­na­tions or by directly ask­ing stu­dents and teach­ers, but if we have auto­mated weekly assess­ments this kind of data is all over. So it is eas­ier tun­ing the classes’ dis­tri­bu­tion along the year, ded­i­cat­ing more time and exer­cises to those issues we know stu­dents need more time to assim­i­late and even­tu­ally to con­cen­trate eas­ier sub­jects in less classes.

4. How Weekly Assessment

Weekly assess­ment involves sev­eral dimen­sions. First we need a method­ol­ogy and then imple­men­ta­tion resources. For the method­ol­ogy we have a set of sup­pos­edly good advices and tuto­r­ial sup­port dur­ing classes but in this con­text it is enough to focus on the method the stu­dents need to fol­low each week.

4.1 Our Method

Our learn­ing method­ol­ogy includes a method of weekly work com­posed of the fol­low­ing steps:

• First read the book: The stu­dents should always read the main ref­er­ence of the course before attend­ing the class. This is a well known way of prepar­ing the brain to the con­cepts that will be dis­cussed in class. It is not intended or expected that the stu­dents under­stand every­thing but they will have a pre­cious idea of the main con­cepts of the cur­rent week.

• Par­tic­i­pate in the class: In the sub­se­quent class con­cepts will be con­tex­tu­al­ized, exam­ples will be given and expected dif­fi­cul­ties will be addressed. Although stu­dents need to prac­tice to learn to pro­gram, they should leave the class with the con­cepts well known. They now know-about, they have an idea of how-to-do-it, they are pre­pared to go on and try to do it by them­selves.

• Try every­thing: Then go back to the book and class notes and believe noth­ing—code every­thing and exper­i­ment in the pro­gram­ming envi­ron­ment. Revisit every­thing that was read and said, always cod­ing it. Intro­duce changes in the exer­cises and check if the out­come matches the expected result. Solve and run the asso­ci­ated exer­cises. The biggest prob­lem is not under­stand­ing the basic con­cepts but rather learn­ing to apply them [Lahti­nen et al. 2005].

• Use the lab­o­ra­to­r­ial classes to self assess­ment: A spe­cial set of exer­cises is weekly pre­pared for the stu­dents to per­form a self assess­ment. The lab­o­ra­to­r­ial class is usu­ally used for that pur­pose.

• Assess­ment: Finally, the weekly assess­ment gives feed­back to the stu­dent about the effec­tive­ness of his work that week. This is very impor­tant for sev­eral rea­sons such as avoid­ing study relax­ation, cre­at­ing a timely alert to stu­dents’ inef­fec­tive study and pin­point the parts of the syl­labus that require revi­sion by the stu­dent.

• Tuto­r­ial guid­ance: With the feed­back pro­vided by the assess­ment it is cru­cial to dis­cuss the suc­cess of each stu­dent so far. Com­puter pro­gram­ming is in some exten­sion a yes or no com­pe­tence. This means that grades near zero and grades near hun­dred per cent are usu­ally nor­mal. So in the first weeks, grades bel­low 80% should be care­fully analysed—what hap­pened that jus­ti­fies that the stu­dent did not reach 100%? Not enough study? Not enough exper­i­men­tal prac­tice? Should he study not more but bet­ter? The role of the lab­o­ra­to­r­ial teacher is cru­cial at this point. There are dif­fer­ent stu­dents’ capa­bil­i­ties and the teacher must have answers to all of them even if they are spread by dif­fer­ent lev­els of Bloom’s tax­on­omy [Lis­ter and Leaney 2003]. He must diag­nose and address the per­sonal dif­fi­cul­ties of each stu­dent giv­ing him guid­ance and self con­fi­dence.

Besides the already men­tioned advan­tages from each step of the method, the whole set addresses main con­struc­tivism claims since pro­vides sys­tem­atic pro­gres­sive inter­ac­tions with the same con­cepts with teacher’s guid­ance and con­se­quently facil­i­tat­ing stu­dents’ reflec­tion and appre­hen­sion.

4.2 Self Assessment

Stu­dents have a weekly project com­posed of exer­cises sim­i­lar to the ones in the weekly assess­ment. Once their weekly study is done, and only then, they try to solve all the ques­tions in order to self assess their knowl­edge. In case of dif­fi­cul­ties they should go back to the book or ask teacher for guid­ance.

The res­o­lu­tion of the project is sub­mit­ted to teach­ers for con­trol pur­poses.

4.3 The Weekly Assessment

The weekly assess­ment is not just a weekly assess­ment–it is a weekly assess­ment with weekly feed­back, i.e. the main goal is not just to put some pres­sure on the stu­dents. It would not be the same if the stu­dents did weekly assess­ment but only obtain the grades at the end. An impor­tant fea­ture is that each stu­dent has imme­di­ate feed­back and can still cor­rect in time his study method­ol­ogy.

Unfor­tu­nately it is incred­i­bly sim­ple for a very well inten­tioned stu­dent to diverge from the suc­cess path with­out notice.

We are deal­ing with novice first year under­grad­u­ate stu­dents and it is never too much to stress that we are deal­ing with novice first year under­grad­u­ate stu­dents... This means an extra charge of dis­trac­tions aside from the inher­ent adap­ta­tion issues.

Com­pletely out of that path of suc­cess is the temp­ta­tion to stop study­ing after under­stand­ing a prob­lem’s solu­tion, but before know­ing how to do it him­self. Even if the stu­dent gets to the solu­tion of a kind of prob­lem he must con­sider the time that was nec­es­sary to do it—it is still a long way (ok, maybe not always that long) between the points I-can-solve-it and I-can-solve-it-fast-enough.

It is also not rare that stu­dents study a lot and know very lit­tle, just because they are not doing it well. For instance, a stu­dent can spend daily hours study­ing syn­tax issues, solv­ing prob­lems and the­o­ret­i­cal ques­tions with­out really solv­ing any prob­lem him­self... big mis­take!

Nev­er­the­less, if by any chance an already rare stu­dent per­ceives or sus­pects of flaws in his study method­ol­ogy the nor­mal ten­dency will be to post­pone even­tual cor­rec­tive mea­sures even if he knew one and gen­er­ally that is not the case.

But even worse than that is the sit­u­a­tion of a stu­dent that has the chance to have an atten­tive teacher that diag­no­sis one of those described issues and gives the stu­dent guid­ance and con­crete cor­rec­tive mea­sures... nev­er­the­less the stan­dard stu­dent will not trust the teacher’s diag­no­sis.

All this issues can be solved in time if you give the stu­dent the oppor­tu­nity to fail fast if some­thing is going wrong. This oppor­tu­nity is called weekly assess­ment and is the main secret that allows the stu­dent: (i) to per­ceive that there is an issue to address, (ii) to believe the issue should be addressed, (iii) to believe the issue will not dis­ap­pear by itself, and (iv) to address it in time.

Weekly assess­ment will give the stu­dent the oppor­tu­nity to cor­rect what­ever may be wrong with­out hav­ing to wait till next (year or semes­ter) edi­tion of the course.

4.4 Implementation Resources

Some authors ([Costello 2007] and [Duke et al. 2000]) agree on the mer­its of fre­quent assess­ment but have a small enough num­ber of stu­dents or a big enough num­ber of hours x teach­ers.

We do not have such resources and have almost two hun­dred stu­dents and only a lab­o­ra­to­r­ial teacher. Nev­er­the­less, a third fac­tor in the equa­tion may be auto­mated test­ing of the prob­lems solved by stu­dents.

5. Our Outcomes with Weekly Assessment

Since we intro­duced weekly assess­ment the num­ber of retained stu­dents was dra­mat­i­cally reduced.

How­ever, this is not pos­si­ble with­out some extra resources. In the case we choose to invest in auto­mated assess­ment.

A first and strong indi­ca­tor of the weekly assess­ment suc­cess is the evo­lu­tion of the per­cent­age of stu­dents that stay till the end, i.e. that do not drop at the mid­dle of the semes­ter.

Table 1 shows this evo­lu­tion for the last five years. The read­ing of this table is not com­pletely lin­ear because some other fac­tors changed mean­while. How­ever, the evo­lu­tion is quite clear and shows that the efforts in increas­ing the assess­ment fre­quency resulted in greater per­cent­age of stu­dents that stick till the end of semes­ter.

Table 1. Stu­dents that stick till the end of semes­ter per­cent­age evo­lu­tion

In 2004/2005 a small project was quar­terly assessed; i.e. there were two assess­ment points per semes­ter.

Dur­ing 2005/2006 small prob­lems res­o­lu­tion in com­put­ers’ lab were added to the assess­ment on a monthly basis.

In 2006/2007 a weekly auto­mated the­o­ret­i­cal-prac­ti­cal (TP) assess­ment was imple­mented and com­ple­mented with a quar­terly lab­o­ra­to­r­ial (L) assess­ment.

Dur­ing 2007/2008 the fre­quency of lab­o­ra­to­r­ial assess­ment was increased to monthly.

Finally, 2008/2009 was the year when the whole assess­ment was auto­mated and weekly per­formed.

The ratio of approval is also grow­ing—4% in 2005/2006, 2% the year after, 1% in 2007/2008 and finally a huge 13% jump last year with the weekly fully auto­mated assess­ment—a 20% total improve­ment in five years! It should be men­tioned that this evo­lu­tion was achieved nei­ther by shrink­ing the syl­labus nor by decreas­ing the level of rigor imposed to the course over the years.

Another impor­tant indi­ca­tor is the num­ber of stu­dents being able to suc­ceed the dis­ci­pline on first attempt or bet­ter said the num­ber of attempts a stu­dent needs in order to suc­ceed.

For Table 2 con­struc­tion we con­sid­ered the divi­sion suc­cess-before and suc­ces­safter weekly assess­ment imple­men­ta­tion.

Table 2. Per­cent­age of approvals in first, sec­ond and third attempts

Achiev­ing suc­cess in first attempt is very impor­tant for sev­eral rea­sons. There is an asso­ci­ated cost for the stu­dent which is both mon­e­tary and func­tional. For teach­ers and uni­ver­sity sec­ond attempts rep­re­sent an increased com­plex­ity in resources and man­age­ment.

Con­se­quently, a twenty per cent incre­ment in first attempt suc­cess is a very pos­i­tive result.

Acknowledgments

We would like to express our grat­i­tude to Miss Sónia Valente, sec­re­tary of the Tech­nolo­gies and Infor­ma­tion Sys­tems under­grad­u­ate course board for her kind sup­port in obtain­ing some extra data about course past edi­tions.

References

• Becker, D. A. and M. Devine (2007). Auto­mated assess­ments and stu­dent learn­ing. Inter­na­tional Jour­nal of Learn­ing Tech­nol­ogy 3, 5–17.

  Ben-Ari, M. (1998). Con­struc­tivism in com­puter sci­ence edu­ca­tion. SIGCSE Bul­letin 30, 257–261.

  Cook, W. R. (2008). High-level prob­lems in teach­ing under­grad­u­ate pro­gram­ming lan­guages. ACM SIG­PLAN Notices 43(11), 55–58.

  Costello, F. J. (2007). Web-based elec­tronic anno­ta­tion and rapid feed­back for com­puter sci­ence pro­gram­ming exer­cises. In O’neill, G., S. Hunt­ley-Moore and P. Race (Eds.), Case Stud­ies of Good Prac­tices in Assess­ment of Stu­dent Learn­ing in Higher Edu­ca­tion. Dublin.

  Duke, R., E. Salz­man, J. Burmeis­ter, J. Poon and L. Mur­ray (2000). Teach­ing pro­gram­ming to begin­ners—choos­ing the lan­guage is just the first step. Pro­ceed­ings of the Aus­tralasian Con­fer­ence on Com­put­ing Edu­ca­tion 2000. Mel­bourne (Aus­tralia).

  Gomes, A. and A. J. Mendes (2007). Learn­ing to pro­gram—Dif­fi­cul­ties and solu­tions. Inter­na­tional Con­fer­ence on Engi­neer­ing Edu­ca­tion—ICEE 2007. Coim­bra (Por­tu­gal).

  Had­jer­rouit, S. (2005). Con­struc­tivism as guid­ing phi­los­o­phy for soft­ware engi­neer­ing edu­ca­tion. SIGCSE Bul­letin 37, 45–49.

  Jenk­ins, T. (2002). On the dif­fi­culty of learn­ing to pro­gram. 3^rd Annual Con­fer­ence of LTSN-ICS.

  Kolb, D. and R. Fry (1975). Toward an applied the­ory of expe­ri­en­tial learn­ing. In Cooper, C. (Ed.), The­o­ries of Group Process. Lon­don: John Wiley.

  Lahti­nen, E., K. Ala-Mutka and H.-M. Järvi­nen (2005). A study of the dif­fi­cul­ties of novice pro­gram­mers. Pro­ceed­ings of the 10^th annual SIGCSE Con­fer­ence on Inno­va­tion and Tech­nol­ogy in Com­puter Sci­ence Edu­ca­tion. Caparica (Por­tu­gal).

  Lis­ter, R. and J. Leaney (2003). Intro­duc­tory pro­gram­ming, cri­te­rion-ref­er­enc­ing, and Bloom. Pro­ceed­ings of the 34^th SIGCSE Tech­ni­cal Sym­po­sium on Com­puter Sci­ence Edu­ca­tion. Reno, NV (USA).

  Lui, A. K., R. Kwan, M. Poon and Y. H. Y. Che­ung (2004). Sav­ing weak pro­gram­ming stu­dents: Apply­ing con­struc­tivism in a first pro­gram­ming course. SIGCSE Bul­letin 36, 72–76.

  Milne, I. and G. Rowe (2002). Dif­fi­cul­ties in learn­ing and teach­ing pro­gram­ming—views of stu­dents and tutors. Edu­ca­tion and Infor­ma­tion Tech­nolo­gies 7, 55–66.

  Ran­jeeth, S. and R. Naidoo (2007). An inves­ti­ga­tion into the rela­tion­ship between the level of cog­ni­tive matu­rity and the types of errors made by stu­dents in a com­puter pro­gram­ming course. Col­lege Teach­ing Meth­ods and Styles Jour­nal 3, 31–40.

  Robins, A., J. Roun­tree and N. Roun­tree (2003). Learn­ing and Teach­ing Pro­gram­ming: A Review and Dis­cus­sion. Com­puter Sci­ence Edu­ca­tion 13(2), 137–172.

  Spohrer, J. C. and E. Soloway (1986). Novice mis­takes: Are the folk wis­doms cor­rect? Com­mu­ni­ca­tions of the ACM 29, 624–632.

  Steffe, L. P. and J. Gale (Eds.) (1995). Con­struc­tivism in Edu­ca­tion. Hills­dale, NJ: Lawrence Erl­baum Asso­ci­ates.

  Teague, D. and P. Roe (2008). Col­lab­o­ra­tive learn­ing: towards a solu­tion for novice pro­gram­mers. Pro­ceed­ings of the Tenth con­fer­ence on Aus­tralasian Com­put­ing Edu­ca­tion 78. Wol­lon­gong (Aus­tralia).

  Winslow, L. E. (1996). Pro­gram­ming ped­a­gogy—A psy­cho­log­i­cal overview. SIGCSE Bul­letin 28, 17–22.

  Wulf, T. (2005). Con­struc­tivist approaches for teach­ing com­puter pro­gram­ming. Pro­ceed­ings of the 6^th Con­fer­ence on Infor­ma­tion Tech­nol­ogy Edu­ca­tion. Newark, NJ (USA).