Orig­i­nal source pub­li­ca­tion: Brito, M. A. and F. de Sá-Soares (2014). Assess­ment Fre­quency in Intro­duc­tory Com­puter Pro­gram­ming Dis­ci­plines. Com­put­ers in Human Behav­ior 30, 623–628.
The final pub­li­ca­tion is avail­able here.

Assess­ment Fre­quency in Intro­duc­tory Com­puter Pro­gram­ming Dis­ci­plines

Miguel A. Brito and Fil­ipe de Sá-Soares

Depart­ment of Infor­ma­tion Sys­tems, Cen­tro Algo­ritmi, School of Engi­neer­ing, Uni­ver­sity of Minho, Guimarães, Por­tu­gal

1. Introduction—Reasons for Failure

The lit­er­a­ture iden­ti­fies sev­eral rea­sons for stu­dents fail­ure when learn­ing to pro­gram [Robins et al. 2003], with some address­ing the cur­ricu­lum, oth­ers focus­ing on method­olo­gies [Cook 2008], on resources, or on the best first pro­gram­ming lan­guage to use [Duke et al. 2000]. The inher­ent dif­fi­culty of pro­gram­ming [Teague and Roe 2008; Winslow 1996] and con­trol­ling what kind of mis­takes stu­dents are more likely to make [Spohrer and Soloway 1986] are also addressed.

How­ever, a well-known empir­i­cal truth about teach­ing pro­gram­ming is that a moti­vated stu­dent needs lit­tle guid­ance so he will suc­ceed no mat­ter how bad are the over­all con­di­tions, teach­ers included. Sim­i­larly, a stu­dent not moti­vated to spend some weekly hours prac­tic­ing will fail no mat­ter what the teacher says or how well the teacher explains all about com­puter pro­gram­ming.

Our teach­ing prac­tice also led us to note that many stu­dents do not have a real­is­tic idea about their effec­tive study per­for­mance.

• Some think that under­stand­ing the solu­tions pre­sented by the teacher or book is enough. They will prob­a­bly try to solve their first prob­lem dur­ing the assess­ment itself.

• Oth­ers go a lit­tle fur­ther and do some exer­cises but stop train­ing as soon as they reach a solu­tion for a cer­tain kind of prob­lem. How­ever, being able to solve a 10-min prob­lem in a cou­ple of hours is far from suf­fi­cient and prob­a­bly the stu­dent is not even sure how the solu­tion works.

• A more insid­i­ous kind of prob­lem emerges with stu­dents that always study in a (same) group. Fre­quently, what hap­pens is that although every­one fol­lows and con­trib­utes to the solu­tion, it is always the same stu­dent that per­forms the first step from the prob­lem state­ment. The oth­ers just fol­low the lead and gen­uinely believe they can alone solve the prob­lem from the very begin­ning.

  In this paper the main con­cern is to address the need to per­suade stu­dents to crit­i­cally analyse their study method­ol­ogy and progress over the semes­ter. So the focus is on fail­ure rea­sons that can be over­come by tun­ing com­puter pro­gram­ming method­ol­ogy of study.

Papers about dif­fi­cul­ties in com­puter pro­gram­ming learn­ing can be found in the lit­er­a­ture [Jenk­ins 2002]. In [Gomes and Mendes 2007] dif­fi­cul­ties are divided into five cat­e­gories:

• The teach­ing meth­ods—this addresses the lack of per­son­al­ized super­vi­sion and imme­di­ate feed­back. Teach­ers must ensure stu­dents fol­low the most appro­pri­ate learn­ing approach while respect­ing dif­fer­ent learn­ing styles, such as indi­vid­ual vs. group study.

• The focus on syn­tac­ti­cal details instead of pro­mot­ing prob­lem solv­ing capa­bil­i­ties also needs to be addressed.

• The study meth­ods—learn­ing com­puter pro­gram­ming is a very exper­i­men­tal and intense task, much dif­fer­ent of other dis­ci­plines stu­dents are used to that are based on for­mu­las or pro­ce­dures mem­o­riza­tion. Pro­gram­ming demands a lot of extra class work.

• The stu­dent’s abil­i­ties and atti­tudes—there is a stu­dents’ gen­er­al­ized lack of prob­lem solv­ing capa­bil­i­ties. They read the prob­lem descrip­tion instead of ana­lyz­ing it and jump to solve some­thing prior to under­stand the prob­lem. They also lose many learn­ing oppor­tu­ni­ties because when fac­ing a dif­fi­culty they sim­ply give up or ask for help which basi­cally leads to the same out­come.

• The nature of pro­gram­ming—the high level of abstrac­tion needed and the syn­tax com­plex­ity are also men­tioned as dif­fi­cul­ties.

  The job of pro­gram­ming requires skills such as ana­lyt­i­cal think­ing, cre­ative syn­the­sis, and atten­tion to detail. An addi­tional fun­da­men­tal skill is the abil­ity to abstract, con­sist­ing in ‘‘the process of iden­ti­fy­ing com­mon pat­terns that have sys­tem­atic vari­a­tions’’ [Gabriel 1989]. A pro­gram­mer needs to apply abstrac­tion when ana­lyz­ing com­pu­ta­tional prob­lems, as well as to instan­ti­ate abstract pro­gram­ming con­cepts and tech­niques to solve par­tic­u­lar com­pu­ta­tional prob­lems.

  The syn­tax com­plex­ity of pro­gram­ming lan­guages raises sev­eral dif­fi­cul­ties to novice pro­gram­ming stu­dents, since they may have to simul­ta­ne­ously strug­gle with the inher­ent com­plex­ity of the prob­lem that has to be solved and the syn­tac­ti­cal speci­fici­ties of the pro­gram­ming lan­guage used to con­vey the com­pu­ta­tional solu­tion to the prob­lem.

• Psy­cho­log­i­cal effects—the lack of moti­va­tion for sev­eral pos­si­ble rea­sons, being well known that an unmo­ti­vated stu­dent will hardly suc­ceed. This is aggra­vated by the fact that pro­gram­ming courses usu­ally gain a rep­u­ta­tion that passes from stu­dent to stu­dent of being dif­fi­cult that so many of them start already feel­ing defeated. There is also the fact that com­puter pro­gram­ming is usu­ally taught at the very begin­ning of higher edu­ca­tion courses, coin­cid­ing with a tran­si­tion and insta­bil­ity period in the stu­dents’ life [Teague 2008].

2. Constructivism in Education

Con­struc­tivism is a learn­ing the­ory in which Jean Piaget argues that peo­ple (and chil­dren in par­tic­u­lar) build their knowl­edge from their own expe­ri­ence rather than on some kind of infor­ma­tion trans­mis­sion.

Later, based on Piaget’s insights on the expe­ri­en­tial learn­ing par­a­digm, impor­tant works have been devel­oped, such as Kolb’s Expe­ri­en­tial Learn­ing Model [Kolb and Fry 1975] which rein­forces the role of per­sonal exper­i­ment in learn­ing and sys­tem­atizes iter­a­tions of reflec­tion, con­cep­tu­al­iza­tion, test­ing and back again to new expe­ri­ences. A rich set of works about con­struc­tivism in edu­ca­tion can be found in [Steffe and Gale 1995].

Mean­while, the dis­cus­sion was brought to the com­puter sci­ence edu­ca­tion field, with the claim that real under­stand­ing demands active learn­ing on a lab envi­ron­ment with teacher’s guid­ance for ensur­ing reflec­tion on the expe­ri­ence obtained from prob­lem solv­ing exer­cises. In other words, pas­sive com­puter pro­gram­ming learn­ing will likely be con­demned to fail­ure [Ben-Ari 1998; Had­jer­rouit 2005; Wulf 2005].

Indeed, effec­tive learn­ing accord­ing to the con­struc­tivist per­spec­tive demands the men­tal con­struc­tion of viable mod­els. As argued by [Lui et al. 2004], learn­ing to pro­gram is a dif­fi­cult endeav­our because the learn­ing process is sus­cep­ti­ble to sev­eral haz­ards. Although all stu­dents face these haz­ards [Lui et al. 2004] observe that weak stu­dents get stum­bled and stalled more eas­ily when they encounter such haz­ards. From our expe­ri­ence of teach­ing com­puter pro­gram­ming courses, we think that the char­ac­ter­is­tics of weak stu­dents that mag­nify the impacts of those learn­ing haz­ards are, to some extent, shared by
the aver­age novice stu­dent of com­puter pro­gram­ming—the stu­dent pop­u­la­tion tar­get of this paper. Those char­ac­ter­is­tics include lack in train­ing in abstrac­tion processes, lack of a prior foun­da­tion for anchor­ing the con­struc­tion of new knowl­edge, and low lev­els of con­fi­dence in them­selves, in the teach­ers, and in the study mate­ri­als or prac­tices.

The main rea­son for the authors’ con­cern with con­struc­tivism is the con­vic­tion that teach­ing must always be focused on the peo­ple learn­ing process. Even in orga­ni­za­tions in gen­eral, there is already the belief that the inte­gra­tion of knowl­edge has achieved lim­ited suc­cess pri­mar­ily because it has focused on treat­ing knowl­edge as a resource instead of focus­ing on the peo­ple learn­ing process [Grace and But­ler 2005].

3. Why Weekly Assessment

An inter­est­ing study that crosses stu­dents’ indi­vid­ual cog­ni­tive level with the types of errors made [Ran­jeeth and Naidoo 2007] sug­gests the need to adopt inno­v­a­tive strate­gies in order to counter the seem­ingly per­pet­ual rate of fail­ure and at the same time increase the inten­sity for stu­dents with bet­ter cog­ni­tive level.

What­ever the rea­son, the ulti­mate truth is that a lot of things can go wrong when learn­ing com­puter pro­gram­ming, espe­cially in under­grad­u­ate courses.

So it is vir­tu­ally impos­si­ble to pre­vent them all, mainly because stu­dents tend to over­es­ti­mate their own under­stand­ing [Lahti­nen et al. 2005] and usu­ally they are not very open to fol­low teacher’s good advices, espe­cially if fol­low­ing those advices means more work.

This leads to the only win­ning strat­egy we have found so far: fail fast to learn sooner. If fre­quent assess­ment oppor­tu­ni­ties are given to the stu­dent, no mat­ter the rea­son why he is even­tu­ally per­form­ing badly, two impor­tant goals are imme­di­ately achieved:

• there is still time to change stu­dent’s study method­ol­ogy and

• the teacher finally gets some real atten­tion from the stu­dent to his good advices.

Even for stu­dents who do not need to fail to cor­rect even­tual study errors, we observe that weekly assess­ment is an extra moti­va­tion for not post­pon­ing study and con­se­quently they will also attend next class bet­ter pre­pared [Becker and Devine 2007].

Other medium term issues related with tun­ing the dis­ci­pline from year to year are also addressed by weekly assess­ment. The order­ing of dif­fer­ent con­cepts by dif­fi­culty [Milne and Rowe 2002] can be inferred from final exam­i­na­tions or by directly ask­ing stu­dents and teach­ers, but if we have auto­mated weekly assess­ments this kind of data is read­ily avail­able. So it is eas­ier tun­ing the classes’ dis­tri­bu­tion along the year, ded­i­cat­ing more time and exer­cises to those issues we know stu­dents need more time to assim­i­late and even­tu­ally to con­cen­trate eas­ier sub­jects in less classes.

4. How Weekly Assessment

Weekly assess­ment involves sev­eral dimen­sions. First we need a method­ol­ogy and then imple­men­ta­tion resources. For the method­ol­ogy we have a set of sup­pos­edly good advices and tuto­r­ial sup­port dur­ing classes but in this con­text it is enough to focus on the method the stu­dents need to fol­low each week (those imple­men­ta­tion resources, advices and tuto­r­ial sup­port will be addressed in future research reports).

In the roadmap pre­sented by Lytras in his man­i­festo for knowl­edge and learn­ing [Lytras and Sicilia 2005], knowl­edge and learn­ing issues were arranged into five crit­i­cal pil­lars, namely objects, processes, strate­gies, sys­tems, and per­for­mance. One of these pil­lars—‘knowl­edge and learn­ing processes’–is here addressed with a sys­tem­atic method for guid­ing stu­dents. The other pil­lars are also present, but they are not a core issue in this paper.

4.1 Our Method

Our learn­ing method­ol­ogy includes a method of weekly work com­posed of the fol­low­ing steps:

• First read the book: the stu­dents should always read the main ref­er­ence of the course before attend­ing the class. This is a well known way of prepar­ing the brain to the con­cepts that will be dis­cussed in class. It is not intended or expected that the stu­dents under­stand every­thing but they will have a pre­cious idea of the main con­cepts and tech­niques addressed in the cur­rent week.

• Par­tic­i­pate in the class: in the sub­se­quent class, con­cepts and tech­niques will be put in con­text, exam­ples will be given and expected dif­fi­cul­ties will be dealt with. Although stu­dents need to prac­tice to learn to pro­gram, they should leave the class with the con­cepts and tech­niques well known. They now knowabout, they have an idea of how-to-do-it, they are pre­pared to go on and try to do it by them­selves.

• Try every­thing: then go back to the book and class notes and believe noth­ing—code every­thing and exper­i­ment in the pro­gram­ming envi­ron­ment. Revisit every­thing that was read and said, always cod­ing it. Intro­duce changes in the exer­cises and check if the out­come matches the expected result. Solve and run the asso­ci­ated exer­cises. The biggest prob­lem is not under­stand­ing the basic con­cepts or tech­niques but rather learn­ing to apply them [Lahti­nen et al. 2005].

• Use the lab­o­ra­to­r­ial classes to self assess­ment: a spe­cial set of exer­cises is weekly pre­pared for the stu­dents to per­form a self assess­ment. The lab­o­ra­to­r­ial class is usu­ally used for that pur­pose.

• Assess­ment: finally, the weekly assess­ment gives feed­back to the stu­dent about the effec­tive­ness of his work that week. This is very impor­tant for sev­eral rea­sons such as avoid­ing study relax­ation, cre­at­ing a timely alert to stu­dents’ inef­fec­tive study, and pin­point­ing the parts of the syl­labus that require revi­sion by the stu­dent.

• Tuto­r­ial guid­ance: with the feed­back pro­vided by the assess­ment it is cru­cial to dis­cuss the suc­cess of each stu­dent so far. Com­puter pro­gram­ming is to some exten­sion a yes or no com­pe­tence. This means that grades near zero and grades near hun­dred per cent are usu­ally nor­mal. So in the first weeks, grades bel­low 80% should be care­fully ana­lyzed—what hap­pened that jus­ti­fies that the stu­dent did not reach 100%? Not enough study? Not enough exper­i­men­tal prac­tice? Should he study not more but bet­ter? The role of the lab ses­sions teacher is cru­cial at this point. There are dif­fer­ent stu­dents’ capa­bil­i­ties and the teacher must have answers to all of them, even if they are spread by dif­fer­ent lev­els of Bloom’s tax­on­omy [Lis­ter and Leaney 2003]. He must diag­nose and address the per­sonal dif­fi­cul­ties of each stu­dent giv­ing him guid­ance and self con­fi­dence.

Besides the already men­tioned advan­tages from each step of the method, the whole set addresses main con­struc­tivism claims since it pro­vides sys­tem­atic pro­gres­sive inter­ac­tions with the same con­cepts and tech­niques under teacher’s guid­ance and con­se­quently facil­i­tat­ing stu­dents’ reflec­tion and appre­hen­sion.

4.2 Self Assessment

Stu­dents have a weekly project com­posed of exer­cises sim­i­lar to the ones in the weekly assess­ment. Once their weekly study is done, and only then, they try to solve all the ques­tions in order to self assess their knowl­edge.

In case of dif­fi­cul­ties they should go back to the book or ask teacher for guid­ance.

The res­o­lu­tion of the project is sub­mit­ted to teach­ers for con­trol pur­poses.

4.3 The Weekly Assessment

The weekly assess­ment is not just a weekly assess­ment—it is a weekly assess­ment with weekly feed­back, i.e., the main goal is not just to put some pres­sure on the stu­dents. It would not be the same if the stu­dents did weekly assess­ment but only obtain the grades at the end of the term. An impor­tant fea­ture is that each stu­dent gets imme­di­ate feed­back and can still cor­rect in time his study method­ol­ogy.

Unfor­tu­nately, it is incred­i­bly sim­ple for a very well inten­tioned stu­dent to diverge from the suc­cess path with­out notice.

We are deal­ing with novice first year under­grad­u­ate stu­dents and it is never too much to stress that we are deal­ing with novice first year under­grad­u­ate stu­dents... This means an extra charge of dis­trac­tions aside from the inher­ent adap­ta­tion issues.

Com­pletely out of that path of suc­cess is the temp­ta­tion to stop study­ing after under­stand­ing a prob­lem’s solu­tion, but before know­ing how to do it him­self. Even if the stu­dent gets to the solu­tion of a kind of prob­lem he must con­sider the time that was nec­es­sary to do it—usu­ally, it is still a long way between the points I-can-solve-it and I-can-solve-it-fast-enough.

It is also not rare that stu­dents study a lot and know very lit­tle, just because they are not doing it well. For instance, a stu­dent can spend daily hours study­ing syn­tax issues, solv­ing prob­lems and the­o­ret­i­cal ques­tions with­out really solv­ing any prob­lem him­self. This trans­lates into a defi­cient level of knowl­edge with the aggra­vat­ing cir­cum­stance of induc­ing the stu­dent in a pseudo state of accom­plish­ment.

Nev­er­the­less, if by any chance an already rare stu­dent per­ceives or sus­pects of flaws in his study method­ol­ogy the nor­mal ten­dency will be to post­pone even­tual cor­rec­tive mea­sures even if he knew one and gen­er­ally that is not the case.

But even worse than that is the sit­u­a­tion of a stu­dent that has the chance to have an atten­tive teacher that diag­no­sis one of those described issues and gives the stu­dent guid­ance and con­crete cor­rec­tive mea­sures... nev­er­the­less the stan­dard stu­dent will not trust the teacher’s diag­no­sis.

All these issues can be solved in time if one gives the stu­dent the oppor­tu­nity to fail fast if some­thing is going wrong. This oppor­tu­nity is called weekly assess­ment and in our view it con­sti­tutes the dif­fer­en­tial fac­tor that allows the stu­dent:

• To per­ceive that there is an issue to address.

• To believe the issue should be addressed.

• To believe the issue will not dis­ap­pear by itself.

• To address it in time.

Weekly assess­ment will give the stu­dent the oppor­tu­nity to cor­rect what­ever may be wrong with­out hav­ing to wait till next (year or semes­ter) edi­tion of the course.

4.4 Implementation Resources

Some authors ([Costello 2007; Duke et al. 2000]) agree on the mer­its of fre­quent assess­ment but have a small enough num­ber of stu­dents or a big enough num­ber of teacher x hours.

We do not have such resources and have almost 200 stu­dents and only a lab ses­sions teacher. Nev­er­the­less, a third fac­tor in the equa­tion may be auto­mated test­ing of the prob­lems solved by stu­dents.

As dis­cussed fur­ther in this doc­u­ment, the use of an expanded learn­ing man­age­ment sys­tem (LMS) helped solv­ing the lack of resources issue, but also turned out to be a cen­tral point of infor­ma­tion that the stu­dents appre­ci­ate. As con­cluded in [Grace and But­ler 2005], LMS ben­e­fits are wide and include:

• Users empow­er­ment in man­ag­ing their own learn­ing.

• Facil­i­tat­ing the cre­ation of learn­ing struc­tures and processes.

• Encour­ag­ing the rou­tin­i­sa­tion of learn­ing.

• Pro­mot­ing a learn­ing cul­ture.

5. Our outcomes with weekly assessment

Since we intro­duced weekly assess­ment the num­ber of retained stu­dents was dra­mat­i­cally reduced.

How­ever, this is not pos­si­ble with­out some extra resources. In the case we choose to invest in auto­mated assess­ment.

5.1 Addressed Reasons for Failure

Weekly assess­ment proved to be a pow­er­ful tool in address­ing com­puter pro­gram­ming learn­ing dif­fi­cul­ties. Even when not directly involved, some­times facil­i­tates deal­ing with those issues.

Fol­low­ing the above iden­ti­fied five cat­e­gories of rea­sons for com­puter pro­gram­ming learn­ing fail­ure, weekly assess­ment virtues are here ana­lyzed:

• The teach­ing meth­ods: the imme­di­ate feed­back pro­vided by auto­mated tests allows the teacher to analyse each stu­dent sit­u­a­tion indi­vid­u­ally and to give the con­se­quent best guid­ance to the stu­dent. Weekly assess­ment does not address the even­tual exces­sive focus on syn­tac­ti­cal details but gives the teacher the pos­si­bil­ity of being aware in time of stu­dent’s even­tual lack of prob­lem solv­ing capa­bil­i­ties. Any­way in this course syn­tax would never be an issue because of the pro­gram­ming lan­guage that is used.

• The study meth­ods: learn­ing com­puter pro­gram­ming is a very exper­i­men­tal and intense task. It is hard to keep the rhythm week after week because a lot of dis­trac­tions may occur in the stu­dent’s life. Weekly assess­ments mark the cadence and even in bad weeks the stu­dent will make an extra effort to keep
  the pace.

• The stu­dent’s abil­i­ties and atti­tudes: the lack of prob­lem solv­ing capa­bil­i­ties is indeed a major issue. Weekly assess­ment gives the teacher the oppor­tu­nity to repeat­edly, week after week, iden­tify stu­dents with this kind of dif­fi­culty and insist in ana­lyz­ing method­olo­gies that hope­fully will tend to pay still in time. Fast oppor­tu­nity of fail­ure is good to get the stu­dent atten­tion but also car­ries out repeated oppor­tu­ni­ties for chang­ing or improv­ing less pos­i­tive atti­tudes.

• The nature of pro­gram­ming: the high level of abstrac­tion needed and the syn­tax com­plex­ity have already been addressed in this dis­cus­sion. For those stu­dents with more dif­fi­culty at this level we do not know alter­na­tives to prac­tice a lot. In this sense, the rhythm imposed by weekly assess­ment is a great help, assist­ing in the for­ma­tion of sound study habits by the stu­dents.

• Psy­cho­log­i­cal effects: it is our expe­ri­ence that the above men­tioned psy­cho­log­i­cal effects may be addressed in great extent. Weekly grades force the stu­dents to talk and exchange expe­ri­ences which avoid silent post­pon­ing of anguish and suf­fer­ing. Once again it also fur­nishes the teacher with the knowl­edge and the oppor­tu­nity to quickly point the light at the end of the tun­nel.

5.2 Success Evolution

A first and strong indi­ca­tor of the weekly assess­ment suc­cess is the evo­lu­tion of the per­cent­age of stu­dents that stay till the end, i.e., that do not drop at the mid­dle of the semes­ter.

Table 1 shows this evo­lu­tion for the last eight years. The read­ing of this table is not com­pletely lin­ear because some other fac­tors changed mean­while, such as instruc­tors, pro­gram­ming lan­guage, and the redesign of the study plans due to the Bologna process. How­ever, the evo­lu­tion is quite clear and shows that the efforts in increas­ing the assess­ment fre­quency resulted in greater per­cent­age of stu­dents that stick till the end of semes­ter.

Table 1: Stu­dents that Stick Till the End of Semes­ter Per­cent­age Evo­lu­tion

In 2004/2005 a small project was quar­terly assessed; i.e., there were two assess­ment points per semes­ter.

Dur­ing 2005/2006 small prob­lems res­o­lu­tion in com­put­ers’ lab were added to the assess­ment on a monthly basis.

In 2006/2007 a weekly auto­mated the­o­ret­i­cal-prac­ti­cal (TP) assess­ment was imple­mented and com­ple­mented with a quar­terly lab­o­ra­to­r­ial (L) assess­ment.

Dur­ing 2007/2008 the fre­quency of lab­o­ra­to­r­ial assess­ment was increased to monthly.

Finally, 2008/2009 was the year when the whole assess­ment was auto­mated and weekly per­formed.

The ratio of approval is also grow­ing—4% in 2005/2006, 2% the year after, 1% in 2007/2008 and finally a huge 13% jump on 2008/2009 with the weekly fully auto­mated assess­ment—a 20% total improve­ment in five years! Since then, the ratio has sta­bi­lized in the range 75–80%. It should be men­tioned that this evo­lu­tion was achieved nei­ther by shrink­ing the syl­labus nor by decreas­ing the level of rigor imposed to the course over the years.

Another impor­tant indi­ca­tor is the num­ber of stu­dents being able to suc­ceed the dis­ci­pline on first attempt or bet­ter said the num­ber of attempts a stu­dent needs in order to suc­ceed.

For Table 2 con­struc­tion we con­sid­ered the divi­sion suc­cess-before and suc­cess-after weekly assess­ment imple­men­ta­tion.

Table 2: Per­cent­age of Approvals in First, Sec­ond and Third Attempts
(each attempt takes place in a dif­fer­ent aca­d­e­mic year)

Achiev­ing suc­cess in first attempt is very impor­tant for sev­eral rea­sons. There is an asso­ci­ated cost for the stu­dent which is both mon­e­tary and func­tional. For teach­ers and uni­ver­sity sec­ond attempts rep­re­sent an increased com­plex­ity in resources and man­age­ment.

Con­se­quently, a 21% incre­ment in first attempt suc­cess is a very pos­i­tive result.

5.3 Students’ Opinion

Weekly test­ing is a key fac­tor for suc­cess in our method­ol­ogy and it has proven results. Nev­er­the­less, results can still be rein­forced with the stu­dents’ per­cep­tions–after all they are a highly involved and inter­ested part in this process.

Our Farewell ques­tion­naire asks for the stu­dents’ opin­ion and they agree with us. When asked about, in the whole dis­ci­pline, what they con­sider to be the most pos­i­tive fac­tor, almost 25% of the stu­dents indi­cate weekly test­ing.

When directly asked about weekly test­ing, stu­dents are almost unan­i­mous in their answers as illus­trated in Table 3.

Table 3: Farewell Ques­tion—Impor­tance of Weekly Test­ing

The grow­ing strength of favor­able opin­ions about weekly test­ing is prob­a­bly influ­enced by the yearly improve­ment in the auto­mated test­ing plat­form. In fact, our pro­gram­ming exten­sion began to be just a sub­mit-once-with­out-feed­back scheme which pro­voked many wrong answers due to very small errors, such as syn­tac­ti­cal ones. I 2008/009 we achieved a ver­sion that allows mul­ti­ple solu­tion sub­mis­sions of the same prob­lem dur­ing the avail­able time. Although the sys­tem does not tell the stu­dent where the errors are located, it informs the stu­dent that the pro­posed solu­tion is not cor­rect, pro­vid­ing him with the oppor­tu­nity to search for and cor­rect mis­takes in the pro­gram­ming code, before grad­ing the answer.

We find auto­mated tests not to be the ideal solu­tion but a needed one. Nev­er­the­less, stu­dents appear to like or at least not to dis­like this solu­tion. To eval­u­ate the pos­si­bil­ity of doing man­ual assess­ment by enlarg­ing the inter­val between assess­ments, i.e., doing fewer assess­ments, we were sur­prised by the fact that just a minor­ity of stu­dents was against auto­mated test­ing even in the case of being pos­si­ble to per­form weekly man­ual assess­ments. Obvi­ously, as sug­gested by Table 4, that presents aver­age results from 2008/2009 to 2011/2012, for a total of 480 responses, more stu­dents pre­fer auto­mated tests if the alter­na­tive is to decrease the num­ber of assess­ments, but even with a large num­ber of man­ual assess­ments they still pre­fer auto­mated assess­ment.

Table 4: Farewell Ques­tion—Weekly Auto­mated vs. Man­ual Assess­ment

That is 64% of the stu­dents pre­fer a weekly auto­mated assess­ment than a man­ual one with a two weeks inter­val; only 15% pre­fer this man­ual assess­ment. When asked about a quar­terly man­ual assess­ment the per­cent­age of stu­dents pre­fer­ring auto­mated assess­ment grows to 70% and the per­cent­age against decreases to 12%.

These some­how unex­pected results led us to the intro­duc­tion of a new ques­tion in 2008/2009, to explic­itly ask stu­dents for their pre­fer­able way of assess­ment regard­less of any other con­sid­er­a­tions.

Using a Lik­ert scale with five posi­tions, from one (I totally dis­like) to five (It is my favourite assess­ment type) the stu­dents’ feed­back is resumed in Table 5 (data reports to last four years, for a total of 480 responses).

Table 5: Farewell Ques­tion—Favourite Assess­ment Type

This con­firmed stu­dents’ pref­er­ence for auto­mated tests assess­ment type. Although we do not have pro­found answers for these pref­er­ence (for instance, one could argue that a man­ual assess­ment of their answers should be more accu­rate and fair than an auto­mated assess­ment) the infor­mal feed­back we have from stu­dents indi­cates that the instan­ta­neous grade which auto­mated tests give is prob­a­bly the main rea­son for these pref­er­ence when com­pared with a man­u­ally assessed writ­ten exam­i­na­tion.

6. Final Remarks and Future Work

We find weekly assess­ment one of the most effec­tive ways of lead­ing stu­dents by the suc­cess path in the realm of intro­duc­tory com­puter pro­gram­ming dis­ci­plines.

With a sub­stan­tial num­ber of stu­dents and a reduced num­ber of teach­ers the only way to get it done demands auto­mated assess­ment help.

Sur­pris­ingly, what we ini­tially thought to be the lesser of two evils turned out to be per­ceived by stu­dents as an addi­tional ben­e­fit (cf. Table 4 dis­cus­sion).

The fre­quency of assess­ment that we advo­cate in intro­duc­tory com­put­ing dis­ci­plines is the nodal fac­tor of our approach to com­puter pro­gram­ming learn­ing. Some of the ben­e­fits we found with the appli­ca­tion of this teach­ing method­ol­ogy where addressed in this paper and may be syn­the­sized as fol­low­ing. First, it cre­ates a method­i­cal study rhythm among stu­dents which puts in cen­tre stage the need for prac­tice in know-how-to-do-it dis­ci­plines such as com­puter pro­gram­ming. Sec­ond, it allows the stu­dents to explic­itly acknowl­edge that they are not able to solve a pro­gram­ming prob­lem fast enough and by them­selves. Third, it cre­ates the oppor­tu­nity for the stu­dents, assisted by the teacher, to crit­i­cally analyse their study method­ol­ogy and to adjust it in a time­frame that max­i­mizes the prob­a­bil­ity of recov­ery. Fourth, it assists in the build­ing of stu­dents’ com­puter pro­gram­ming knowl­edge from their own expe­ri­ence, con­sol­i­dat­ing knowl­edge when the assess­ment is pos­i­tive and launch­ing the stu­dent in a knowl­edge recon­struc­tion process when the assess­ment is neg­a­tive.

All the exper­i­ments reported in this doc­u­ment have been sup­ported by the multi-par­a­digm pro­gram­ming lan­guage that brings addi­tional pos­i­tive syn­er­gies to the whole process, not dis­cussed in this paper. Next steps include instan­ti­at­ing the appli­ca­tion of the method to more clas­sic par­a­digms using C, Java and other imper­a­tive lan­guages.

Acknowledgements

We would like to express our grat­i­tude to Miss Sónia Valente, sec­re­tary of the Infor­ma­tion Sys­tems and Tech­nol­ogy Sys­tems under­grad­u­ate pro­gram board for her kind sup­port in obtain­ing some extra data about pro­gram past edi­tions.

This work is financed with FEDER funds by the Pro­grama Opera­cional Fatores de Com­pet­i­tivi­dade—COM­PETE and by Fun­dos Nacionais via FCT—Fun­dação para a Ciên­cia e Tec­nolo­gia for the project: FCOMP-01-0124-FEDER-022674.

References

• Becker, D. A. and M. Devine (2007). Auto­mated assess­ments and stu­dent learn­ing. Inter­na­tional Jour­nal of Learn­ing Tech­nol­ogy 3, 5–17.

  Ben-Ari, M. (1998). Con­struc­tivism in com­puter sci­ence edu­ca­tion. SIGCSE Bul­letin 30, 257–261.

  Cook, W. R. (2008). High-level prob­lems in teach­ing under­grad­u­ate pro­gram­ming lan­guages. SIG­PLAN Notices 43, 55–58.

  Costello, F. J. (2007). Web-based elec­tronic anno­ta­tion and rapid feed­back for com­puter sci­ence pro­gram­ming exer­cises. In O’neill, G., S. Hunt­ley-Moore and P. Race (Eds.), Case Stud­ies of Good Prac­tices in Assess­ment of Stu­dent Learn­ing in Higher Edu­ca­tion. Dublin.

  Duke, R., E. Salz­man, J. Burmeis­ter,J. Poon and L. Mur­ray (2000). Teach­ing pro­gram­ming to begin­ners—Choos­ing the lan­guage is just the first step. Pro­ceed­ings of the Aus­tralasian con­fer­ence on com­put­ing edu­ca­tion. Mel­bourne(Aus­tralia).

  Gabriel, R. P. (1989). Draft report on require­ments for a com­mon pro­to­typ­ing sys­tem. Sig­plan Notices 24, 93–165.

  Gomes, A. and A. J. Mendes (2007). Learn­ing to pro­gram—Dif­fi­cul­ties and solu­tions. Inter­na­tional Con­fer­ence on Engi­neer­ing Edu­ca­tion—ICEE 2007. Coim­bra (Por­tu­gal).

  Grace, A. and T. But­ler (2005). Learn­ing man­age­ment sys­tems: A new begin­ning in the man­age­ment of learn­ing and knowl­edge. Inter­na­tional Jour­nal of Knowl­edge and Learn­ing 1, 12–24.

  Had­jer­rouit, S. (2005). Con­struc­tivism as guid­ing phi­los­o­phy for soft­ware engi­neer­ing edu­ca­tion. SIGCSE Bul­letin 37, 45–49.

  Jenk­ins, T. (2002). On the dif­fi­culty of learn­ing to pro­gram. 3^rd Annual Con­fer­ence of LTSN-ICS.

  Kolb, D. and R. Fry (1975). Toward an applied the­ory of expe­ri­en­tial learn­ing. In Cooper, C. (Ed.), The­o­ries of Group Process. Lon­don: John Wiley.

  Lahti­nen, E., K. Ala-Mutka and H.-M. Järvi­nen (2005). A study of the dif­fi­cul­ties of novice pro­gram­mers. Pro­ceed­ings of the 10^th annual SIGCSE Con­fer­ence on Inno­va­tion and Tech­nol­ogy in Com­puter Sci­ence Edu­ca­tion. Caparica (Por­tu­gal).

  Lis­ter, R. and J. Leaney (2003). Intro­duc­tory pro­gram­ming, cri­te­rion-ref­er­enc­ing, and Bloom. Pro­ceed­ings of the 34^th SIGCSE Tech­ni­cal Sym­po­sium on Com­puter Sci­ence Edu­ca­tion. Reno, NV (USA).

  Lui, A. K., R. Kwan, M. Poon and Y. H. Y. Che­ung (2004). Sav­ing weak pro­gram­ming stu­dents: Apply­ing con­struc­tivism in a first pro­gram­ming course. SIGCSE Bul­letin 36, 72–76.

  Lytras, M. D. and M. A. Sicilia (2005). The knowl­edge soci­ety: A man­i­festo for knowl­edge and learn­ing. Inter­na­tional Jour­nal of Knowl­edge and Learn­ing 1, 1–11.

  Milne, I. and G. Rowe (2002). Dif­fi­cul­ties in learn­ing and teach­ing pro­gram­ming—views of stu­dents and tutors. Edu­ca­tion and Infor­ma­tion Tech­nolo­gies 7, 55–66.

  Ran­jeeth, S. and R. Naidoo (2007). An inves­ti­ga­tion into the rela­tion­ship between the level of cog­ni­tive matu­rity and the types of errors made by stu­dents in a com­puter pro­gram­ming course. Col­lege Teach­ing Meth­ods and Styles Jour­nal 3, 31–40.

  Robins, A., J. Roun­tree and N. Roun­tree (2003). Learn­ing and teach­ing pro­gram­ming: A review and dis­cus­sion. Com­puter Sci­ence Edu­ca­tion 13, 137–172.

  Spohrer, J. C. and E. Soloway (1986). Novice mis­takes: Are the folk wis­doms cor­rect? Com­mu­ni­ca­tions of the ACM 29, 624–632.

  Steffe, L. P. and J. Gale (Eds.) (1995). Con­struc­tivism in Edu­ca­tion. Hills­dale, NJ: Lawrence Erl­baum Asso­ci­ates.

  Teague, D. and P. Roe. (2008). Col­lab­o­ra­tive learn­ing: Towards a solu­tion for novice pro­gram­mers. Pro­ceed­ings of the Tenth Con­fer­ence on Aus­tralasian Com­put­ing Edu­ca­tion. Wol­lon­gong, NSW (Aus­tralia).

  Winslow, L. E. (1996). Pro­gram­ming ped­a­gogy—A psy­cho­log­i­cal overview. SIGCSE Bul­letin 28, 17–22.

  Wulf, T. (2005). Con­struc­tivist approaches for teach­ing com­puter pro­gram­ming. Pro­ceed­ings of the 6^th Con­fer­ence on Infor­ma­tion Tech­nol­ogy Edu­ca­tion. Newark, NJ (USA).