Differences in the use of electronic and printed versions of a university mathematics workbook

The recent months have shifted contact teaching to the online environment and distance learning and students are dealing more and more with digital materials in various e-learning systems. The question is whether the online electronic materials are as effective as their printed versions for the students using them for self-study purposes. This paper presents research focusing on university students' work with an electronic and printed version of a mathematics workbook. The main research focuses on differences regarding error rate, the number of used hints, and the time they need to spend to solve 111 mathematical problems covering four topics of their introductory course of Mathematics such as limits, graphs, differentiation, and applications of derivatives. One hundred fifty-seven university students participated in the research working with sets of mathematical problems with multi-choice answers taken from the Khan Academy, including step-by-step hints. At the same time, the students were recording their errors, time, and the number of used hints using a questionnaire. The electronic sets were transformed into an electronic workbook and afterward into a printed version of this workbook. Obtained data were analysed using the Random Mixed Model as it enables to mix the used mathematical problems with different variance. The most exciting finding of this research was that the students working with the electronic version of the workbook work significantly faster but at the expense of errors. Students working with the interactive version of the workbook used significantly fewer hints.


INTRODUCTION
With today's massive spread of electronic media into schools, we question whether new technologies will not suppress classic textbooks, blackboards, and chalk. It has been many The offer of digital teaching and learning materials spreads worldwide in two ways. On the one hand, more and more publishers issue besides printed textbooks also their digital versions (as the most common format -PDF). On the other hand, an increasing number of teaching portals offer various interactive materials, including Computer-Aided Assessment (CAA). Students praise their immediate feedback and possibilities of repeated practicing with different variables or parameters, teachers, them, the easy variability when preparing tests for their students.
Our research focuses on students' practice at home during their introductory mathematics course using digital interactive tests and different aspects between the printed and electronic versions of a mathematics workbook. First, let us start to deal with mathematical texts generally. Nebeský [23] focused on the specifics of the Czech mathematical text. He examined the functioning of natural language in the mathematics environment and tried to set borders between the natural and artificial aspects. Units of the mathematical text, where their linguistic function is precisely given, he calls rigid, while the other units he calls live. He claims that thanks to the richness of the Czech language, the same expressions may function in mathematical texts as a rigid unit and a life unit, always depending on its particular meaning. In this case, it is necessary to replace life expressions with a synonym or to reformulate the given sentence.
On the contrary, rigid units have to be left. Authors of mathematical texts face a choice as to which content they should use brief expressions based on life units or if they should use a ponderous formulation based on rigid units. Moreover, they have to consider carefully if the consequence of thoughts is helpful sufficiently for the readers to understand the text's content correctly. Writing mathematical texts means mastering a complex apparatus in which content, methodological, pedagogical, and psychological components intermingle. It is crucial to find their mutual harmony.
The specificity of mathematical texts considering their comprehension was investigated by Fang & Schleppegrell [11]. They claim that it is essential to understand mathematical terminology and comprehend everyday words and their meanings for solving mathematical problems. On the other hand, the understanding of everyday vocabulary is not sufficient for language comprehension in mathematics. Morgan, Craig & Schuette [21] concluded the same together with Abedi & Lord [1], who conducted an exhaustive study focusing on testing word problems in which 1,174 students, for whom English is no their mother tongue, participated. The students were given the origins of word problems and then edited versions with more straightforward vocabulary and shorter sentences. They recorded an average improvement in performance in mathematics in the case of more than 1,000 students. The improvement was statistically significant with students with a lower command of English, a lower socialeconomic status, and more unsatisfactory performance in mathematics.
The structure of the mathematical text should not be underestimated when assigning test problems. Gueudet & Trouche [14] claim that it is essential to combine suitably three important components when testing students in mathematics. They mention the material component (paper, computer), the mathematical content component (terminology, tasks, and techniques), and finally, the didactical component (organizational elements and effective planning of the mathematical subject matter). Also, the organization committee of the international Kangaroo of Mathematics Competition investigates formulations of mathematics competition problems every year in detail as these problems are then translated into many other languages. The translations have to be made very carefully, knowing both languages well, but it is also necessary to understand the language of mathematics well.
The fact that mathematicians think very carefully about every word they use in their texts was verified by research conducted by Shanahan, Shanahan & Misischia [29], who made a profound inter-subject study of reading as a tool for the development of suitable teaching strategies. On top of all that, they proved that the successful performance in solving mathematical problems goes hand in hand with the language mastery of the students and the complexity of the text.
Dostal & Robinson [8] define in their paper, called Doing Mathematics with Purpose: Mathematical Text Types, four types of mathematical texts (proof text, algorithmic text, algebraic/symbolic text, and visual text), their purpose and their key functions. Dostal & Robinson [8] investigated mathematical literacy, too. They claim that mathematics learning includes reading and writing various types of mathematical texts that may be control, algorithmic, algebraic (symbolic), and visual. Shanahan, Shanahan & Misischia [29] investigated the differences in how chemists, historians, and mathematicians read text specific to their disciplines. Unlike the chemists and historians, the mathematicians in their study did not consider sources when reading and evaluating a text or another visual element. They believe these elements as unified and identically necessary.
The sets of problems in our research consisted of rather algebraic and visual texts. The control text was partially present in particular problems. In our study, we consider all types of the mathematical text of the test problems as unified. We do not distinguish whether the problems' assignment was algebraic or visual in the consequent statistical analysis.
Computers provide a range of opportunities for developing more interactive, authentic, and engaging tests [33], they are also increasingly used in the workplace and in everyday life to deal with problems involving numbers, quantities, two or three-dimensional figures, and data. It is also important to point out that some computer-based tasks cannot exist in a paper test because of their response format (e.g., "drag and drop"), or they require students to use the computer as a mathematical tool by interacting with the stimulus to solve a mathematics problem. However, in our study, we consider the used mathematical problems assigned in the form of paper and on the computer as identical with no unique response format in the electronic version. Therefore, there was no need to take the transferability into account, as mentioned, for example, by Lenhard, Schroeders & Lenhard [17] or Noyes & Garland [24]. Jahodová Berková [15] deals with the contribution of CAA in the teaching and learning of mathematics from university students' point of view and concludes the potential of CAA may be seen, above all, in its formative assessment and online basis. The students think that CAA systems should be used mainly to revise newly acquired subject matter but certainly not for the summative testing. On the other hand, teachers see the most important benefit in testing their students. Formative tests (in the form of homework) may monitor improvements during the learning process. In contrast, the summative tests (as final examination tests) function as an assessment tool at the end of the teaching and learning process.
Besides the mentioned factors, there are many other advantages and disadvantages of CAA. Repeated testing problems should improve students' performance, but only if their attitude to the tests is not based on trial and error. If there is no supervision of the students, for example, at home, it is hard to prevent cheating. Therefore, formative testing should help the students consolidate their knowledge and prepare them for a final examination. For this reason, they do not tend to cheat. However, it is crucial to mention that Axtell & Curran [3] claim that if students do not make notes and comments while solving CAA systems problems, they cannot use homework to help them acquire better the subject matter.
Our research is based on voluntary testing of the knowledge our students learned during weekly lectures and seminars. We were interested in the level of acquiring the theory and the ability to solve various mathematical problems during the pandemic period. To see more deeply into the acquisition level, we also recorded the number of hints the students used during the solving process. Unfortunately, we have not found any relevant research dealing with step-by-step hints in solving mathematics problems, representing the interactive structural component of the mathematical apparatus (Zujev [39]; Průcha [25]; Krotký & Mach [16]).
To conclude the introduction part, we would like to say that in the case of any testing in mathematics (formative, summative, paper-based, or digital), it is necessary to provide students with sufficient technical conditions, to formulate unambiguously and precisely mathematical problems and to pay always attention to the mathematical syntax. In the case of summative testing, it is crucial to set the time limit appropriately and select such testing problems so that they cover the required subject matter and may reveal possible defects in students' understanding. The question is if the time limit should be the same as students are taking the same test at school. Another question is if students make mistakes to the same extent when dealing with the paper-based and digital tests. These are the reasons we deal with these two factors (time and error rate) factors in comparing the paper-based and digital formative testing in our study, as we consider that the findings may be beneficial to the teaching practice.

MATERIAL AND METHODS
Our study was based on voluntary formative testing of our students' knowledge during weekly lectures and seminars. We were interested in the level of acquiring the theory and the ability to solve various mathematical problems during the pandemic period. To see more deeply into the acquisition level, we also recorded the number of hints the students used during the solving process. We selected the testing problems used in our study according to the following requirements: digital materials are available in the English and Czech languages, they contain step-by-step hints, they enable student's automatic assessment, and they cover the selected topics of our course of Mathematics. Having explored several educational portals, we chose the open educational resource Khan Academy which meets all the mentioned criteria. Schwartz [30] summarized five key observations about authentic understanding: thanks to the pedagogical experience of the author, Khan Academy is a suitable basis for authentic understanding stabilized with practical examples and problems to solve, offers relevant feedback, it is context-sensitive, and the particular pieces of knowledge are ordered hierarchically.
In the presented study were formulated the following research questions:  Does the workbook version influence the students' error rate when solving the assigned mathematical problems?  Does the workbook version influence the number of the used hints the students need to solve the assigned mathematical problems?  Does the workbook version influence the time the students need to solve the assigned mathematical problems?
The research was conducted during the online teaching period starting in mid of March to the end of May 2020, and 157 students of the Faculty of Economics of the University of South Bohemia in České Budějovice were involved in it; specifically, there were 67 men and 90 women. The students were dealing with 111 mathematics problems in total, forming 27 sets covering four mathematics topics. The sets of problems were taken from the Khan Academy1, including all step-by-step hints, and offered multiple-choice questions with one or more correct answers. The advantages and limitations of MCQ used in mathematics tests are widely discussed, for example, in Torres et al. [36], Sangwin [27] or Sangwin & Köcher [28]. We take advantage that formative tests with MCQ can be assessed automatically using computer systems, such as LMS, and that the consequent statistical analysis can be easily performed. However, we have to admit that our findings may be biased because of the limitations of MCQ, mainly by the fact that MCQ cannot measure some types of learning objectives, and students can guess correct answers.
Students could choose between a printed or an electronic version of the assignment in each of the sets. Those, who decided on the printed version, obtained three sets of materials: assignments of the mathematical problems with multiple-choice questions, corresponding step-by-step hints, and the correct answers. Those, who decided on the digital version, were given a link to a website with all the assignments, hints, and where their answers were automatically assessed. Most of the sets consisted of 4 problems; only one set contained seven problems. The electronic sets were transformed into an electronic workbook and afterward into a printed version of this workbook. The following images illustrate the electronic version of the workbook. The conducted research and subsequent analysis focused on error rate, the score of used hints, and the time necessary for solving particular mathematical problems.
Stoop, Kreutzer & Kircz [34] conducted similar research to authors of this study when they researched the difference in reading and learning from paper-based versus electronic media in a professional and educational setting. The paper-based set consisted of several paragraphs of a book, a dictionary, and a list of questions; the digital version was in the form of consecutive web pages, including test questions and a possibility of translation using a mouse. The group dealing with the digital version gained better results. The authors claim that better performance and quicker work with the text lie in better orientation in electronic texts. However, Mangen, Walgermo & Brønnick [18] claim that scrolling down in digital texts, which should help the readers understand them and orientate in texts, may thwart it, especially when texts are longer than a page. In our study, all problems were assigned within one page, but the students had to use scrolling when they used step-by-step hints.
In Figure 1, you can see a sample of one of the problems. How did the students proceed when trying to solve it? If students knew how to solve it and did not need a hint, they choose one of the offered answers and, in case of the correct answer, they could proceed to another problem. If they worked with the printed version, they had to check their answer with the workbook key. In the case of the electronic workbook, their answers were assessed automatically. If their answer was correct, they were rewarded with a winning notice 'Nice work!' Figure 2.
Step-by-step hint sample Source: Khan Academy If the students knew how to solve the problem, but their answer was not correct, they calculated the given problem again. Usually, the reason for the wrong answer was the students' inadvertence. In both the printed and the electronic version, the students made records of their errors. In the case of a wrong answer in the electronic version, there was an automatic notice 'Not quite yet ... Try again, Get help, or move on.' If the students did not know how to solve some of the problems, they could use hints. A sample of step-by-step hints is presented in Figure 2.
This particular problem offered four hints in total. The students working with the printed workbook could see the printed hints. In the case of the digital version, the students could ask for a hint by clicking on 'Stuck? Watch a video or use a hint'. In both versions, the students made records of the number of hints for each of the problems. Therefore, we could analyze the 'number of the used hints' for both versions of the workbook afterward.
As each of the problems has a different number of hints, the number of used hints was divided by the total number of the hints to get the indicator called 'score of the used hints', which we used in the further analysis.
All sets of mathematical problems served the students to revise and consolidate their knowledge and check their understanding and skills according to the consolidation and control functions of textbooks used by Zuyev [39] that they had learned from lectures and seminars.
The students knew that their work with the workbook was voluntary and would have no impact on their assessment and final marks. Those working with the printed versions of the workbook were asked to record their progress (the number of used hints, their time necessary to solve the problems) into standardized forms and upload the forms in the LMS Moodle.
All the obtained data were then processed using the random-mixed model (McLean, Sanders & Stroup [20]), a statistical model considering random and fixed effects that are useful when measurements are made repeatedly using the same data and the same date may be missing.
The random-mixed model was used to analyze the data obtained from 119 students who worked consistently with only one version of the workbook (50 students worked with the electronic and 69 students with the printed version).
The version of the workbook, denoted in our model as 'VERSION', is a fixed effect. The set of problems, denoted as 'SET', and the students, denoted as 'STUDENT', are random effects. The research question was if the version of the workbook (electronic or printed) had any impact on the error rate, the score of the used hints, and the time necessary for solving particular mathematical problems denoted in the model as 'ERROR', 'PROHINTS', and 'TIME'.

RESULTS AND DISCUSSION
The first of the observed aspects was the error rate indicating how many times the students chose the wrong answer within one particular mathematical problem. Table 1 presents 'Total error rate' indicating the mean of the total error rate made by all students in the given problem. Columns 'Error rate E' and 'Error rate P' indicate the means of the total error rates regarding the electronic and printed version. Columns 'Difference E' and 'Difference P' present differences between the total error rate and the error rates in the electronic and printed versions. If the value in these columns is positive, it means that the error rate in that particular version is higher than the total one and vice versa. Observing Table 1, it is evident that the error rate of the students using the electronic version is higher than in the printed version. In total, this is true for 26 problems out of 27. Table 1. Mean values of error rates and differences between versions E and P Source: Results processed by the authors The identified differences were also verified statistically. To verify that the version of the workbook influences the error rate, we used the Poisson distribution in the random-mixed model. The random effects were particular sets of problems ('SET') and particular students ('STUDENT'). The fixed effect is the used version of the workbook ('VERSION'). All this means that each of the 27 sets may have higher and lower error rates, and it also applies to the students. Using the random effect, we give to the shift of the error rate, within the sets of problems and within the students, a random influence, as some sets may be more difficult than the others and some of the students being weaker and some stronger. The random-mixed model enables us to prove the significance via the random shift within the sets of problems and the students. The created model 'model.glmer1' is described below. Table 2 presents the variance and the standard deviation concerning a particular student or a particular set of problems. Table 3 shows calculated p-values revealing if the differences in the error rates concerning the version of the workbook are statistically significant, even on a low significance level. The statistical investigation verified that the students working using the electronic version of the workbook had had a statistically higher error rate than the students working with the printed version.     Table 5 presents the variance and the standard deviation regarding the students or the set of problems.
As the model 'model.glmer2' does not give us a p-value, we created another model 'model.glmer21' (see below) and using F-test, we compared these two models. The calculated p-value, presented in Table 6, revealed that the difference between the number of the used hints regarding the electronic and the printed version is statistically significant. The statistical investigation verified that the students working with the electronic version of the workbook had used the available hints less than the students working with the printed version (Table 7). The third observed aspect was the time necessary for solving the mathematical problems measured in seconds. Table 6 presents the mean values of the total time necessary for solving the problems ('Total Time') and the mean values of each version's total time ('Time E' and 'Time P'). Similar to the previous aspect, columns 'Difference E' and 'Difference P' present the differences in times regarding the electronic and printed versions and the total time. The investigation identified the fact that for 23 sets of problems, the students working with the electronic version needed less time than their counterparts. The identified differences were also verified statistically. To verify that the version of the workbook influences the time necessary for solving the set of problems, we used the normal distribution. In the random-mixed model, we again compared a random effect 'SET' and 'STUDENT' with the fixed effect 'VERSION'. The models were constructed in the same way as the previous models 'model.glmer2' and 'model.glmer21', only instead of 'PROHINTS' we tested the time 'TIME'. Table 8 presents the corresponding variance, and the standard deviation and Table 9 presents the calculated p-value. The calculated p-value shows that the differences are not statistically significant. It is necessary to mention that the input parameters of particular pieces of research differ. In some studies, students could choose the reading format. In others, students were divided randomly into two groups, so some of them might have worked with the format they disliked. It is also essential to mention the length of the text they worked with. Most of the studies considered narrative texts or longer study texts. This makes a significant difference considering our research when our students worked with short texts, formulae, and also with graphs. Moreover, mathematical texts have, because of their strict logical structure, used symbols, and other specifics, more complicated structure than other texts of scientific character.
The conducted research revealed that the students working with the electronic version of the workbook have a statistically higher error rate than those working with the printed version. This conclusion has also been verified by the random-mixed model and is in accordance with the conclusions made by Lenhard, Schroeders & Lenhard [17]. They revealed that students function more quickly when their knowledge is tested digitally but at the expense of accuracy. This evokes several questions. Do electronic teaching and learning materials distract students' concentration and attention? Do students read assignments in electronic materials less carefully, which leads to a higher error rate? Unfortunately, we have not found any relevant research comparing electronic and printed mathematics textbooks concerning students' performance. However, there are pieces of research on e-textbooks focused primarily on comprehension and reading speed of individuals accessing text content through a stand-alone computer, even though they are not consistent.
Studies by Green et al. [12] focused on differences in comprehension between numerical data presented in illustrated diagrams and tables and written text on paper or a screen. Green et al. [12] suggest that the presentation of numerical information in graphs and tables shortens students' response time compared to data described in plain text. Sidi et al. [31] minimized the burden of reading in their study, and they tested short demanding logical problems. Their outcomes confirmed a significantly lower success rate of students taking tests on a computer.
Some studies concluded that reading traditional (printed) textbooks comes with better reading comprehension (Mayer, Heiser & Lonn [19]; Dillon [7]). On the other hand, Rockinson-Szapkiw et al. [26] claim that students using the electronic version of a core textbook were more active than their counterparts using the paper textbook. However, their final academic performance did not differ significantly. Some researchers, such as Daniel & Woody [5], Sun, Shien & Huang [35]; Young [37] or Grzeschik et al. [13] revealed in their studies that reading comprehension between paper-based and electronic documents differs only negligibly.
A long-term (2000 -2017) comparison of a paper-based and electronic reading was conducted by Delgado et al. [6], who found that the advantage of paper-based reading was gradually increasing over the years, mainly regarding informational texts or a combination of informational and narrative texts. The benefit of paper-based reading was also confirmed when the reading time was constrained. Ackerman & Goldsmith [2] claimed that self-paced paper-based and screen-based reading performance differed. The lower performance of the screen-based reading was caused by excessive self-assurance, as the common perception of presentations lowers mobilization of cognitive sources. Also, Singer & Alexander [32] revealed a higher paper-based reading performance regarding questions on particular pieces of information. Regarding questions on the main and key points, the reading medium was not essential.
Another aspect of the presented research was the number of used hints. The mean value of the used hints is statistically lower in the case of the electronic version, which could be caused by the fact that the students cannot reach 100% if they asked for any of the hints. To achieve 100%, the students had to solve the sets repeatedly. This may lead to the conclusion that the students think carefully about solving problems before using any of the hints. Unfortunately, as already mentioned above, we have not found any relevant research dealing with the use of step-by-step hints in solving mathematics.
The last observed aspect was the total time necessary for solving the presented mathematical problems. Although the mean values of time required for solving the sets were higher regarding the printed version in 23 sets out of all 27, the difference was not statistically significant. Findings of some studies dealing with the time used by pupils and students when reading printed and electronic texts are not unequivocal. Initial experimental studies suggested that reading long passages of information took longer when using an electronic format than reading a paper text (Dillon [7]; Mayer, Heiser & Lonn [19]). Dillon [7] found reading from a screen increased the length of time it took to read a text by 20-30%. Mayer, Heiser & Lonn [19] confirmed that readers had faster reading rates for paper text when compared to screen text during 25-minute reading sessions. Daniel & Woody [5] detected higher reading time of students reading newspaper texts at home.
On the other hand, Najjar [22] investigated the efficiency of the teaching process when multimedia is implemented. He lists many pieces of research that confirm that teaching with multimedia shortens the learning process significantly, and he also declares that interactivity has a strong influence on learning; students learn faster and gain better attitudes to learning. However, recent studies indicate that the differences in time between various media are not statistically significant (Eden & Eshet-Alkalai [10]; Young [37]).
Many researchers focus on screen-based reading and comprehension, but their outcomes are not consistent, not reading time and understanding. The authors mostly agree that digital reading is characterized as non-linear when readers skip from one place to another, search for keywords, and select the content. Due to this style, readers do not read in a concentrated and profound way (Durant & Horava [9]; Wolf [38]; Cull [4]).
We may pose an obvious question whether the less time the students spent working with the electronic version does not go hand in hand with the proven higher error rate. Lenhard, Schroeders & Lenhard [17] made the same conclusions as they revealed that students function more quickly when their knowledge is tested digitally but at the expense of accuracy. They also worked with the error rate, which they defined as the ratio of errors to the number of completed items in tests focusing on general comprehension of elementary school pupils. They conclude that the higher error rate is closely connected with the age of pupils when younger pupils make errors more often than their older counterparts. They also mention reasons for the higher error rate in the case of digital materials. The first reason may be playing computer games where speed is very often more important than accuracy. Another reason may be the manipulation with a computer mouse. While marking a correct answer with a pen requires a movement of the whole hand or even an arm, clicking on a mouse requires only a small movement of a finger. This may cause the marking of a wrong answer instead of the intended.

CONCLUSIONS
The presented research seems to be one of the first research pieces focusing on the difference in student's work with an electronic and printed version of a mathematics workbook regarding the error rate, the number of used hints, and the time needed to solve several sets of mathematical problems. We are fully aware that some aspects may influence our study's outcomes and that it is not possible to generalize the conclusions. As the students worked with the workbook at home without any supervision, we cannot guarantee that they recorded all the requested items. Our findings are based purely on the evidence provided by our students and by the computer system in which the students were working. Therefore, more research in this field will be necessary to conduct to do so. There is a trend to digitize course content to be more accessible to students, so it seems unavoidable to conduct more research on digital resources and differences between the printed and digital textbooks and workbooks in various elementary, secondary, and university subjects and courses.