Chapter 1 · Method and the microscopeLesson 1.5CS
Advanced scientific investigation: errors, risks and arguing for results
What would it take to convince someone who does not believe you that a well’s water has too much nitrate in it, or that hand-washing saves lives?
- Size
- from 1 mm to 1 nm
- Level
- cell and cell structures
- Time
- one lesson in the lab
Before you read: what do you think?
Researchers in Bucharest expected the wells in the two communes (rural districts) closest to the capital to have the least nitrate. The limit for drinking water is 50 mg/l. What did they find in 10 wells in Brănești, in Ilfov county?
See the answer
In five wells nitrate was between 272 and 493 mg/l, more than five times the limit, and in four it could not be detected. The starting hypothesis was disproved for Brănești; the authors point out that a non-ferrous metallurgy plant used to operate in the commune.
In 1846, at the Vienna General Hospital, women giving birth in the First Clinic died five times as often as those in the Second Clinic. The physician Ignaz Semmelweis looked for the difference between the two and imposed a simple rule: washing hands in a solution of chlorinated lime. Yearly mortality in the First Clinic fell from 11.4% in 1846 to 1.3% in 1848, but many colleagues did not believe him.
From a real problem to a hypothesis
An advanced investigation starts from a problem in a real context, in health, the environment or biotechnology, and narrows it down until it becomes a question that can be answered. Semmelweis started from a problem: puerperal fever, an infection after childbirth, was killing many women. His question: why do more women die in the First Clinic than in the Second?
The two clinics had one difference: the first was staffed by doctors and students, who also performed autopsies, and the second by midwives, who did not. When a colleague, Jakob Kolletschka, was accidentally pricked by a student’s knife during an autopsy and died with the same symptoms as the sick women, Semmelweis formulated his hypothesis: doctors’ hands carry disease-causing particles from the corpses to the patients. The independent variable became hand-washing with chlorinated lime, and the dependent variable, mortality.
Random errors and systematic errors
Random error makes repeated measurements vary, sometimes up, sometimes down: a hurried reading, a drop slightly too big or too small, a stopwatch started sometimes too early, sometimes too late. It is reduced by repeating measurements and calculating the mean. Natural differences between individuals, between two seeds for instance, also make results vary, although they are not measurement errors; that is why several individuals are used.
Systematic error shifts all the results in the same direction: a balance that always reads 2 g too high, an out-of-date reagent, a method that underestimates. Repeating does not reduce it, because every measurement is wrong in the same way. It is uncovered by calibrating the instrument, by samples of known value or by a second method, independent of the first.
Two terms come from this distinction. Precision shows how close repeated measurements are to one another. Accuracy shows how close they are to the true value. An uncalibrated instrument can be very precise and still inaccurate.
Risks and the strategy of the investigation
Before starting, the risks are identified: to safety (chemicals, glass, cultures of microorganisms, which at school are never grown from unknown samples), to ethics (people, animals) and to the result (sources of error, lost samples). For each one, you note how it is prevented.
Then the strategy is chosen. An experiment changes a variable on purpose. For a reliable comparison, a control group and random assignment are used. When that is not possible, for the wells of a village for example, an observational study is carried out, comparing groups that already exist. The strategy also sets how many samples are taken, where, when and how they are coded, so that the person measuring does not know which sample is in front of them.
Arguing and improving
You argue for a result in three steps: the claim (what you assert), the evidence (the data that support it) and the reasoning (why the evidence leads to the claim). A complete argument also discusses alternative explanations and says why they are less likely.
Semmelweis had strong evidence, but he did not convince people. He had no mechanism his colleagues would accept, and his main work did not appear until 1861. His explanation was accepted only decades after his death, when Louis Pasteur, Joseph Lister and Robert Koch demonstrated the role of microorganisms in disease.
Every investigation ends with its limitations and with proposals: more samples, other seasons, other places, a more accurate method, a variable that was left uncontrolled.
Worked example
Argue for Semmelweis’s result
The claim: hand-washing with chlorinated lime reduced mortality in the First Clinic.
The evidence: yearly mortality fell from 11.4% in 1846 to 1.3% in 1848. The difference is 11.4 − 1.3 = 10.1 percentage points; the ratio 11.4 ÷ 1.3 ≈ 8.8 shows that the risk of death fell almost ninefold.
The reasoning: beforehand, the only known difference between the clinics was the hands that came from autopsies; once the hands were washed, the first clinic’s mortality came down towards that of the midwives’ clinic.
Alternative explanations to discuss: did something else change in the hospital at the same time? Did fewer sick patients arrive? Semmelweis changed a variable on purpose, but the women were not assigned at random, and mortality is compared before and after the change, so these possibilities have to be ruled out with data. An improvement: a month-by-month comparison with the Second Clinic, where the rule did not change.
In Romania
Nitrate in the wells of four villages
Maria Pele, Gina Vasile and Mihaela Artimon, of the University of Agronomic Sciences and Veterinary Medicine of Bucharest, published their analyses of well water from four places in 2010: Matca (Galați county, vegetables in greenhouses and polytunnels), Săhăteni (Buzău county, vineyards), Brănești (Ilfov county, a former non-ferrous metallurgy plant) and Clinceni (Ilfov county, no intensive farming). They took samples from 7–10 wells in each village, at depths of 5–60 m, and kept the samples at 4 °C.
Nitrate was measured by two independent methods, by spectrophotometry and with a nitrate-sensitive electrode; the results were similar, and the authors used their mean. All seven wells in Matca and seven of the eight in Săhăteni were over the limit of 50 mg/l, with values up to 540.8 mg/l.
At Clinceni, the published mean of the eight wells was 49.78 mg/l, below the limit, but three wells exceeded it: 52.08, 50.03 and 66.58 mg/l. Excess nitrate is dangerous above all for babies, in whom it can cause methaemoglobinaemia; the study mentions 11 cases in Buzău county in 2007.
In short
- An advanced investigation starts from a real problem in health, the environment or biotechnology, turned into a question that can be answered.
- Random error varies in both directions and is reduced by repeats; systematic error shifts all the results in the same direction and is uncovered by calibration or by a second method.
- Precision shows how close repeated measurements are to one another; accuracy shows how close they are to the true value.
- The strategy can be an experiment, which changes a variable on purpose, or an observational study, which compares groups that already exist.
- An argument has a claim, evidence and reasoning, and discusses alternative explanations.
- The investigation ends with its limitations and with proposals for improvement.
Common mistakes
WrongIf I repeat the measurement many times, I get rid of every error.
RightRepeating reduces random error. A systematic error, such as an uncalibrated instrument, repeats itself in the same way in every measurement.
WrongThe mean for the village wells is below the limit, so the water from every well is fit to drink.
RightThe mean hides the individual values. At Clinceni the published mean was 49.78 mg/l, but three wells out of eight were over the limit.
WrongIf the data are clear, the conclusion convinces on its own.
RightThe conclusion has to be argued for: evidence, reasoning, alternative explanations ruled out. Semmelweis had good data and did not convince people without an accepted mechanism.
Check yourself
Seven questions. For the first six you see the explanation at once, with the paragraph of the lesson that supports it; for the written ones you compare with a good answer.
A balance reads 2 g more than the true mass at every weighing. What kind of error is this, and how do you uncover it?
Answer
b) Systematic error; you uncover it by weighing an object of known mass. All the results are shifted in the same direction, so the error is systematic. Repeating gives the same extra every time; a standard weight brings it to light.
Why did the researchers measure the nitrate in the wells by two different methods?
Answer
d) To check that a single method was not shifting the results systematically. Two independent methods that give similar results make a systematic error in either of them unlikely.
What strategy did the study of the wells in the four villages use?
Answer
a) An observational study, comparing groups that already exist. The researchers changed nothing in the wells; they compared villages that already existed, with different activities.
Four pupils measure the same 50 mm leaf with a ruler whose end is broken off and get 53, 53, 53 and 53 mm. What are their results like?
Answer
c) Precise but inaccurate. The values are identical to one another (precise), but all of them are far from the true value (inaccurate), because of a systematic error.
By how many percentage points did yearly mortality in the First Clinic fall between 1846 (11.4%) and 1848 (1.3%)?
percentage pointsAnswer
10.1 percentage points. The difference between two percentages is expressed in percentage points: 11.4 − 1.3 = 10.1.
Sort the sources of error: random or systematic?
Answer
Random error: A hurried reading of the scale, sometimes too high, sometimes too low; A stopwatch started sometimes too early, sometimes too late.
Systematic error: A thermometer that always reads 1 °C too low; An out-of-date reagent that gives values too low for every sample; A pipette that delivers slightly more than it should every time it is used.
Systematic errors shift all the results in the same direction; random errors vary in both directions.
A village council wants to find out whether the wells next to fertilised fields have more nitrate than those in the village centre. Propose the strategy of the investigation, two sources of error and one improvement.
A good answer
It is an observational study: several wells are chosen in each area, say eight each, sampled on the same day, in clean bottles, kept cold and coded so that the laboratory does not know the area. Sources of error: an uncalibrated instrument (systematic error), checked with a solution of known concentration, and repeated readings of the same sample that differ slightly from one another (random error), reduced by repeating the measurement and calculating the mean; the natural differences between wells and between days call, in turn, for more samples. Improvement: sampling again in another season, after the fields have been fertilised, and recording the depth of each well.
A complete answer chooses the right strategy (observational), asks for several samples and coding, names one systematic and one random error with the way each is controlled, and proposes a concrete improvement.
Practical work for this lesson
- Germination: from hypothesis to a testable protocol – Guided experiment or analysis of invented data, one hour of planning and analysis, plus five days of observation if you do the experiment (CS)
Terms
- random error
- An error that makes repeated measurements vary in both directions; it is reduced by repeats.
- systematic error
- An error that shifts all the results in the same direction; it is uncovered by calibration or by a second method.
- precision
- How close repeated measurements of the same quantity are to one another.
- accuracy
- How close a measurement is to the true value.
- observational study
- An investigation that compares groups that already exist, without the researcher changing a variable on purpose.
- argumentation
- Supporting a claim with evidence and with the reasoning that links the evidence to the claim.
Sources and link to the programme
- Medical Journal of Australia (2018) – Ignaz Philipp Semmelweis (1818–1865): herald of hygienic medicine – 1846: mortality in the First Clinic five times that in the Second; the doctors and students performed autopsies, the midwives did not; Kolletschka pricked by a student’s knife at an autopsy; hand-washing in a solution of chlorinated lime; 11.4% in 1846 and 1.3% in 1848; the colleagues who did not believe him; the main work of 1861; acceptance of germ theory decades after his death, through Pasteur, Lister and Koch
- M. Pele, G. Vasile, M. Artimon (2010) – Studies regarding nitrogen pollutants in well waters from Romania, Scientific Papers USAMV București, Seria A, vol. LIII – the four places and their activities; 7–10 wells per village, depths of 5–60 m, storage at 4 °C; the spectrophotometric method and the ion-selective electrode, with similar results; the values in the tables (Clinceni mean 49.78 mg/l, Brănești up to 493, Săhăteni up to 540.8, Matca 149–452); the WHO limit of 50 mg/l; 11 cases of methaemoglobinaemia in Buzău in 2007
- NIST – Measurement Uncertainty – uncertainty as part of the result of a measurement
- UC Berkeley – Understanding Science: Fair tests, a do-it-yourself guide – fair comparison, controlling variables, repeats
- Stang, Standl, Poole (2022) – A twenty-first century perspective on concepts of modern epidemiology in Ignaz Philipp Semmelweis’ work on puerperal sepsis, European Journal of Epidemiology – chlorine washing was introduced on purpose in the First Clinic, without random assignment; the before-and-after comparison and the comparison with the Second Clinic; possible confounding factors
From the programme: “Stages of the advanced scientific investigation/method based on observations and experiments: defining a problem in a real health/environment/biotechnology context, formulating the hypothesis, setting the variables, identifying risks/sources of error, deciding on the experimental strategy, analysing the data obtained, drawing up and presenting the conclusions, arguing for the results, proposing new ideas to improve the investigation” (specialism, Natural Sciences). Specific competences addressed: 1.2, 2.2, 4.1, 4.2.