• Question: How many times do you carry out an experiment before you have reliable results?

    Asked by Ellie to Abbie, Melanie, Paul, Stu, Tom on 10 Mar 2016.
    • Photo: Stuart Inglis

      Stuart Inglis answered on 10 Mar 2016:

      Hi Ellie,

      Haha admittedly I don’t often get involved in experimentation, we aim for “doing it right, first time” so if I took a couple of attempts I don’t think my boss would be too pleased!!

      However on the experimentation side of things it really depends on what you’re doing, however three is a good number to aim for as a rule of thumb. 😉

    • Photo: Tom Rooney

      Tom Rooney answered on 11 Mar 2016:

      There is no real rule for this, the minimum I am for is about 5 and then I see how much the data varied for an experimental run that should have been identical. Then if that data is very consistent I can move on to do the rest of the set of data, this involves changing one parameter a little and doing another 5 runs. Although if there is a lot of variance, or I think something looks a bit odd then I will do a lot of experimental runs. I had one very long and high resolution experiment that took a while to do each time, but I did it more than 20 times just to find a very small effect I thought might be there. After I compared all 20 experiments together, the really small effect I saw was actually really obvious now!

    • Photo: Paul Webb

      Paul Webb answered on 11 Mar 2016:

      I don’t really experiment

      But I will try something and see if it fixes it

      If not try something else

      Try agen

      And agen

      Until it’s fixed or you admit you need help lol

    • Photo: Abbie Hutty

      Abbie Hutty answered on 13 Mar 2016:

      It depends on what you’re doing and how likely the results are to vary, and also what the impact is if the thing doesn’t work like you expect the next time!

      For example if you’re measuring something like say the electrical resistance between one part of a structure and the other, we’d measure it three times, and if the numbers are all similar, then we are fine to assume that’s what the result would be if we went back and measured it again.

      If we’re measuring something like the strength of a material, where you might get some quite different results if your test set up varies a bit from test to test, and you are testing different samples each time, you’d want to do at least 5 – 10, and you’d want those results to all be pretty similar before you’d be confident that the values you’re getting can be expected on any other samples.

      If you’re still getting big differences between your results after you’ve done 5-10, it probably means there is something going on that you don’t understand, or you need to test more to have confidence that your test set-up is correct. So then you have to look really carefully at your test procedure and any small differences between your samples that you are testing before you try the test again, and try to minimize the differences in your test until all your results are similar and you can be confident that you could do it again and get the same results each time!

    • Photo: Melanie Zimmer

      Melanie Zimmer answered on 17 Mar 2016:

      It depends what you do – in my case, every time I write a new line of code, I test it with the tools I write the code in. This helps me to identify if the syntax (symbolic representation) of the programming language, but in order to test the semantics (meaning) I have to test it within the environment it is aimed for. This environment is normally not the actual one it is used in, but we try to simulate it as good as we can so that once we put the code into the system, it should run smoothly (which is never really the case, but that is the idea). I agree with Stu that 3 is a good rule of thumb as a couple of attempts would be too costly for our company and the end-user.

Comments