Experimental design? What?

Wow, it's been a little while since I've written for this blog! For this week's assignment, we are to talk about Experimental Design. We were asked to read two articles and watch a video and write a post pertaining to the content. I will go in to more detail than usual, as I am to lead discussion in class on Wednesday, and experimental design is the topic. Depending on how the discussion lead goes, I may post my slides and notes for those who are interested. 

The first paper from the Harvard Business Review "Finally, A Majority of Executives Embrace Experimentation" by H. James Wilson and Kevin Desouza talks about Greg Linden's idea of product suggestions on Amazon product pages, and what he went through to test the idea before he was allowed to implement the idea on the Amazon website. This article delves deeper in to the concept of experimentation and tells us the three different types of experimentation: prototyping, simulation, and test groups. Prototyping is more of a trial-and-error approach, where slightly different variations of the proposed product are created and tested to see which fits the environment the best. After testing, the prototype would be refined and tested again, and the cycle of refining and testing would be repeated until the prototype gains satisfaction from the producer. Simulation is the use of computer programs and scaled experiments to experiment with different designs; simulation has become huge in the 21st century as computers have become increasingly advanced. Test groups take two products/pages and show a certain group the original, and the other group the other with one variable to see which responds better. Test groups are often used by Amazon to experiment with new features on their web pages, and is how Linden implemented the experiment of his suggestion feature for Amazon. It is important to note the differences between each type of experimental design, and the uses for each. Simulation is best used for things like car safety features, because it would be more efficient to use a computer program to analyze all of the different reactions to a crash. Simulation is a quick and efficient way to experiment, whereas prototyping takes much longer, like Tony Stark's suits in the Iron Man movies and comics. Tony has so many suits because he will tweak something or add a new feature, and want to test it out, and in the Iron Man movies, Tony ends up staying up for days at a time working on his suits. To me, the easiest way to explain experimental design is to think of it as implementing scientific theory in a business environment. Whoever said "I'm not a science person! That's why a majored in business!" I hate to burst your bubble, but science and it's theories are literally everywhere. 

The second article we were asked to reflect on is also from the Harvard Business Review, and is titled "How to Design Smart Business Experiments" by Thomas H. Davenport. Davenport's article looks in to how experimental design should and should not be used in business environments, and when it should and should not be used. The article is quite lengthy, and if you are interested in learning more about experimental design, I would encourage you to take a look as I will not be covering everything it talks about in this write-up. The most important part of this article is this sentence: 

"Formal testing makes sense only if a logical hypothesis has been formulated about how a proposed intervention will affect a business."

This sentence means that there is not always a need to test, but when a test is needed one must go about it in the proper way. Experimental design is based on scientific theory, and follows the same steps, just with more of a focus on re-testing and refining to find the perfect variable. However, in business there is the challenge of risk and reward. While some may choose to take the risk of failure from implementing a change without testing, the safe route will always be to test the waters with the new variable. 

Experimental design uses the same processes as scientific theory, but is focused more on refining the hypothesis and variables. Image from "How to Design Smart Business Experiments" by Thomas H. Davenport of the Harvard Business Review. 

The video we watched was a "Ted-Ed" video called "Not all scientific studies are created equal" by David H. Schwartz, and helps to explain how experimental design is based on scientific theory. Schwartz talks about scientific studies and reliability of information. Schwartz goes through the different variables of random clinical trials and epidemiological studies. Random clinical trials are better known as voluntary experiments with medications and placebos. Randomly selected participants will receive either the medication or the placebo in either a single blind, or a double blind experiment; the variable being tested is the medication, and whether or not it effects the actual thing it is supposed to be targeting. The FDA often requires there are two random clinical trials for each new drug produced before it is marketable to the public. Epidemiological studies simply observe the subjects going about their regular every day behaviour instead of putting the participants in to variable groups. Epidemiological studies are not the preferred method of scientific experimentation, as there is a large margin for error; but are good for measuring the health effects of things like certain herbs or ingredients in food. Participants who already ingest the ingredient in question are known as the cohort, and there is also a control group of participants who do not ingest the ingredient. However, we can not rely on even the best of epidemiological studies to give us reliable information because every epidemiological study has inherent flaws. Sometimes, participants may not be randomly assigned to cohorts and control groups, but are grouped based on why they take a certain herbal supplement; this is known as selection bias. There is also the issue of confounding variables, like whether or not the participant has better access to healthcare or leads a healthy lifestyle. 

After reading these articles and listening to the Ted-Ed video, I have begun to wonder just how reliable all of the studies about what goes in to our food are, based on what types of testing and what variables the producers tested. On the business side of things, we were asked to think about what kinds of experiment we could implement in our client's AdWords campaign for the Google Online Marketing Challenge. What I would like to experiment with is the landing page of our client's website, and how the landing page might change the subscription conversion rate. I would also like to try changing small parts of the "product page" to optimize the online affiliate marketing that our client uses to create monetary conversions. The first thing to do would be to take one variable and create a hypothesis: "If we change part A of the web page, there will be more conversions". The next step would be to create a plan/method to implement the experiment, down to the details of how many people see the original page and how many see the new page, and possible new HTML coding. Then the experiment would be implemented, analyzed, and then we would decide based on the results whether or not to keep the landing page the same and test another concept, or to implement the new landing page.