![\"gmat](\"https:\/\/www.manhattanprep.com\/gmat\/wp-content\/uploads\/sites\/18\/2020\/01\/mprep-blogimages-wave1-46-e1579900623899.png\")
<\/p>\n<\/p>\n
I was naturally very interested to hear that my company, Manhattan Prep, had earned the top spot for score improvement in the recent Poets & Quants <\/span>survey<\/span><\/a> of GMAT test prep companies. The excited part of me wants to dance around shouting, \u201cYay! Manhattan Prep is the best!\u201d\u00a0<\/span><\/p>\n But the rational part of me is saying\u2026hmm. Of course, I think we\u2019re the best too, but I don\u2019t think that average score improvement is a great metric to use (no disrespect to Poets & Quants, which I think is an excellent resource for aspiring business school students). I\u2019d rather that you talk to friends, look at verified reviews from a source like <\/span>Trustpilot<\/span><\/a>, and <\/span>attend any free sessions<\/span><\/a> available to judge for yourself. (I\u2019ve got a free session coming up soon\u2014come say hi!)<\/span><\/p>\n <\/p>\n The study concludes that all of our GMAT students who responded to the survey had an overall average improvement of 91.3 points, higher than any other company\u2019s average. We also had the highest average improvement for GMAT classes (104.1 points) and classes <\/span>plus<\/span><\/i> tutoring (118.8 points).<\/span><\/p>\n That sounds fantastic! Why am I questioning this? A few reasons, actually. (The teacher in me has to add: Bonus points if you brainstorm yourself before you keep reading.)<\/span><\/p>\n Let\u2019s start at the starting point. Is our class students\u2019 104.1 point jump from, for example, 400 to 504.1? Or is that 104.1 points from, for example, 640 to 744.1? The impressiveness of \u201c104.1″ depends pretty heavily on your starting point\u2014it\u2019s far harder to improve by 100 points at the higher end of the scale.<\/span><\/p>\n Imagine that, for some reason, a particular company tends to attract people who have higher average starting scores. For instance, my company offers an Advanced GMAT Course for which we require a 650 minimum starting score to enroll. Our students in this class can literally only improve by 150 points\u2014or fewer, if they started higher than 650\u2014so how would you factor that into this study? (Despite that, we still ended up with the highest average improvement in this study, but I could easily imagine a different scenario in which one company\u2019s <\/span>average<\/span><\/i> score improvement was lower than another\u2019s and yet <\/span>more<\/span><\/i> impressive\u2014if their students were ending with higher actual scores.)<\/span><\/p>\n When you get to b-school you\u2019re going to learn about <\/span>sampling<\/span><\/a> in statistics. A good data set, or sample, is <\/span>representative<\/span><\/i> of the population you\u2019re trying to study. When that data set doesn\u2019t properly represent the desired population for some reason, your data is skewed in some way and you risk drawing faulty conclusions.<\/span><\/p>\n I have two big questions around the representativeness of the data sample in this study. I\u2019ll start with my greater concern.<\/span><\/p>\n Someone who slacks off obviously isn\u2019t going to have the same results as someone who works more diligently. Maybe\u2014on average\u2014the people willing to spend money on a class are also more serious about studying in the first place. By the way, this is called self-selection bias; you\u2019ll need to <\/span>know this concept<\/span><\/a> for b-school, too.<\/span><\/p>\n Let\u2019s extend that idea. The article mentions puzzlement as to why tutoring results weren\u2019t the best of all\u2014you\u2019re paying a premium, so you should get better results, right?\u00b9<\/span><\/p>\n It\u2019s very common for people to do what they can on their own before spending money on tutoring (a significant percentage of our tutoring students do this). This is a smart use of your money\u2014only use the tutor for what you really need\u2014but this leaves less \u201clift\u201d available to attribute to the tutoring because you\u2019re only using it for the last, hardest 30-50 points.<\/span><\/p>\n If certain types of students tend to self-select into classes or tutoring\u2014or into choosing one particular company for some reason\u2014then that introduces bias into the data and into any comparison you make across companies. These factors could be controlled for in a randomized study, but that would be a far harder study to conduct.<\/span><\/p>\n My second biggest concern revolves around participation bias (another <\/span>key concept you\u2019ll need for b-school<\/span><\/a>\u00b2<\/span>). People are more likely to report their experience with something if they’re either really happy or really upset and that can skew the data. (That’s why it\u2019s so frustrating to read Yelp reviews\u2014you can\u00a0<\/span>always<\/span><\/i>\u00a0find 5 people who LOVE this restaurant and 5 others who absolutely hate it. And there might be 500 who felt pretty good about it but didn\u2019t bother to write a review.)<\/span><\/p>\n The smaller the number of data points, the more risk that participation bias could significantly skew the results. While 1,000 data points overall is a good number, some of the cuts of the data in this survey are well under 50 data points. I\u2019m really uncomfortable drawing any conclusions based on 15 or 30 data points.<\/span><\/p>\n All of the above, by the way, is why you’ve never seen any Manhattan Prep claim about score improvement in our 20 years in this business. There isn’t a great but cost-efficient way to get truly representative data for this calculation and, even then, your starting point makes a big difference in evaluating how good that score improvement is. A 40-point improvement is seriously impressive if your starting point is 720. (And, hey, then you and your 99th percentile score can <\/span>work for us<\/span><\/a>!)<\/span><\/p>\n Sure! The main data point applying to Manhattan Prep did come from more than 100 respondents, so that makes me feel a little better about our stat.<\/span><\/p>\n It\u2019s still the case that, as happy as I am to see any article in which Manhattan Prep is ranked #1 for something good, I just don\u2019t feel comfortable accepting the accolade based on this particular data point. As I said, I still think we\u2019re the best, but I\u2019d rather try to prove that via <\/span>other means<\/span><\/a>\u2014and, mostly, I\u2019d like you to <\/span>decide for yourself<\/span><\/a>\u00b3<\/span>\u00a0what you think best fits your needs and learning style. (Wait, did I just refuse to accept our Oscar? <\/span>)<\/span><\/p>\n \u00a0<\/p>\n You can attend the first session of any of our online or in-person GMAT courses absolutely free! We\u2019re not kidding. <\/i><\/b>Check out our upcoming courses here<\/i><\/b><\/a>.<\/i><\/b><\/p>\n \u00a0<\/span><\/i><\/p>\n Stacey Koprince<\/a>\u00a0is a Manhattan Prep instructor based in Montreal, Canada and Los Angeles, California.<\/strong>\u00a0Stacey has been teaching the GMAT, GRE, and LSAT \u00a0for more than 15 years and is one of the most well-known instructors in the industry. Stacey loves to teach and is absolutely fascinated by standardized tests.\u00a0Check out Stacey\u2019s upcoming GMAT courses here<\/a>.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":" I was naturally very interested to hear that my company, Manhattan Prep, had earned the top spot for score improvement in the recent Poets & Quants survey of GMAT test prep companies. The excited part of me wants to dance around shouting, \u201cYay! Manhattan Prep is the best!\u201d\u00a0 But the rational part of me is […]<\/p>\n","protected":false},"author":6,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[880],"tags":[233,329,57329],"yst_prominent_words":[57320,57319,54090,57327,53741,57324,53635,53781,57314,53778,57326,56585,55708,57315,57312,54850,55938,54116,53760,57328],"class_list":["post-18541","post","type-post","status-publish","format-standard","hentry","category-gmat-101","tag-gmat","tag-gmat-score","tag-gmat-score-improvement"],"_links":{"self":[{"href":"https:\/\/www.manhattanprep.com\/gmat\/wp-json\/wp\/v2\/posts\/18541","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.manhattanprep.com\/gmat\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.manhattanprep.com\/gmat\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.manhattanprep.com\/gmat\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/www.manhattanprep.com\/gmat\/wp-json\/wp\/v2\/comments?post=18541"}],"version-history":[{"count":6,"href":"https:\/\/www.manhattanprep.com\/gmat\/wp-json\/wp\/v2\/posts\/18541\/revisions"}],"predecessor-version":[{"id":18567,"href":"https:\/\/www.manhattanprep.com\/gmat\/wp-json\/wp\/v2\/posts\/18541\/revisions\/18567"}],"wp:attachment":[{"href":"https:\/\/www.manhattanprep.com\/gmat\/wp-json\/wp\/v2\/media?parent=18541"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.manhattanprep.com\/gmat\/wp-json\/wp\/v2\/categories?post=18541"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.manhattanprep.com\/gmat\/wp-json\/wp\/v2\/tags?post=18541"},{"taxonomy":"yst_prominent_words","embeddable":true,"href":"https:\/\/www.manhattanprep.com\/gmat\/wp-json\/wp\/v2\/yst_prominent_words?post=18541"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}What was the starting point?<\/b><\/h3>\n
Is this sample representative?\u00a0<\/b><\/h3>\n
(1) Are the sub-groups comparable?<\/b><\/h4>\n
(2) Who participated in the study?<\/b><\/h4>\n
Is there anything you <\/b>do<\/i><\/b> like about this study?<\/b><\/h3>\n
1<\/b> Yes, I\u2019m a geek and have footnotes in a blog post. \ud83d\ude42 My colleague Reed Arnold pointed out to me that the tutoring students in the study also reported a much lower number of tutoring hours compared to class hours for those in classes\u2014so you could potentially argue that the tutoring <\/span>was<\/span><\/i> more efficient in terms of hours spent per point earned.<\/span><\/h6>\n
2<\/b> Shout-out to my colleague Daniel Fogel for telling me the official name of this type of bias.<\/span><\/h6>\n
3<\/b> I\u2019m hosting a free webinar on February 6\u2014click that link for details. Come say hi and ask questions about the GMAT!<\/span><\/h6>\n
\n![\"stacey-koprince\"](\"https:\/\/cdn2.manhattanprep.com\/gmat\/wp-content\/uploads\/sites\/18\/2015\/06\/stacey-koprince-150x150.png\")
<\/a><\/p>\n