How Accurate is Think Secret?

by James R. Stoup Oct 06, 2006

Everybody loves a good rumor, at least, thats what the editors over at Think Secret hope. As the web’s premiere Apple-related rumor monger they have become the place to go for all of the best Mac gossip. They may not be always correct, but they are always close, right? Well, as it turns out, not really.

First, let me say that I am a long time reader of TS and that I always enjoy their gossip. As a user of Mac products and a writer for a Mac related blog, I always like getting the best information I can, as soon as I can. However, recently I have begun to have doubts as to just how good they are at predicting future trends and products.

So, in an effort clear the air and find out just how good a job they do, I had a look at their archives and compared them to Apple’s press releases dating back to the beginning of 2005. Here is what I found.

Initial problems, how do you define “accurate”?
Before I can make a judgement on how accurate Think Secret’s reporting is I have to define just what I mean by “accurate”. Unfortunately this is a very tough problem as demonstrated by this example:

Lets say that I predict that Apple will release a new iPod in the next year. And sure enough 8 months from now they update one of their iPods to give it more storage. Was my prediction accurate?

Well, it was true. After all, I said they would release a new iPod and they released a new iPod, albeit 8 months later. So does “true” equal “accurate”? Not always. But wait, I can already guess what you are going to say. You are going to complain that I said “new iPod” when in fact I should have said “upgraded iPod”. Ok, consider the prediction changed. Now am I accurate?

Well, yes, technically. But it wasn’t a very useful prediction was it? Since the term “upgraded iPod” could mean anything. If it got a new paint job, more storage, a brighter screen or better headphones I could claim it was “upgraded”. I made an obvious guess, based on readily available data about a strong product line that ultimately turned out to be true. Was my “prediction” true? Technically, yes. Am I a market visionary with inside information? Not really.

So, right off the bat we need to change the term “accurate” to “accurate and useful”. This means that if Think Secret predicts that Apple will marginally increase the size of an iPod, bump the speed of a processor or do a point upgrade to a piece of existing software, then I can’t give them credit for having inside knowledge. All of those predictions are mundane enough that they could be made by anyone looking at Apple’s historic road map. So, no credit for the easy ones!

However, this still leaves us with the problem of the “machine gun” approach to forcasting the future. A technique used all to often by sites like Think Secret. In this method you spew the same predictions as many times as you can and hope that you hit something eventually. How do you define “accurate” in such a situation? Here is an example of this problem:

I come to work one Monday and tell my co-worker that I predict it will rain that day. The day passes with no rain. So I come in Tuesday and once again predict that it will rain that day. And once again I am wrong. I repeat this every day of the week. On Friday, it rains. I then look at my co-worker and proudly proclaim “See! I told you it would rain today.”

Once again, my prediction was technically correct, but practically useless since I had made the same prediction over and over throughout the week. So, in the same vein, I can’t give Think Secret credit for predicting something if they make the same “prediction” with “insider knowledge” every month for half a year until the product is eventually released.

So where does this leave us? Well, we have eliminated from consideration the obvious and the redundant, so now, all I am counting are the important predictions. I have broken the remaining portion of this article into two different parts.

The first section lists predictions from Think Secret’s point of view. Here is how to read this information. On each date listed below, Apple released some product or service. I reviewed Think Secret’s predictions for the week leading up to that date. Then, I compared what they predicted would be released with what actually came out. If they predicted things close enough to be in the ball park then I marked that guess as “right”. If they predicted something that didn’t show up then that guess was marked “wrong”. And anything Apple released that they didn’t mention at all was marked as “missed”.

Think Secret’s Accuracy for Major Product Launches from January 2005 - September 2006

January 11, 2005
Right: iPod shuffle, Mac Mini
Wrong: FireWire Audio Breakout Box

June 6, 2005
Missed: Intel switch. Repeating what the WSJ said doesn’t count!

August 2, 2005
Missed: Mighty Mouse

September 7, 2005
Right: New iPods, ROKR phone
Wrong: ALL DETAILS - Nano (not the Mini), Colors wrong, size wrong, size of ROKR phone, Shuffle not upgraded

October 12, 2005
Right: iPod
Wrong: Wrong size for iPod
Missed: Video iPod, iMac/FrontRow/iSight, buying videos from the iTMS

January 10, 2006
Right: Nothing of substance
Wrong: MacBook Pro came out, NOT the MacBook. iMac when Intel, Shuffle did not get an upgrade

February 28, 2006
Right: HiFi speakers
Missed: Intel based Mac Mini

April 5, 2006
Missed: Boot Camp

April 24, 2006
Right: 17” MacBook Pro

May 16, 2006
Right: MacBook

May 23, 2006
Missed: Apple & Nike

August 7, 2006
Right: Mac Pro
Wrong: Mac Pro in same case (not different case as they reported), no new Nano
Missed: All of Leopards features

September 12, 2006
Right: Upgraded iPod and Nano, new colors, movies
Wrong: iMac mounting kit?
Missed: iTV, games

11 correct predictions
8 incorrect predictions
11 missed announcements

What do these numbers mean? Well, it means that for all major announcements from Apple, Think Secret only gets about half of them right (11/22). For every Mac Book Pro they correctly predict, they miss a Video iPod. For every HiFi speaker they get right they miss a switch-to-Intel story. Taken another way a little less than half (8/19) of their predictions are completely wrong.

Putting it all together it looks like this: If you take all of the major predictions from Think Secret, then about half of them will be crap. Of the half that comes true (whatever number that may be), you can assume that there will be an equal number of surprises from Apple.

The second section of this piece looks at things from Apple’s point of view. Here, only the products Apple actually released are considered. (as opposed to the previous section which included TS’s incorrect guesses) This chart shows how close TS got to the actual product specs. A score of 0 means they failed to predict the product at all. The lower the score the less detail was correct, the higher the score the closer they got to the truth. For instance, TS completely nailed the iPod Shuffle (earning a 5) but only got partial details correct (they claimed a complete redesign when in fact the case stayed the same. Though they did get the processor correct) on the Mac Pro (earning a 3).

Complete list, Things Apple Released:
[0 = missed, 1(wrong) . . . 5(right)]
iPod shuffle - 5
Mac Mini - 5
Intel switch - 0
Mighty Mouse - 0
iPod Nano - 2
ROKR - 3
Video iPod - 2
iMac/FrontRow/iSight - 0
TV shows - 0
MacBook - 5
Intel iMac - 1
HiFi Speakers - 0
Intel Mac Mini - 0
Boot Camp - 0
17” MacBook Pro - 5
MacBook - 5
Apple & Nike - 0
MacPro - 3
New iPod & Nano - 5
Movies - 4
Games - 0

Looking at these results we see that, for the most part, Think Secret gets things either completely right or completely wrong. So, when they have a real source they apparently get everything from this source. Product name, specs, description, pricing, availability, I mean everything! However they do miss many products entirely. And I believe that this can be mainly credited to Apple’s lockdown on internal leaks. Because in reading over Think Secret’s archived postings it is clear that they tend to get leaks only in certain departments within Apple. Mainly their computer hardware section and their iPod section. But with software, media and periphials they don’t seem to do so well. So, when Think Secret has genuine insider information they do pretty good. However, when they have woefully incomplete information, all too often they put it out as the real thing and just fill in the missing pieces with their best guesses. The problem with this less-than-winning strategy is that they don’t guess very well. They misread the Nano announcement pretty bad, the ROKR was only partially right and they weren’t even sure the Video iPod was going to exist until the day before the launch (all that week they predicted it wouldn’t happen for another 6 months).

So, to recap, when they actually have insider information they tend to be close to 100% correct. However, when they only have a few clues then their guesses aren’t much better than any other pundit’s predictions. The annoying part about all of this is that they don’t make much of a distinction between the “locks” and the “guesses”. Of course, its all very much obvious in hind sight, but how does that help us now?

Some of you might read this article and conclude that I have some agenda against Think Secret. That I massaged the data to paint them in a negative light. Or maybe you think I didn’t interpret my own data correctly. Well, you are of course welcome to think anything you like, however the truth is much less sensational. Many people base their purchasing plans on the information that sites like Think Secret provide. And this is a good thing, in moderation. After all, who doesn’t mind waiting a bit for a better piece of technology? Or a better deal or current products? I am not out to get Think Secret, AppleInsider or any of the rest of those sites. I am simply trying to inform my readers that these sites aren’t nearly as accurate as they would have you think. I am trying to remind everyone that you need to judge for yourselves what to believe. Remember, if it seems to good to be true then its probably been confirmed by a super-secret “inside source”.


  • I love this article. I contemplated writing a similar piece, but was just too lazy to do all the leg work. Great job.

    One suggestion I would give (again, I am too lazy to do this myself), but out of all of Think Secret’s “accurate” predictions, go back and compare then to AppleInsider’s predictions.

    Now, check the Dates.

    AppleInsider pretty much predicts everything at least 1 or more days BEFORE ThinkSecret, and usually gets it right. Think Secret seems to take AI’s predictions, add of change a detail (usually wrong) and then make their bold prediction.

    I do not doubt Think Secret at one point HAD an inside source close to Apple, but I feel they have lost that source in the last 16 months or so, and are pretty much guessing at this point.

    We occasionally post rumors on our site as well, and almost all have turned out to be false (I’ll save you the rouble of doing any research!). We usually write that we suspect we are being dicked around with, but we write that we are publishing them anyway because hey, you never know ( I think the only rumor that ever turned out to be legit was a tip on the Nike Apple thing about 7 days before they announced it). But Think Secret presents info as if they have a real source. I think that is the problem.

    Almost all Mac-related sites get random anonymous e-mails making predictions. Because of Think Secret’s track record a few years back, they still have “street cred” in the Mac community, but honestly, if you are into rumor mongering, Apple Insider has been really accurate lately (at least that is the impression I get. I would love to see a piece on them next!)

    Thanks again for doing some actual research and leg work.

    -The Doc

    Dr. Macenstein had this to say on Oct 06, 2006 Posts: 2
  • most of these “inside sources” are actually analysts from investment firms.  i think what appleinsider and thinksecret do is extrapolate from analysts’ guidance and make their own predictions.

    check out this site to see how bad analysts are at predicting apple rumors:

    remember, rumor sites are just that. they’re in the business of propogating rumors, true or not.

    g5u1 had this to say on Oct 06, 2006 Posts: 9
  • Hey James.

    Just checking in.  Nice article. You’re doing good work here.

    Update your freakin’ bio!

    Your ever-lovin’ father-in-law.  Skip

    Skip Hall had this to say on Oct 08, 2006 Posts: 1
  • Excellent.  I’ve always wondered how much I should “trust” the rumor sites, it’s the sort of question that just sort of skitters through your thoughts as you’re deciding to hold off on a new iPod because the “next big thing from Apple” is on the way.  I’d be interested to see how AppleInsider and Macrumors stack up to ThinkSecret…follow up article?  Series?  I’d read.

    elfurbe had this to say on Oct 09, 2006 Posts: 1
  • Quick thought, when was the iMac 24” mounting kit released?

    sendai had this to say on Oct 11, 2006 Posts: 1
  • Page 1 of 1 pages
You need log in, or register, in order to comment