AVS Forum banner
1 - 20 of 129 Posts

·
Banned
Joined
·
29,681 Posts
Discussion Starter · #1 ·
HomeArticlesIT/DatacenterEditorials

Dispelling Backblaze's HDD Reliability Myth - The Real Story Covered



Read more at http://www.tweaktown.com/articles/6028/dispelling-backblaze-s-hdd-reliability-myth-the-real-story-covered/index.html#UX5OD0IAgPphmv71.99
Quote:
Introduction


Companies have a funny tendency to do just about anything for publicity. Being a storage writer requires staying up to date on the news, and in the normal course of my week, I watch companies clamor for attention daily. Attempts range from the mundane to the hilarious, and sometimes even questionable. Some will do anything to separate themselves from the thunderous cacophony of the thousand other companies screaming for attention.



Personally, I'm happy I'm not the one tasked with promoting companies or their products. I can see that there must be somewhat of an addictive property to nailing it though, as we observed from the splash of Backblaze headlines in the last few weeks.


Backblaze has had their name emblazoned on the front page of every tech website this week with the results of their HDD reliability blog post. Backblaze's blog is a running anthology of their company, and their efforts to provide users with unlimited online backup for a scant $5 a month. The very fact that you read that sentence makes this all worthwhile for Backblaze - they just got their message out.


This latest post is a result of their initial blog entries covering how long HDDs last, and the follow-up that enterprise HDD's are less reliable than consumer drives. Needless to say, there are holes in the methodology big enough to drive a truck through. However, the headlines led to questions from the public about more detailed drive failure rates, and Backblaze complied with the latest post, "Which Hard Drive Should I Buy?"


While Backblaze is somewhat clear about the results, they don't explain the test environment, and they don't do a good job of explaining why much of their data is worthless to the typical consumer. Further digging unearths questionable practices, at least if the goal is to gather data on drive reliability. Backblaze follows the open source mentality of sharing data on their enclosures, and even share schematics so readers can build their own Backblaze servers. This provides us some insight into their activities.


Reading their blog posts during the HDD crisis in 2011 is enthralling, they really do go to any length to continue operation in a cost-efficient manner. Unfortunately, that doesn't fall in line with determining the winners or losers of HDD reliability. To add insult to injury, most tech websites have picked this story up and proclaimed the results as a definitive guide to drive reliability.


The Backblaze results have been tallied and placed in nice charts, and winners proclaimed. Hitachi and Western Digital appear to be the winners, and Seagate comes in a distant third. However, in this case, even the winners are losers. Let's fire up the truck and drive through the holes in this test, and explain exactly why the results shouldn't affect anyone's purchasing decision.


Backblaze unabashedly sources the cheapest consumer drives they can find for storing their customers' data. Their noble goal is to provide storage as cheaply as possible, and then pass those savings along to their customers. They do not use drives indiscriminately, each model goes through a short test phase to assure the drive suits their purposes. Pairing these cheap drives with various RAID and replication schemes provides enough integrity to safely store data.



During the Thailand flooding in October 2011, Backblaze had to go to extreme lengths to procure enough drives to sustain operations. With the prices of HDD's rising exponentially, and supply in short stock, they literally took to the streets as a drastic form of damage control. They named the practice 'drive farming'.


First, they found that buying external drives was very cost effective. The whims of the market dictate that external drives are often cheaper than internal desktop drives. These drives come in small enclosures and are usually connected via USB or eSATA. Backblaze buys these drives, and strips the drives from the enclosures in a process they lovingly refer to as "shucking". Much like shucking a head of corn, they rip the enclosure apart and out pops a useable HDD.


During the Thailand flooding in October 2011, Backblaze had to go to extreme lengths to procure enough drives to sustain operations. With the prices of HDD's rising exponentially, and supply in short stock, they literally took to the streets as a drastic form of damage control. They named the practice 'drive farming'.


First, they found that buying external drives was very cost effective. The whims of the market dictate that external drives are often cheaper than internal desktop drives. These drives come in small enclosures and are usually connected via USB or eSATA. Backblaze buys these drives, and strips the drives from the enclosures in a process they lovingly refer to as "shucking". Much like shucking a head of corn, they rip the enclosure apart and out pops a useable HDD.




At first they pillaged area Fry's and Costco's until some employees were eventually banned from purchasing more drives. They fanned out further and enlisted the help of friends and family to continue purchasing drives, even detailing complex drive purchasing schedules to skirt drive purchasing limitations. When this approach began to fizzle in the local area, they even pondered going cross-country in Ryder truck. This eventually led to crowdsourcing. Backblaze offered their readers a $5 bounty per drive for drives purchased and sent to them during a Costco sale.


This practice exhibits amazing ingenuity and a laudable effort to deliver the best bang for the buck to their customers. Unfortunately, it doesn't mesh well with creating a stable sample pool to determine drive reliability. Backblaze also acknowledges using drives that were known RMA's, and refurbished, in the sample pool.





The majority of Backblaze's failures occur in the first few weeks of service, which is understandable considering their drive purchasing methods. The typical 'bathtub curve' of failures is expected with many storage devices, with the highest chance of failure in the beginning and ending stages of the product's life. However, it is feasible to conclude that their drive sourcing methods tainted their results.


Interestingly enough, Backblaze won $5 million in venture funding. Soon after, Costco was offering a sale on external HDD's with a limit of five per person. Even with the massive funding, they chose to take advantage of the sale to get the best bang for their buck. As they stated, old habits die hard.


Unfortunately, these drives are included in their failure ratings.


Backblaze also extends their ingenuity to the server rack. They have designed purpose-built Storage Pod enclosures, and share the schematic freely online. This commendable commitment to information sharing also sheds light on their 'failure rate' data.


The Storage Pod is currently on revision 3.0, with two prior incarnations requiring upgrades to deal with a number of design problems, most notably vibration.




Vibration is every hard drive's enemy, and creates an exponential amount of wear on components. Vibration even has performance implications. A typical desktop HDD experiences a relatively vibration-free existence in a stable environment, and is designed accordingly. One of the major differences in enterprise HDD design is vibration resistance technology. This allows the drive to function well and stand up to the wear and tear of the server chassis and rack.


More HDDs installed in an enclosure raises the amount of vibration. Backblaze packs 45 HDDs per enclosure for maximum storage density. While the drives are initially exposed to vibration from their neighbors inside the server, once placed into the rack, they are exposed to even more vibration from other servers. This creates the 'perfect storm' of vibration, and the use of consumer drives results in horrendous failure rates, as evidenced by the data from Backblaze.





It is no wonder that Backblaze has continued to refine their chassis to provide more resistance to vibration: the early models merely had nylon spacers to dampen vibration. Taking a closer look at their data, we can see that the drives in use the longest suffer the highest failure rates. One likely reason is simple: these older drives are in revision 1.0 of their storage enclosures, which suffer from significant vibration issues that merited a redesign.


Unfortunately for Seagate, these drives are predominantly from their product lines. This paints them in a very unforgiving light due to obvious chassis issues, with a misleading annual failure rate of 25.4% that would surely put Seagate out of business, if it were realistic.


Backblaze has left a significant amount of information out of the disclosed failure rate data. Segmenting these drives to different chassis revisions would be the responsible approach to information dissemination. We can rest assured that the older drives aren't in the best chassis available, revision 3.0 wasn't released until February 2013.


Another issue that affects the lifespan of an HDD is drive temperature. Variations in temperature and humidity have an impact upon drive life. Reading through the Backblaze blog, one comment that drew my eye was on server rack temperature;

... we've observed in the past three years that: 1) hard drives in pods in the top of racks run three degrees warmer on average than pods in the lower shelves; 2) drives in the center of the pod run five degrees warmer than those on the perimeter; 3) pods do not need all six fans-the drives maintain the recommended operating temperature with as few as two fans; and 4) heat doesn't correlate with drive failure (at least in the ranges seen in storage pods).







Backblaze claims that drive temperature doesn't affect drive life. That is counter to the observations of many others, including drive manufacturers. There is a reason for specified temperature ranges for HDD's. Though they are likely within these ranges, the Backblaze drives cannot be directly compared to each other with varying temperature ranges, let alone other drives. Once again, the lack of pertinent information makes any real conclusions impossible, and the uneven nature of the test environment spoils the data.



Workloads


Each drive is designed meticulously to provide a tightly-defined service level in its intended environment. These guidelines determine not only the design, but also the type of components used. The most cost-effective drives are designed to deliver exactly the correct performance and longevity in their intended environment, and nothing more. Utilizing robust components above the workload requirements of the drive is wasteful, and adds cost unnecessarily. This design efficiency also means the drives are more likely to fail in untoward conditions.



Backblaze procures the cheapest possible HDD on the market at all times, regardless of its workload rating, and then subjects them to a harsh environment that is virtually guaranteed to destroy the drive. This leads to higher failure rates than observed in the wild. This reflects just how precisely these drives are engineered to fulfil their stated purpose, and nothing more.


Another concern is the direct comparisons between drives, even though they endure varying workloads. Comparing drives of the same make and model is impossible if the same workload isn't applied. Expanding that out to compare different models and manufacturers is even more ridiculous. There is no way to know how many times the drives have spun up and down, and how many times the drives were subjected to varying types of data requests.


Random data requires more movement, and thus creates more wear and tear on delicate HDD heads. Spinning up and down, and also entering and recovering from various sleep states, also wears the drives differently over the course of time. With no real rhyme or reason to the workload distribution, let alone the environment, direct comparisons are impossible.


Only one thing is certain; the drives were subjected to workloads well beyond their design limits.


Final Thoughts


The data from Backblaze should not influence a purchasing decision by any consumer, regardless of what type of drive they are purchasing. The innumerable variables, and lack of documentation, ensures the results are unreliable. Even for the winners, the results aren't good; the failure rates are exponentially higher than those observed in the real-world. One should question whether these companies could survive financially with the massive warranty return rates in real-world scenarios.


We covered some of the most obvious holes in the methodology behind the Backblaze comparisons, but there are many more, such as sample size. With varying numbers of drives for each model, it is possible that some bad batches may have made their way into the sample pool, thus further skewing the numbers.



There is no clearer example of this than in their blog post titled "Enterprise Drives: Facts or Fiction?". In this blog post, Backblaze compared 368 enterprise HDDs, presumably purchased as a batch, to 14,719 consumer drives. Along with the fact that a bad batch may skew the numbers, Backblaze admits they subjected the drives to various chassis, temperatures and workloads. This creates data that is essentially worthless for comparative purposes, but when paired with a catchy title, it serves the purpose of attracting attention.


The enthusiast in me loves the Backblaze story. They are determined to deliver great value to their customers, and will go to any length to do so. Reading the blog posts about the extreme measures they took was engrossing, and I'm sure they enjoyed rising to the challenge. Their Storage Pod is a compelling design that has been field-tested extensively, and refined to provide a compelling price point per GB of storage.


It is the release of the data, in handy charts and graphs that encourage misrepresentation, which brings out the data-storage stickler in me. HDD manufacturers spend billions of dollars in R&D, and their labs are designed to characterize and measure the reliability and endurance of their storage solutions.


The Backblaze environment is the exact opposite. I do not believe I could dream up worse conditions to study and compare drive reliability. It's hard to believe they plotted this out and convened a meeting to outline a process to buy the cheapest drives imaginable, from all manner of ridiculous sources, install them into varying (and sometimes flawed) chassis, then stack them up and subject them to entirely different workloads and environmental conditions... all with the purpose of determining drive reliability.


Of course that wasn't the intention, but that is how some will interpret the data. In my opinion, the intoxicating allure of media coverage overwhelmed common sense, and Backblaze released these numbers with a catchy title that would attract attention. The tech media is to blame as well, with many posting the information with little or no research. Unfortunately, the Backblaze blog post will be copy/pasted innumerable times for years to come as an authoritative source of data, when it is the furthest thing from a comprehensive study imaginable.


Read more at http://www.tweaktown.com/articles/6028/dispelling-backblaze-s-hdd-reliability-myth-the-real-story-covered/index5.html#iLFiOVkFF8G0dFIe.99


Thoughts ???
 

·
Registered
Joined
·
13,150 Posts
Good article.


Unfortunately there is almost no other similar data (good or bad) on hard drive reliability which is why the original article was so popular.


Since there are millions of hard drives out there it's really a shame. The consumer should know more about this subject.
 

·
Banned
Joined
·
29,681 Posts
Discussion Starter · #3 ·

Quote:
Originally Posted by assassin  /t/1515424/dispelling-backblazes-hdd-reliability-myth-the-real-story-covered/0_100#post_24297523


Good article.


Unfortunately there is almost no other similar data (good or bad) on hard drive reliability which is why the original article was so popular.


Since there are millions of hard drives out there it's really a shame. The consumer should know more about this subject.

I agree. I wish there was more information. The only thing that seems certain from that article is the Hitachi appears to be very good. They had average age of drives 1.5-2 years and high sample size. If this article above is right and the conditions were less ideal, then the fact Hitachi did well in them regardless suggests they are good. I can't imagine a scenario where if you improve conditions the drive reliability would fall.


I looked into those Hitachi drives a bit after reading that article and I found they not only have double the MTBF rating of WD or Seagate (2 millions versus 1 million) but they also have double the warranty (5 year warranty vs only 2 or 3 years)


Additionally I discovered they are available in 7200rpm and 4TB sizes (something that is hard to get as a combo) so you can get big capacity and good performance too.


I also found they sell under $189 for the 4TB. That seems like a great choice for storage. The 5 year warranty basically ensures you will have a working drive for 5 years. That is value right there.


So I did appreciate reading that first Backblaze article.


I also appreciated reading this article above as a counter to it, the combination of the two keeps me in check. I found a Hitachi drive I love, but I don't stop buying the Seagate models I love and had good luck with as long as the price remains great.
 

·
Registered
Joined
·
744 Posts

Quote:
Originally Posted by assassin  /t/1515424/dispelling-backblazes-hdd-reliability-myth-the-real-story-covered/0_100#post_24297523


Good article.

Not really. I skimmed quickly through the article and kept thinking how poor the logic was and how sorely lacking it was with any supporting evidence to its claims. Then I went back and saw that it was written by Paul Alcorn. If I had seen the byline earlier, I would not have been surprised at how bad the article was, since I have read a number of things he has written in the past. As far as I can tell, he has no educational credentials in statistics or engineering, and it really shows. Compare that to the credentials of Brian Beach, who has had a distinguished engineering career and has a strong background in engineering and statistics.


Rather than waste time on Alcorn's article, just consider that there is only one important question to consider when deciding whether backblaze's data can be useful in making HDD purchase decisions: whether any of Backblaze's methodology is flawed in a way that favors one manufacturer over another. Few of supposed methodological flaws mentioned in Alcorn's article obviously favor one manufacturer, and where there is some small possibility of bias, Beach has already discussed it in his original article.
 

·
Banned
Joined
·
29,681 Posts
Discussion Starter · #5 ·

Quote:
Originally Posted by jim2100  /t/1515424/dispelling-backblazes-hdd-reliability-myth-the-real-story-covered/0_100#post_24297641


Not really. I skimmed quickly through the article and kept thinking how poor the logic was and how sorely lacking it was with any supporting evidence to its claims. Then I went back and saw that it was written by Paul Alcorn. If I had seen the byline earlier, I would not have been surprised at how bad the article was, since I have read a number of things he has written in the past. As far as I can tell, he has no educational credentials in statistics or engineering, and it really shows. Compare that to the credentials of Brian Beach, who has had a distinguished engineering career and has a strong background in engineering and statistics.


Rather than waste time on Alcorn's article, just consider that there is only one important question to consider when deciding whether backblaze's data can be useful in making HDD purchase decisions: whether any of Backblaze's methodology is flawed in a way that favors one manufacturer over another. Few of supposed methodological flaws mentioned in Alcorn's article obviously favor one manufacturer, and where there is some small possibility of bias, Beach has already discussed it in his original article.

I am amazed you know the author. Are you sure he has no credentials or just guessing ? What makes you dislike him ? Any other bad articles ?
 

·
Registered
Joined
·
346 Posts

Quote:
Originally Posted by jim2100  /t/1515424/dispelling-backblazes-hdd-reliability-myth-the-real-story-covered#post_24297641


Not really. I skimmed quickly through the article and kept thinking how poor the logic was and how sorely lacking it was with any supporting evidence to its claims. Then I went back and saw that it was written by Paul Alcorn. If I had seen the byline earlier, I would not have been surprised at how bad the article was, since I have read a number of things he has written in the past. As far as I can tell, he has no educational credentials in statistics or engineering, and it really shows. Compare that to the credentials of Brian Beach, who has had a distinguished engineering career and has a strong background in engineering and statistics.


Rather than waste time on Alcorn's article, just consider that there is only one important question to consider when deciding whether backblaze's data can be useful in making HDD purchase decisions: whether any of Backblaze's methodology is flawed in a way that favors one manufacturer over another. Few of supposed methodological flaws mentioned in Alcorn's article obviously favor one manufacturer, and where there is some small possibility of bias, Beach has already discussed it in his original article.

I agree with you - this article talks about the data Backblaze presented as a myth... considering that they presented the facts of their own experience with thousands of drives, I'm not sure how anyone can logically claim it's a myth. Their methodology, while not perfect by any means, was probably more representative of the experience that a typical consumer is likely to have. An enterprise customer can buy direct from the manufacturer, get drives shipped properly every time (no poor packaging, no poor delivery handling, etc), and use enclosures built to maximize drive reliability over cost savings. Backblaze used consumer drives that may or may not have been packaged and delivered perfectly. Every drive group would have suffered from a certain amount of delivery abuse, which I strongly suspect to be one of the leading causes of early failure.


The only possible point that I think this article makes is the lack of clear data on survival rate in each pod design. Beyond that, there's not really a bias shown to one manufacturer or another. It is also necessary to recognize that the Seagate 1.5TB failure rates shouldn't count as a mark against overall Seagate failure - they were a known bad line of drives that are an anomaly compared to the other capacities.
 

·
Registered
Joined
·
744 Posts

Quote:
Originally Posted by Mfusick  /t/1515424/dispelling-backblazes-hdd-reliability-myth-the-real-story-covered/0_100#post_24298591


I am amazed you know the author.

I'm surprised you do not. You remind me of him (Paul Alcorn, aka computurd). A lot.
 

·
Banned
Joined
·
29,681 Posts
Discussion Starter · #8 ·

Quote:
Originally Posted by mcturkey  /t/1515424/dispelling-backblazes-hdd-reliability-myth-the-real-story-covered/0_100#post_24298706


I agree with you - this article talks about the data Backblaze presented as a myth... considering that they presented the facts of their own experience with thousands of drives, I'm not sure how anyone can logically claim it's a myth. Their methodology, while not perfect by any means, was probably more representative of the experience that a typical consumer is likely to have. An enterprise customer can buy direct from the manufacturer, get drives shipped properly every time (no poor packaging, no poor delivery handling, etc), and use enclosures built to maximize drive reliability over cost savings. Backblaze used consumer drives that may or may not have been packaged and delivered perfectly. Every drive group would have suffered from a certain amount of delivery abuse, which I strongly suspect to be one of the leading causes of early failure.


The only possible point that I think this article makes is the lack of clear data on survival rate in each pod design. Beyond that, there's not really a bias shown to one manufacturer or another. It is also necessary to recognize that the Seagate 1.5TB failure rates shouldn't count as a mark against overall Seagate failure - they were a known bad line of drives that are an anomaly compared to the other capacities.

I can't imagine a scenario where if you made the study better would somehow suggest Hitachi was not reliable. If they did good, they did good. Period. They seem head and shoulders above the rest. Bigger sample size and period tested, and Hitachi was rather consistent across all their models. If I had to draw a conclusion it would be that Hitachi was good.


But is seems Seagate might have been disadvantaged a bit, and WD did not have enough tested long enough. Only 346 WD RED models with avg age being only .5 years is not enough data for me. Hitachi had over 12,000 hard drives with average age 2 years, and they tested thousands of each model and each model did well. Hard to poke holes in that. Perhaps some redemption or a rematch for Seagate is needed, but I can't see how you can knock Hitachi for doing well in the less than ideal scenario. If you made the test easier would Hitachi do worse ?? I doubt it.


I'd love to see the newer Seagate models and WD models and how they hold up over the next year. That will be interesting to me.
 

·
Banned
Joined
·
2,696 Posts

Quote:
Originally Posted by Mfusick  /t/1515424/dispelling-backblazes-hdd-reliability-myth-the-real-story-covered#post_24298908


I can't imagine a scenario where if you made the study better would somehow suggest Hitachi was not reliable. If they did good, they did good. Period.

The problem is, that's not a logical conclusion you can arrive at from that post.


You can certainly conclude that those specific models of Hitachi drives are reliable. They certainly didn't skew anything to make the drives last longer.


The problem is you can't just assume that based on a couple of models used that the entire brand is particularly reliable.


If you're going to allow for dismissing the Seagate 1.5TB drives as "a bad model" then you have to allow for the possibility that the Hitachis used could have been "a good model"


You can't pick and choose your outliers on a whim.
 
  • Like
Reactions: assassin

·
Registered
Joined
·
13,150 Posts
I think almost any article (good or bad, pro or con) on this subject is worthy of reading. Extremely relevant to this forum where we are all (okay many of us anyway) looking to store our media.
 

·
Registered
Joined
·
744 Posts

Quote:
Originally Posted by assassin  /t/1515424/dispelling-backblazes-hdd-reliability-myth-the-real-story-covered/0_100#post_24299170


I think almost any article (good or bad, pro or con) on this subject is worthy of reading.

What an odd thing to say. What use is an article that states a number of incorrect things, or talks nonsense? What if Mfusick's posts on the subject of HDDs were collected into an article?
 

·
Registered
Joined
·
13,150 Posts

Quote:
Originally Posted by jim2100  /t/1515424/dispelling-backblazes-hdd-reliability-myth-the-real-story-covered#post_24299178


What an odd thing to say. What use is an article that states a number of incorrect things, or talks nonsense? What if Mfusick's posts on the subject of HDDs were collected into an article?

Because just as you have done it brings light to the subject. I had no idea about the very insightful things you brought up about the author. Now I do. And now I understand both the rebuttal and the original article much better. And it strengthens the original article even more, imo.


And just like mfusick's confused posts will be refuted and disproven with actual fact and science and not just loud and overly wordy opinion.
 

·
Banned
Joined
·
29,681 Posts
Discussion Starter · #13 ·

Quote:
Originally Posted by ajhieb  /t/1515424/dispelling-backblazes-hdd-reliability-myth-the-real-story-covered/0_60#post_24298997


The problem is, that's not a logical conclusion you can arrive at from that post.


You can certainly conclude that those specific models of Hitachi drives are reliable. They certainly didn't skew anything to make the drives last longer.


The problem is you can't just assume that based on a couple of models used that the entire brand is particularly reliable.


If you're going to allow for dismissing the Seagate 1.5TB drives as "a bad model" then you have to allow for the possibility that the Hitachis used could have been "a good model"


You can't pick and choose your outliers on a whim.

You make a valid point. Perhaps other models may not be the same, so you can't paint the entire MFG with a broad brush. I agree and have said this before too.


But if you were forced to make a guess or a purchase decision the limited data does suggest Hitachi a good choice. That's not conclusive but it's still something.
 

·
Registered
Joined
·
1,390 Posts

Quote:
Originally Posted by assassin  /t/1515424/dispelling-backblazes-hdd-reliability-myth-the-real-story-covered#post_24299170


I think almost any article (good or bad, pro or con) on this subject is worthy of reading.

That's the attitude that propagates the "teach the controversy" crap. The notion that all theories are valid and all opinions worthy, this is the very essence of anti-science.


The article in this thread is click-bait trash. The author stacks hypothetical upon hypothetical to rebut a pile of data. He neither supplies new evidence nor performs any statistical analysis to support his conclusion that Seagate merely fell victim to a syzygy of biases. Amusingly, his attempt to generate ad traffic failed here, because the one thing Mfusick respects less than science is other people's copyrights.
 

·
Registered
Joined
·
13,150 Posts

Quote:
Originally Posted by EricN  /t/1515424/dispelling-backblazes-hdd-reliability-myth-the-real-story-covered#post_24300153


That's the attitude that propagates the "teach the controversy" crap. The notion that all theories are valid and all opinions worthy, this is the very essence of anti-science.


The article in this thread is click-bait trash. The author stacks hypothetical upon hypothetical to rebut a pile of data. He neither supplies new evidence nor performs any statistical analysis to support his conclusion that Seagate merely fell victim to a syzygy of biases. Amusingly, his attempt to generate ad traffic failed here, because the one thing Mfusick respects less than science is other people's copyrights.

Very true. What I meant was that I like articles that get people talking about actual reliability DATA and in that regard I like all articles because it gets us talking.(especially on a subject like this where there is almost zero actual data).


Some of the best articles in science are articles about experiments or trials that fail to prove their hypothesis. With the new information provided about the rebuttal author that is where we are which makes the original article that much better, imo. I didn't know this before because I didn't know much about the author.
 

·
Banned
Joined
·
29,681 Posts
Discussion Starter · #17 ·
I haven't seen any evidence of the author that's negative only Jim's opinion on him.


Not saying Jim is wrong or I don't agree either; just the jury might still be out rather than case closed. I don't actually know enough to have an opinion on the author. I just thought the article was interesting. I agree with Assassin any article is a good one if it gets information spread or discovered.
 
1 - 20 of 129 Posts
Top