frustrated with search engine!  

Hi! It looks like it's set up to check the most recent 300 posts now by default, but you should still be able to get older results by selecting a range under the "Search For Posts From..." dropdown box and pick how far back you want to go. It's been working for me using that.

I also find that it is not working. I frequently search based on a word or words that I am positive is in the thread, usually the title, and I consistently get 0 hits back. I keep searching in roundabout ways, and eventually I find it.

Here is an example and no I'm not kidding. Do a search on "puke dude" (don't ask :D). You will get a result on this thread:
http://www.avsforum.com/avs-vb/showt...ight=puke+dude
but you will not get a result on this thread (and you should)
http://www.avsforum.com/avs-vb/showt...ight=puke+dude

Eeek! Looks like I'm -way- wrong on my last post (above)! No matter what date range I try, using the search terms:

plasma faq

for the Plasma/LCD forum just brings up the most current thread. That should be bringing back a lot of hits.

It is being looked into. Have no fear. :)

Phew!

Thought I was losing my mind there for a while. Couldn't understand why "HTPC crt resolution" wasn't returning any results !!

I'll await the fix with baited breath.....

Ok...With looking into this issue...here is the major issue here...

The search function of a site this large is the #1 issue with site performance. The reason is because each time a search is done, that has a large number of returns, say over 3000, it needs to LOCK the search index and make a TMP file in order to sort it. When that happens, all other DB functions get put on hold and the DB server SLOWS WAY DOWN for all the requests (from a lot of people on-lone) are just waiting their turn.

So...to get around this issue I had tried to limit the search results that get returned. I set a starting point for the return for the most recent 300 (The default) and the DB server was happy again. This is great when you are only looking for ONE word. The issue that has come up now however is looking for more than one word (normal I know). Here is what happens in the world of MySQL database when you do this...

Say you look for HOME DEPOT...The database looks up the first word...HOME and finds the first 300 results and holds them. Then it goes and looks up the second word DEPOT and holds that. Great, it now has found the most recent 300 posts with the word HOME and DEPOT in them...but your request was to find posts that had both the words in the same post...so now it takes the found sets of both and looks to see which post of the found sets contain BOTH words and gives you the result. Thus why it may have found 300 posts for each word, the limit, it only returns 15 posts for they are the only ones with both words in the 300 post of each found.

The word HOME returns = 298
The word DEPOT returns = 297

But of both found sets...only 15 contain both.

Thus our issue now. The problem is, because this is how the database, not the software, functions...we have no work around. What we are going to do however is expand the search numbers further and make something like Quick Search (Most Recent), Moderate Search (more returns) and Extended Search (Most Returns). Or something like that. We need to limit the search results for we just have too much traffic and even with a FASTER DB server...we would still have this issue due to the MySQL database issue in this area.

We hope to be able to have the new lookup on-line later today. I hope the above helps explain the issues involved.

The other option is another site just for AVS Archives and we pull off all but the current year from this site and move it to an archive that you can not post to.

I've not used this brand of forum software before, but I've heard there's a way to duplicate the forum data every now and then, say once a day, and use THAT for searching. That way, you're not locking the 'real' data for searches.

Thanks David. I believe that did the trick.

(knock on wood)

Hopefully most will use the default so it doesn't slow us down. But once again, I now seem find most anything quickly. :cool:

Crossing mt fingers.


Randy...Yes, it can be done, but with alot of work arounds. Not to mention...you will then never see the live data.

Still having problems. I do a search on a single uncommon word (more than three characters) and the thread isn't found and I only get two or three results. I do a search a different uncommon word and the thread is found. I can give examples if you need them...

I do not faind any issue I am sorry to say.

http://www.avsforum.com/avs-vb/showt...258#post912258

OK, here's an example. The thread above contains the words "radar detector".

Do a search on the word "radar" and enter my user name "QQQ" in the search by user name field to limit the number or results. You will NOT get a hit on the thread above.

Now do a search on the word "detector" and enter my user name "QQQ" in the search by user name field to limit the number of results. This time you WILL get a hit on the thread referenced above.

Both searches should find the thread but only one does. I find this happening frequently.

Did you open it up to "Slowest Search (all possible results)" in the options? I just tried "radar", "detector" and radar detector" all three find that thread for me. Guess I did also search "Show Results as threads".

I can't recall a time it hasn't found something I knew was there, since we got these current search options.

Come to think of it... this whole forum has been performing well for me as of late, knock on wood... and of course... thanks David! :)

That does it. David keeps changing the default options on us and I didn't notice :). I'm still not sure why it would find one word but not the other using the default but as long as I can select the "slow search" it's great.

The bad news is "slow search" hurts the server the most. :(

But if the other is not even accurate, it's really the only one we should use... right? Hate to say that because I know you've said it taxes the server, but inaccurate searches are pointless.

Well, it definitely makes me less likely to tell members (especially new ones) to use the search feature, since they will use the default options. In fact I seem to notice more and more threads lately where someone says "I did use the search feature but I couldn't find anything".

On the other hand, if it means the site slowing to a crawl, or a less than perfect default search, that's an easy choice for David.

Randy...Read the reason for this above. It is not that it is not correct results, but how MySQL does the search with a limit put on it.

I understand the reason and I'm not placing blame or anything - just stating that if the results aren't accurate it's difficult to use the "fastest" search option.

Honestly this website is a great wealth of information, I love it and use it often, but it is starting to crumble under its own weight.

During the day the system is totally unusable. At this moment there are over 1400 users online and the SQL database is at 98.5% load.

I sit on a 3mb pipe and have no problems talking to any website with snappy results the world over. However at the middle of the day eastern time this site is deadly slow (30 seconds to change pages) searches are turned off for hours at a time.

I am curious about stats on the system, operating system and how often the server is at 90+% load.

Bottomline is this, you have a wealth of information and no ability to search it at all, search it accurately, or quickly. Therefore you have a pile of data.

Frankly I would pay for access to this site were the above issues resolvable or resolved.

You obviously don't generate much revenue here, how can we help?

I say put a Paypal donate button online and lets buy some real hardware or software!

Thanks for listening!

WE just bought new hardware. The issue is MySQL can not support a large effort and this software uses MySQL. The thing that hurts this site is the searching. We have over 1.6 million posts...that is a lot of data and when a search is done that returns a lot of hits, MySQL has to lock the Database to make it happen...thus bottlenecking the rest of the site.

Current hardware...

Web Server = Dell PowerEdge 2650 with Dual 2.4 Ghz Processors and 2 Gigs DDR memory. Using Dual 18 gig 15K RPM drives sitting in a mirror config on a 128 meg Caching Controller.

DB Server = Web Server = Dell PowerEdge 2550 with Dual 1 Ghz Processors and 2 Gigs ECC memory. Using three 18 gig 10K RPM drives sitting in RAID 5 config on a 128 meg Caching Controller.

We are not maxing out the processors, but MySQL is not too happy with the searches.

We need to dump off some data to another DB and use it as a archive I fear.

I feel for you David. MySQL is fairly good for reads (which, of course are the majority of your calls) but not too clever with writes.

It's a shame that VBulletin only works with MySQL - something like a SQL-Server back-end would be more scalable and efficient.

Did you ever enquire whether they are planning to port the app to another back-end DB? Maybe with a major site like this asking, they might consider it.

Andy.

I wish! They have no interest in porting anyting over. They have enough issues supporting one platform. Such a same...great software but can not handle larger sites like this. Most of their clients are not anywhere close to this size.

What about distributing the load along several servers?
It seems that the Tivo community is actually peeled off to another box or service, I don't know how good you are at SQL or what your resources are but you could split the system into quarters or eighths on multiple boxes.

The key I think is to find a system that supports a real scalable SQL system and port the data but that can be a time consuming, costly task.

Ugh, this is depressing.
Why didn't you know when you started this that it would be so successful? Was your crystal ball in the shop that week? <grin>

How old is the oldest message? (just curious)

We can have all the web servers we want, but the data still have to be in one place. That is the issue.

Message age? Hummm...not sure.

I know you've just upgraded but 15Krpm drives would be sure to give you some relief.

Aplogies if this is all stuff you're already familiar with, but these should also help:
a) Put Data and Index files on different physical drives. Create separate RAID arrays (a good way to do it is to put the Data on a RAID-5 array, and the indexes on a RAID-1 array - this increases read speed).
b) Investigate Replication. I don't know much about Replication in MySQL, but in Oracle this works well. You should be able to query against the replica database, and keep the master for updates. Again - keep the replica and master on different drives.
c) Keep your TMPDIR on a different drive from your data and index. This is typically very active so a single RAID-1 array is a very good choice.

I hope some of this is of use.
Andy.

Hi David.
I think the search engine stopped working again.
I just did a search for "remap channels" as well as "remap channel" under the ReplayTV forum using the Slowest Search Method and I got only recent results. For example, the following thread (which I bookmarked) did not show up on the results:
http://www.avsforum.com/avs-vb/showt...hreadid=152718