.. include:: /Includes.rst.txt .. _analysing-indexed-data: ========================== Analysing the indexed data ========================== The indexer is constructed to work with TYPO3's page structure. Opposite to a crawler which simply indexes all the pages it can find, the TYPO3 indexer MUST take the following into account: - Only cached pages can be indexed.Pages with dynamic content - such as search pages etc - should supply their own search engine for lookup in specific tables. Another option is to selectively allow certain of those "dynamic" pages to be cached anyway (see the static_page_arguments concept used by some plugins) - Pages in more than one language must be indexed separately as "different pages". - Pages with message boards may have multiple indexed versions based on what is displayed on the page: The overview or a single message board item? This is determined by the static_page_arguments value. - Pages with access restricted to must be observed! - Because pages can contain different content whether a user is logged in or not and even based on which groups he is a member of, a single page (identified by the combination of id/type/language/static_page_arguments) may even be available in more than one indexed version based on the user-groups. But while the same page may have different content based on the user-groups (and so must be indexed once for each) such pages may just as well present the SAME content regardless of usergroups! This is the very most tricky thing. .. _complex-scenarios: Understanding these complex scenarios... ======================================== The best thing to do is to grab an example. Please refer to the picture below while reading the bulletlist here: #. The overview in general shows **one line per "phash-row"** (a single row from the index\_phash table). Such a row represents a single hit in a searching session. In other words, each line with grayish background in the overview may be a search-hit. The columns of these rows are: - Title: The search-result title. - [icon]: Click here to remove the indexed information for this entry (will be re-indexed on the next hit). - pHash: The "id" of the search row. The hash is calculated based on id/type/language/MP/static_page_arguments/gr\_list of the page when indexed. For external media this is based on filepath/page-interval (for PDF's only) - cHash: Calculated based on the actual content which was indexed. - rl-012: This is the rootline ids for level 0,1,2. Used when searching in certain sections. For instance a search-operation may select all pages with rl1=123 which will result in a search within pages which exist ONLY in the branch of the website where the level1-page has uid=123. - pid.t.l: This is the page-id / type-number / sys\_language uid - Size: How many bytes the indexed page consumed - grlist: This is the gr\_list of the user which initiated the indexing operation. - static_page_arguments: Additional parameters which are identifying the page in addition to the id/type number which usually does that. #. The page "Content elements" has one indexed version. The page-id of the root-page is "1" and the page on level-1 in the rootline had the uid "2". Notice how all subpages to "Content elements" has the exact same rl0 and rl1 value. Where the page "Content elements" does NOT have a value for rl2 so does all the subpages (because they ARE the level 2 themselves!). Furthermore the page has the page-id "2", a type-value of "0" and is indexed with the default language "0". The size was 10.6 KB and the user who initiated the indexing operation was a member of the groups 0,-2,1 (which is effectively fe\_group "1", because 0 and -2 is pseudogroups). #. On the page "Special content" there must have been a link to a local PDF and Word file, since those two are indexed in relation to this page. The PDF-file is located in the path "uploads/media/tsref\_onepage.pdf" relative to the website. Notice that the PDF file is actually indexed three times, one time per page. This is of course configurable. Each indexed section of the PDF-file has the potential to show up as a search-result row of course (because the phash is different per indexed part). The whole point with this is that a large PDF file might contain so much information that it might match all too many search-queries. So breaking a PDF-file down into smaller parts makes it possible for us to indicate exactly WHERE in the PDF-file the search word was found! #. Looking at the word file (and the PDF-file as well) we see that they are found on BOTH the page "Special content" and on the page "ISEARCH example". But looking at the phash values (for the word-file it is "268192666") it is the SAME value in both cases. So this means, that the Word and PDF file is indexed only once - when it is first discovered! Later when another page is indexed and a link to the same document appears, then the document is not indexed as another document, but rather an entry in the index\_section table is made indicating that this result row is also found available (linked to) from another page/section.Say you are doing a search in the section from "Content elements" and outwards in the page tree. The word- document is matched in the search, but it will appear only once in the search result. Now, if one of the two pages where the Word document was either hidden or access restricted the word-document would still be matched (because one of the pages is accessible for the user). But if BOTH pages with the link to the word document is not accessible for the user doing the search, then the word document will not be included in the search result. #. Here we can see that the pages "Special content", "Advanced" and "Menu/Sitemap" is indexed twice each. The reason is that those three pages has had different content depending on whether or not a user was logged in!In the case of the page "Special content" the reason is that the page contained a content element which was visible for users which was a member of group number #1. Therefore the page was different in the two cases.The page "Advanced" has a user-login form and that form looks different whether a user is logged in or not.Finally the page "Menu/Sitemap" apparently changed. There reason was that this page includes a sitemap and that sitemap displayed some extra pages when the logged in users hit the page and so the content was not the same as without login.Another thing which is interesting is that two different users must have visited those pages. We can see that because the page "Special content" was apparently indexed with the usergroup combination "1,2". Later another user hit the page but only a member of group "1". However the page content was the SAME. And because those two users saw the very same page, it was not indexed a third time, but it was instead noted down that a user with membership of only group "1" did also see this same page. That comparison was based on the cHash (contentHash) which is a hash-value based on the actual content being indexed. So when the user with group "1" only came to the page, the indexer engine realize that the page as it looked has already been indexed because another phash-row with that content hash was already available. #. These pages does not contain any tricks it appears. According to the grlist's both users with membership of group "1,2" and group "1" only *as well* as surfers who did not at all login ("0,-1" is the pseudo- group for no login) as visited the page. And because only one indexed version exist the page must have had the same content to present all users regardless of their login-status.The reason why the page "Your own scripts" does not contain a grlist value "0,-2,1,2" as the others do is simply because no user with that combination of usergroups has ever visited the page! #. txt and html documents can also be indexed as external media. In the case of HTML-documents the documents is detected and used. .. figure:: ../Images/ComplexScenarioOverview.png :alt: Several complex scenarios Several complex scenarios On the image below we are looking at another scenario. In this case the static_page_arguments is obviously used by the plugin "tt\_board". The plugin has been constructed so intelligently that it links to the messages in the message board without disabling the normal page-cache but rather sending the tt\_board\_uid parameter along with a so called "cHash". If this is combined correctly the caching engine allows the page to be cached. Not only does this mean a quicker display of pages in the message board - it also means we can index the page! .. figure:: ../Images/ComplexScenarioPlugin.png :alt: Complex scenario with tt_board Complex scenario with tt_board As you see the main board page showing the list of messages/threads ("Sourcream and Oni...") is indexed without any values for the parameter tt\_board\_uid (static_page_arguments). Then it has also been indexed one time for each display of a message. In a search result any of these five rows may appear as an independent result row - after all they are to be regarded as a single page with unique content, despite sharing the same page-id! Another interesting thing is that while the main page has inherited the page title for the search-result ("Sourcream and ...") each of the indexed pages with a message has got another title - namely the subject line of the message shown! Thus a search matching three of these five pages will not shown three similar page-titles but a unique page title relative to the actual content on the page. It is the tt\_board plugin that sets the page-title itself by an API-call. The only glitch here is that the tt\_board plugin has falsely allowed the main page to be cached twice. See the first and last phash-row. The last row has got the parameter "&tt\_board\_uid= " sent and the tt\_board plugin should not have allowed that! Because looking at the content hash of the first and last we realize that it's the SAME hash (84186444) and therefore the SAME content! However being two separate result rows they will both be displayed in the search result as separate hits. The responsibility for this lies with the plugin. However such occurrences can be automatically filtered out during the search result display. But it's better to avoid this kind of stuff. The last example below has three main issues to discuss: #. The page "Other languages" is apparently available in three languages. Which ones are not possible to determine unless we know the value from the sys\_languages table. In this case the default language (zero - 0) is english and the language with id 1 and id 2 is danish and german versions of the page.When a search is conducted each page may turn up as a result page but with a little flag telling if the page was found in another language than the main language on the website (see second illustration hereafter) #. If there is no phash-rows found for a page this can mean three things: #. Either the page is not cached. In this case both the tt\_products and tt\_news plugins apparently disables the caching of the page thereby disabling any indexing of the pages. Searching in news and products must be done with a searching function looking up directly in the news and products tables. #. In the case with other pages the reason may be that the pages has never been visited and therefore not indexed yet! Indexing of pages in TYPO3 happens during the rendering of the page - there is currently no "crawler" to assist this job. #. Finally the reason for a page not being indexed can be the combination of 1 and 2: That the page has never been visited. And if it was visited, the cache would have been disabled. #. These numbers just tells us that: - the page "Lists" was indexed once by a user with membership of group 1 and 2. - the page "Addresses" was also indexed by a user with membership of group 1 and 2 but has since been visited by a user without login. Both instances yielded a similar page and it was therefore not indexed twice.This raises the question about the page "Lists": Is that access- restricted for users without login or has a user without login just never visited that page since no "0,-1" grlist has been detected? Both could be the answer. On pages which has access-restriction (or a whole section in an intranet) such pages would obviously not have been indexed by no-login users. However in this case nothing indicates that the page should be hidden for non-login users and so we must conclude that the page has simply not yet been visited by a no-login user - otherwise it would look like the page "Addresses" having also the "0,-1" list detected. - The "Guestbook" page was indexed by a user without login only. .. figure:: ../Images/ComplexScenarioMoreExamples.png :alt: More complex scenarios More complex scenarios Finally the image below shows how localized versions are displayed in the search results .. figure:: ../Images/ViewResultsLocalized.png :alt: Localized search results Localized versions showing up in the search results