XML query and parsing

[spacestout]

I'm not sure where to post this. I've been hard-linking search queries to a search engine and getting their branded search results. Now we're transitioning to an XML feed. This is what they want me to provide them:

"We require 4 CGI parameters in all your queries:

query - The users query
sip - The server IP address. This MUST be the IP address of the server from which the user is making the INITIAL request. In most cases this will be the IP address of your downstream partners.
uagent - The User-Agent HTTP header
uip - The users IP address (actual users IP address)
uid - A unique id for each user, preferably based on a cookie although if you can't do this, a hash on the users IP will do.

Result URL should resemble:

(removed so full link would show)://partners.searchengine.com/MyID?uagent=browsertype&sip=216.127.1.1&uip=234543 21&uid=123456&query=casino

Other parameters supported are:

timeout - max number of seconds to wait for results, defaults to 4.

rpp - results per page, defaults to 10

All of the parameters should be url-encoded."

How do I gather the requested info? Are there scripts that will do it? Or will I need to piece scripts together to get it all? Any good tutorials on what all the fields mean? What does "hashing the user's IP" mean?

Is it common knowledge to the XML-savvy just where the XML file will be returned to me? Or do I need to work this out with my search engine provider? Any good scripts on parsing the XML, displaying search results ala google format, and displaying multiple results pages?

Not asking for much am I? So, I need to pass a query with all kinds of user information to the search engine, receive an XML feed, parse it, and display it. Hopefully this is a useful thread for our community. It is all new ground for me.

[RTH10260]

I believe you have already received some answers in another thread, so this may be to fill some remaining blanks

Originally Posted by spacestout

How do I gather the requested info? Are there scripts that will do it? Or will I need to piece scripts together to get it all? Any good tutorials on what all the fields mean? What does "hashing the user's IP" mean?

When you code some simple PHP script your get all this information from the script environment, You get the resulting xml feed back by simply writing this line after building the request line: UNTESTED PSEUDOCODE

PHP Code:

<?
$querystring = 'uagent='.urlencode("browsertype") 
             . '&sip='.urlencode("216.127.1.1") 
             . '&uip='.urlencode("234543 21" 
             . "&uid=".urlencode("123456"
             . "&query=".urlencode("casino");
$request = 'http://partners.searchengine.com/MyID?' . $querystring;
$xmlfeed = file($request);
?>

Originally Posted by spacestout

Is it common knowledge to the XML-savvy just where the XML file will be returned to me? Or do I need to work this out with my search engine provider? Any good scripts on parsing the XML, displaying search results ala google format, and displaying multiple results pages?

Depending how the feed is structured, you will not be parsing this yourself. You will create a CSS file that describes the look and feel of the content. You will write a few wrapper lines of XHTML and just copy the feed as-is into the page.

Should you want to rearrange fields in different order from the input you get back from the search machine, and if your client browsers are MSIE, then you could also send a XSLT transformation description and the IE browser will resolve the file at the client side. I am not uptodate how far FF and other browsers support such advanced XHTML features.

[spacestout]

Thanks for that direction Richard. I posted a reply last night, but I don't see it here today so I'll start over.

My website appears as a search engine to the user. However, I pass the query to a major search engine and they give me an XML file in return. For example, simply typing in the browser http://partners.searchengine.com/MyIDP?query=cars, displays in my browser:

<?xml version="1.0" encoding="ISO-8859-1" ?>
- <searchengineRESPONSE>
- <cookieinfo>
<serverip>xx.xx.xx.xx</serverip>
<cookie />
<totalresults>26</totalresults>
</cookieinfo>
- <items>
- <item>
<title>Find new & used Cars at CarsDirect.com</title>
<url>http://xx.xx.xx.xx/Search?fig=3150&e...FeQFNvXVs=&cb= etc. etc.

So, just typing in the query string in the browser displays the XML file. So, I need to gather the SIP, UIP, UAGENT ,make the UID, and encode it, and then pass it to a file that parses and formats the results. I see from your example how to encode the variables. Do I use the $_SERVER variable to obtain the the SIP, UIP and UAGENT?

I can see making a search.php file to gather the user info and put the query string together. I've been experimenting with a wysiwyg XML editor. It seems like I can convert the XML for display thru XSLT. What I'm not sure on yet is whether this will be a separate file that I'll pass the query string to.

Any thoughts on the best way to do all this?

[RTH10260]

Originally Posted by spacestout

I see from your example how to encode the variables. Do I use the $_SERVER variable to obtain the the SIP, UIP and UAGENT?

The user agent field is $_SERVER["HTTP_USER_AGENT"]
The users IP address is $_SERVER["REMOTE_ADDR"], from the description I don't know what the difference between SIP and UIP should be, use the same for both.
PHP offers you a UID in the form of field $_SERVER["UNIQUE_ID"], just use this magic string as-is, it is guaranteed to be unique. Should you need to send the same UID for followup requests, you would have to save the first UNIQUE_ID in a cookie or in a session variable.

[RTH10260]

Originally Posted by spacestout

I can see making a search.php file to gather the user info and put the query string together. I've been experimenting with a wysiwyg XML editor. It seems like I can convert the XML for display thru XSLT. What I'm not sure on yet is whether this will be a separate file that I'll pass the query string to.

Any thoughts on the best way to do all this?

If you wish to manipulate the XML result file, eg to reorder fields for the display order of your choice, or to remove data that a user shouldn't get to see in the XHTML source of the webpage, then after retrieving the file:
$xmlfeed = file($request);
you save it to a temporary file on the server, apply a XSLT file onto the temporary file using available php extension functionality at the server, no need for much own development, just call the package, then send the output file to the browser with the CSS file that describes the rendering. Use a cronjob to remove the temporary files on the server at intervals.

[spacestout]

You're a godsend Richard. Thanks.

As far as a unique user IP, I'll have to find out if it has to be the same each time for a given user. I also need direction on the SIP. Is this my websites server IP or if my searchbox gets put on someone elses site, then theirs?

If I use the code suggested by Satis on the PHP forum for hashing the UIP

<?
$uid = md5($_SERVER['REMOTE_ADDR']);
?>

will this be the same value for the same uip each time, or does md5 do a random encoding?

Looks like I've got a lot of digging to figure out server extensions and cron jobs.

[RTH10260]

Originally Posted by spacestout

As far as a unique user IP, I'll have to find out if it has to be the same each time for a given user. I also need direction on the SIP. Is this my websites server IP or if my searchbox gets put on someone elses site, then theirs?

IP numbers tend to be the same for quite a while. For a dialup user the number could change if the line gets disconnected. On cable/DSL the number tends to remain same over long time for a specific user.

Originally Posted by spacestout

If I use the code suggested by Satis on the PHP forum for hashing the UIP
. . .
will this be the same value for the same uip each time, or does md5 do a random encoding?

md5 is not a random number generator, else it would be of no value. It's a trapdoor function, eg there exists no reverse function to regain the original value. As such it is (or was) a way to encode passwords. Since about a year it is not recommended to be used for passwords anymore. The cracking algorithm for a certain class of md5 values (words up to 14 characters of alphabet, numbers and some special chars) can turn up the original value within about 30 minutes using huge lookup tables (40BG - 60GB).

Originally Posted by spacestout

Looks like I've got a lot of digging to figure out server extensions and cron jobs.

You are not looking for 'server' extensions but 'php' extensions !
Ref according to the module list at PW: http://us3.php.net/manual/en/ref.domxml.php
(I have not been making sufficient tests to see if it works as advertised. If it doesn't add them to your own account space)

The cronjob is the same as used for clearing up php session files, just applied to the directory where you would be keeping temp files, like:

51 2 */1 * * find $HOME/temp/sessions/ -mtime +2h -delete >>$HOME/logs/cron/clearsessions.txt 2>&1

[spacestout]

Great Richard.

I found that the SIP is my website's server IP, or if my search code is on someone elses site, then it is their site's server IP. The UID can vary, so I'll use the "UNIQUE_ID" you suggested. So, it looks like I have all the pieces to get the user's info, encode it, and make a query.

Does it make sense to have all the search code on my main site, or is it better to make a file such as search.php for getting the user's parameters, encoding, and displaying the query?

The user will initiate a search or query via a search box or hard link on my main site. For example, I may have a hard link:
<a href="search.php?query=casino >casino</a><br>
Then search.php produces the end result $xmlfeed = file($request);

To display the results in a browser ala google style do I create a .xsl file using XSLT? Does the .xsl file reside on my server? Is it listed in the search.php file? How do I pass $xmlfeed into the .xsl (and launch a new browser window). Is it just one more line of code at the end of search.php? Since I created the .xsl to only pick up certain fields from the XML (so no unwanted fields are showing), do I avoid having to do the temporary file and delete it with a cron job? Would I have to do anything special with the read/write settings of the search.php or .xsl files?

Or am I missing a major point of how $xmlfeed is going to get it's contents displayed?

[spacestout]

I'll restate my question so maybe it will be more clear.

If I create search.php and put all the code that takes the query, adds user information, encodes, and produces
$xmlfeed = file($request); I'm wondering what next. How do I get the query string in $xmlfeed to the search engine? Or do I not actually need to send it to them? $xmlfeed contains a URL, which if I just type in the browser, gives me the XML feed. So what is my next step?

I have the Stylus wysiwyg XML editor. Using that, am I creating a file that will take the XML, apply CSS and parsing via XLTS, and display the HTML results in the browser? If so, do I just include this display file in my Search.php file and pass the results of $xmlfeed to it?

If I'm off base, having derived $xmlfeed in search.php (and you knowing I have the XSLT editor), how do I get get the xml parsed, formatted, and displayed in a new browser window?

[RTH10260]

Originally Posted by spacestout

If I create search.php and put all the code that takes the query, adds user information, encodes, and produces
$xmlfeed = file($request); I'm wondering what next. How do I get the query string in $xmlfeed to the search engine? Or do I not actually need to send it to them? $xmlfeed contains a URL, which if I just type in the browser, gives me the XML feed. So what is my next step?

Let me repeat the code snippet from further up this thread:

PHP Code:

<? 
$querystring = " ... whatever ... "); 
$request = 'http://partners.searchengine.com/MyID?' . $querystring; 
$xmlfeed = file($request); 
?>

When you check the PHP manual for function file() (http://us3.php.net/manual/en/function.file.php) you will see that this not only accesses local files, but thru 'wrappers' it can access files with several different protocols. One is HTTP, and when the function sees http:// as part of the filename, it will submit that webrequest and return the reply from the server.

In essence, after file() your result $xmlfeed contains the reply from the server. When the server returns a xml file as in your case, then $xmlfeed is an array of lines with that file content. That's the whole magic for you to get that xml file to work with in your application.

Originally Posted by spacestout

I have the Stylus wysiwyg XML editor. Using that, am I creating a file that will take the XML, apply CSS and parsing via XLTS, and display the HTML results in the browser? If so, do I just include this display file in my Search.php file and pass the results of $xmlfeed to it?

I am very sure you don't want to sit at your pc, waiting around th clock, until a visitor at your website submits a search request, and then first prepare the final response at your desktop

As I alrady explained in post #7 and I will repeat it at this point again, your server script takes that xml file in $xmlfeed and processes it. The script will need to save the content of the variable $xmlfeed in a temporary file, then call a PHP package for XSLT processing with the required parameters, and use the output of the package to send the webpage back to the visitor. The URL to the php routines is listed in the prior post.

Note that the XSLT transformation is only required if the fields and groups of fields in the xml file are not in the same order as needed to present the content on a webpage.

Originally Posted by spacestout

If I'm off base, having derived $xmlfeed in search.php (and you knowing I have the XSLT editor), how do I get get the xml parsed, formatted, and displayed in a new browser window?

It doesn't necessarily have to be a new browser window, the output is just a reply to a search request.

What you should be using for making this ourput as easy as possible, is to prepare this webpage to the XHTML standard. Not plain HTML anymore, but content prepared as a XML compliant file, with a couple of special directives upfront, and all formatting included as CSS, either local in the webpage, of as separate CSS file. Any HTML needs to be 4.01 standard and slightly adjusted to the XHTML rules.

Your reading assignment:
The XHTML standard in general - http://www.w3.org/TR/2002/REC-xhtml1-20020801/

This section and the next have small samples of xhtml files, so that you can see the some differences to a common html webpage:
http://www.w3.org/TR/2002/REC-xhtml1-20020801/#strict.
The second and third sample shows how a xml file appears inbetween the BODY and /BODY tags. That's how your script needs to create the webpage: echo the leading html tags and then just 'echo $xmlfeed;', then echo the final tags.

The differences in coding a plain html webpage and one that is xhtml compliant are explained here:
http://www.w3.org/TR/2002/REC-xhtml1-20020801/#diffs
and in
http://www.w3.org/TR/2002/REC-xhtml1...01/#guidelines

To format the xml file content (as shown in post #3) within the webpage, one creates a CSS definition. The tags of the xml file are directly the names to be used as reference in the style defintion. Sample code in
http://www.w3.org/TR/2002/REC-xhtml1-20020801/#C_13
and
http://www.w3.org/TR/2002/REC-xhtml1-20020801/#C_14