Comments on: Make Cross-Domain AJAX Requests with xdRequest

By: GeekLad

GeekLad — Wed, 03 Feb 2010 23:42:17 +0000

In my demo app, you would have to click through and load enough pages for
your browser to trigger the limit.

By: GeekLad

GeekLad — Wed, 03 Feb 2010 23:41:42 +0000

The rate limit is on the client IP. Since everything takes place in the
browser window (the browser makes the calls to YQL), it would be based on
the browser's IP address.

By: Paul

Paul — Wed, 03 Feb 2010 22:58:50 +0000

As an example, in your demo app you created – what would need to happen in an hour for it to exceed the limit. Maybe that makes more sense as to what I am trying to ask! Thanks again.

By: Paul

Paul — Wed, 03 Feb 2010 22:46:01 +0000

Hi - awesome script! I have one question though that I'm not clear on. The rate limit per IP. let's say I put this script on my site, every time it is triggered in someone's browser, is that instance attributed to my site's IP or the users? Also - if this page is calling data from the same remote site (say CNN.com), will CNN.com be blocked to users of my site after 10000 page views? Of course only if this occurs in an hour. I'm just trying to figure out exactly what IP they are talking about, the user, the site that host's the .js file, or the site that is being "scraped"?

I hope I am making sense. Thanks again!

By: David Hopkins

David Hopkins — Mon, 16 Nov 2009 23:09:54 +0000

Thanks. No change though. I think the issue is with MooTools. I believe they have modified the Array object and I think that is causing the issue.

I have posted a detailed report in the MooTools forum. If someone can find out what's wrong I will let you know.

By: GeekLad

GeekLad — Mon, 16 Nov 2009 22:57:46 +0000

Hmm, I wonder if the version I have in the download section is an older
(damaged version). That for loop doesn't look as though it has been fixed
to work in Internet Exploder. Are you using IE? You may want to try
downloading the latest source in the trunk here:
http://xdrequest.googlecode.com/svn/trunk/xdReq...

By: David Hopkins

David Hopkins — Mon, 16 Nov 2009 22:07:07 +0000

It's a conflict with that in Mootools and the following in xdR:

for(cookie in cookiejar) {
if(cookiejar[cookie].match(url, this.secure())) {
outputCookies.push(cookiejar[cookie]);
}
}

Namely this bit:

cookiejar[cookie].match(…

It's saying that cookiejar[cookie] is an object when I guess it should be a string. However, if I bypass this, it just breaks.

I think it could be to do with some Mootools meddligns with the natives of JS, probably the array since that is what is conflicting in Moo. However, I am very rusty with JS. Maybe its time to moove on

By: GeekLad

GeekLad — Mon, 16 Nov 2009 21:41:24 +0000

Hmm, that's odd. I'm not very familiar with Mootools. I'm not sure why
xdRequest would cause Mootools to break.

By: David Hopkins

David Hopkins — Mon, 16 Nov 2009 21:25:42 +0000

I found out why it doesn't work in conjunction with Mootools. There is a line:

var natives = {'Array': Array, 'Date': Date, 'Function': Function, 'Number': Number, 'RegExp': RegExp, 'String': String};

Which is on line 96 for me. For some reason xdRequest won't run unless you remove the ['Array': Array] part of that object. I'm not sure exactly why this is an issue though.

By: David Hopkins

David Hopkins — Mon, 16 Nov 2009 21:07:22 +0000

Hi,

Excellent project. Been looking for something like this for a while. I am having problems using this in conjunction with other JS files though, namely MooTools and not getting any error output.

I think it may be some variable conflict. More project specific var and func names may help this issue.

By: GeekLad

GeekLad — Sun, 25 Oct 2009 13:32:07 +0000

The YQL servers will see the IP of the client. The remote site will see the
IP of the YQL servers. YQL also does pass along some headers that include
information about the client (for example, a header called X-Forwarded-For
that contains the IP address of the client).

Because of the security measures enforced by browsers, the only way you can
make remote requests 100% within a client is with services that provide an
API that provide JASONP responses. Otherwise, you have to use some sort of
proxy that will fetch the data for you and provide a JSONP response, as YQL
does. I think YQL is the best solution I've seen out there for doing this.

By: Kevin Gigiano

Kevin Gigiano — Sat, 24 Oct 2009 22:57:27 +0000

I see your point.
Correct me if I'm wrong, but the requests made to YQL from the client, when visiting a site using your code.
YQL servers are going to see the IP of someone visiting a site, not the IP of the site itself?

I see your point on the opt out as well. All the more reason to skip Yahoo entirely.
Do you think there is a way to make remote requests completely on the client side.
If you could figure that out, say hello web 3.0!

By: GeekLad

GeekLad — Sat, 24 Oct 2009 22:43:35 +0000

Thanks for the kind words. Please be aware that the possibility of getting
blacklisted still exists. The requests are made via dynamically inserted
scripts to the YQL servers. YQL servers make the requests to the remote
pages and then send the data back to the client, so it is possible that some
servers may blacklist the YQL servers. YQL also provides an opt-out option
for content providers: http://developer.yahoo.com/yql/provider/

By: Kevin Gigiano

Kevin Gigiano — Sat, 24 Oct 2009 19:45:01 +0000

Awesome job!

About a year ago I tried to find a solution to do cross domain requests and was disappointed to find nothing worked.
I understand the security concerns. In certain circumstances, it makes sense not to allow a clients browser to make requests outside the domain of whatever site they happen to be. I can see how a DoS attack could be accomplished to

As a developer, not allowing cross domain scripting sucks. If I want to build a kick ass site that pulls data from multiple locations (many websites), I have to make my server(s) do all the work. At first this seems ok, but if I want that site to scale horizontally on a massive scale, I need lots of $$$ or a way to easily monetize the site (which isn't always obvious at first). Plus, those sites I use for data are eventually going to get pissed and ban the IP of my server(s).

Massive low cost scaling can only be accomplished if the client is able to do some of the work. Plus, I don't have to worry about any of my servers getting blacklisted because all remote data request are handled by the client and appear as normal traffic.
Your cross domain scripting solution is a major accomplishment, possibly an evolution of the web itself.

I applaud and thank you for your work!
You rock!

Kevin

By: GeekLad

GeekLad — Wed, 21 Oct 2009 15:17:37 +0000

I'd be more than glad to add you to the project. Welcome aboard!

By: Davide Zanotti

Davide Zanotti — Wed, 21 Oct 2009 15:11:53 +0000

This project rocks! I would like to contribute, I can refactor the code and make
some simplifications to use it in a more clear and practice way, like:
var req = new xdRequest({
method: "get",
url: "http://www.remotesite.com",
callback: function(data) {
// handle data
},
headers: [
{name: "headername1", value: "headervalue1"},
{name: "headername2", value: "headervalue2"}
]
});

req.execute();

...let me know!