Make Cross-Domain AJAX Requests with xdRequest

October 15, 2009 · Posted in Programming 

xdRequest is an open-source JavaScript library that I’ve developed for making cross-domain AJAX requests. In this day and age, with web browsers becoming very fast and powerful, it has become quite possible to build web applications with much of the work performed by the client. xdRequest makes it possible to easily pull data from other sources and manipulate it all on the client-side. With xdRequest, you will no longer need to perform data extractions/manipulations from other sources on a server. It can all happen right in the browser.

Screenshot of the YQL html data table I had recently been fooling around quite a bit with YQL, which does provide some built-in features to make cross-domain http requests. It can be done with the html open data table. You can try it out from the YQL Console.  Just go to the data tables section on the right side and select Data > html. Unfortunately, the built-in html table is somewhat limited.  You can only make requests using the GET method, and it does not provide any mechanism for sending/receiving HTTP headers.

Someone in the YQL forums posted a YQL feature request to add POST and other HTTP request methods to the html table. Someone responded with the suggestion of building an open data table to do just that. I then started reading the open data table documentation, and came to realize that what you can do with YQL and open data tables is nothing short of astounding. You can use server-side JavaScript to request data from multiple sources and manipulate it in any way you see fit.

I started developing my own custom open data table for the particular project I was working on at the time, and eventually the project morphed into xdRequest. Because I needed to track cookies and redirects in my little project, I realized that it would be great to develop a generic library that could handle such requests. Eventually, I more or less abandoned the previous project to develop xdRequest. Here are some of the features, as highlighted on the project page:

  • Make cross-domain requests using the GET and POST methods
  • Add request headers along with the requests
  • Automatically collect cookies sent by the remote web server
  • Automatically follow redirects and collect cookies along the way
  • Properly store/transmit cookies, much like the popular server-side library, cURL
  • Automatically submits hidden fields detected within forms on webpages

I also put together a relatively simple xdRequest demo. It brings in data from espn.go.com. When the page loads, it makes an xdRequest to ESPN for the University of Florida Football schedule. I’m a die-hard Gator and UF alumnus, so that’s why I decided to start the demo with the UF football schedule. When you can click the hyperlinks in the demo, xdRequests are made to ESPN to load schedules for other teams, as well as the box scores for games. It’s a simple example, but the sky is the limit as to what you can do with xdRequest.

If you’re interested in learning more about the project, be sure to check out the xdRequest project page. If you’re a JavaScript geek and would like to join, just drop me a line and I’ll gladly add you to the project. I’m kind of a n00b when it comes to object-oriented programming with JavaScript, so I could definitely use some help to clean up the code, make enhancements, find bugs, and fix bugs. This is the first “official” Open Source project I’ve ever worked on, so I’m a n00b when it comes to working on OSS as well.

If you like xdRequest and build any cool web applications with it, let me know and I’ll link to it from the project page. If you have any feature suggestions or encounter any bugs, please report them in the xdRequest issue tracker.

Bookmark and Share

Related Posts


Related Websites

Tags: <, , , >

Comments

  • This project rocks! I would like to contribute, I can refactor the code and make
    some simplifications to use it in a more clear and practice way, like:
    var req = new xdRequest({
    method: "get",
    url: "http://www.remotesite.com",
    callback: function(data) {
    // handle data
    },
    headers: [
    {name: "headername1", value: "headervalue1"},
    {name: "headername2", value: "headervalue2"}
    ]
    });

    req.execute();

    ...let me know!
  • I'd be more than glad to add you to the project. Welcome aboard! :-D
  • Awesome job!

    About a year ago I tried to find a solution to do cross domain requests and was disappointed to find nothing worked.
    I understand the security concerns. In certain circumstances, it makes sense not to allow a clients browser to make requests outside the domain of whatever site they happen to be. I can see how a DoS attack could be accomplished to

    As a developer, not allowing cross domain scripting sucks. If I want to build a kick ass site that pulls data from multiple locations (many websites), I have to make my server(s) do all the work. At first this seems ok, but if I want that site to scale horizontally on a massive scale, I need lots of $$$ or a way to easily monetize the site (which isn't always obvious at first). Plus, those sites I use for data are eventually going to get pissed and ban the IP of my server(s).

    Massive low cost scaling can only be accomplished if the client is able to do some of the work. Plus, I don't have to worry about any of my servers getting blacklisted because all remote data request are handled by the client and appear as normal traffic.
    Your cross domain scripting solution is a major accomplishment, possibly an evolution of the web itself.

    I applaud and thank you for your work!
    You rock!

    Kevin
  • Thanks for the kind words. Please be aware that the possibility of getting
    blacklisted still exists. The requests are made via dynamically inserted
    scripts to the YQL servers. YQL servers make the requests to the remote
    pages and then send the data back to the client, so it is possible that some
    servers may blacklist the YQL servers. YQL also provides an opt-out option
    for content providers: http://developer.yahoo.com/yql/provider/
  • I see your point.
    Correct me if I'm wrong, but the requests made to YQL from the client, when visiting a site using your code.
    YQL servers are going to see the IP of someone visiting a site, not the IP of the site itself?

    I see your point on the opt out as well. All the more reason to skip Yahoo entirely.
    Do you think there is a way to make remote requests completely on the client side.
    If you could figure that out, say hello web 3.0!
  • The YQL servers will see the IP of the client. The remote site will see the
    IP of the YQL servers. YQL also does pass along some headers that include
    information about the client (for example, a header called X-Forwarded-For
    that contains the IP address of the client).

    Because of the security measures enforced by browsers, the only way you can
    make remote requests 100% within a client is with services that provide an
    API that provide JASONP responses. Otherwise, you have to use some sort of
    proxy that will fetch the data for you and provide a JSONP response, as YQL
    does. I think YQL is the best solution I've seen out there for doing this.
  • Hi,

    Excellent project. Been looking for something like this for a while. I am having problems using this in conjunction with other JS files though, namely MooTools and not getting any error output.

    I think it may be some variable conflict. More project specific var and func names may help this issue.
  • I found out why it doesn't work in conjunction with Mootools. There is a line:

    var natives = {'Array': Array, 'Date': Date, 'Function': Function, 'Number': Number, 'RegExp': RegExp, 'String': String};

    Which is on line 96 for me. For some reason xdRequest won't run unless you remove the ['Array': Array] part of that object. I'm not sure exactly why this is an issue though.
  • Hmm, that's odd. I'm not very familiar with Mootools. I'm not sure why
    xdRequest would cause Mootools to break.
  • It's a conflict with that in Mootools and the following in xdR:

    for(cookie in cookiejar) {
    if(cookiejar[cookie].match(url, this.secure())) {
    outputCookies.push(cookiejar[cookie]);
    }
    }

    Namely this bit:

    cookiejar[cookie].match(...

    It's saying that cookiejar[cookie] is an object when I guess it should be a string. However, if I bypass this, it just breaks.

    I think it could be to do with some Mootools meddligns with the natives of JS, probably the array since that is what is conflicting in Moo. However, I am very rusty with JS. Maybe its time to moove on :|
  • Hmm, I wonder if the version I have in the download section is an older
    (damaged version). That for loop doesn't look as though it has been fixed
    to work in Internet Exploder. Are you using IE? You may want to try
    downloading the latest source in the trunk here:
    http://xdrequest.googlecode.com/svn/trunk/xdReq...
  • Thanks. No change though. I think the issue is with MooTools. I believe they have modified the Array object and I think that is causing the issue.

    I have posted a detailed report in the MooTools forum. If someone can find out what's wrong I will let you know.
  • Paul
    Hi - awesome script! I have one question though that I'm not clear on. The rate limit per IP. let's say I put this script on my site, every time it is triggered in someone's browser, is that instance attributed to my site's IP or the users? Also - if this page is calling data from the same remote site (say CNN.com), will CNN.com be blocked to users of my site after 10000 page views? Of course only if this occurs in an hour. I'm just trying to figure out exactly what IP they are talking about, the user, the site that host's the .js file, or the site that is being "scraped"?

    I hope I am making sense. Thanks again!
  • The rate limit is on the client IP. Since everything takes place in the
    browser window (the browser makes the calls to YQL), it would be based on
    the browser's IP address.
  • Paul
    As an example, in your demo app you created - what would need to happen in an hour for it to exceed the limit. Maybe that makes more sense as to what I am trying to ask! Thanks again.
  • In my demo app, you would have to click through and load enough pages for
    your browser to trigger the limit.
blog comments powered by Disqus