Make Cross-Domain AJAX Requests with xdRequest

xdRequest is an open-source JavaScript library that I’ve developed for making cross-domain AJAX requests. In this day and age, with web browsers becoming very fast and powerful, it has become quite possible to build web applications with much of the work performed by the client. xdRequest makes it possible to easily pull data from other sources and manipulate it all on the client-side. With xdRequest, you will no longer need to perform data extractions/manipulations from other sources on a server. It can all happen right in the browser.

Screenshot of the YQL html data table I had recently been fooling around quite a bit with YQL, which does provide some built-in features to make cross-domain http requests. It can be done with the html open data table. You can try it out from the YQL Console.  Just go to the data tables section on the right side and select Data > html. Unfortunately, the built-in html table is somewhat limited.  You can only make requests using the GET method, and it does not provide any mechanism for sending/receiving HTTP headers.

Someone in the YQL forums posted a YQL feature request to add POST and other HTTP request methods to the html table. Someone responded with the suggestion of building an open data table to do just that. I then started reading the open data table documentation, and came to realize that what you can do with YQL and open data tables is nothing short of astounding. You can use server-side JavaScript to request data from multiple sources and manipulate it in any way you see fit.

I started developing my own custom open data table for the particular project I was working on at the time, and eventually the project morphed into xdRequest. Because I needed to track cookies and redirects in my little project, I realized that it would be great to develop a generic library that could handle such requests. Eventually, I more or less abandoned the previous project to develop xdRequest. Here are some of the features, as highlighted on the project page:

  • Make cross-domain requests using the GET and POST methods
  • Add request headers along with the requests
  • Automatically collect cookies sent by the remote web server
  • Automatically follow redirects and collect cookies along the way
  • Properly store/transmit cookies, much like the popular server-side library, cURL
  • Automatically submits hidden fields detected within forms on webpages

I also put together a relatively simple xdRequest demo. It brings in data from espn.go.com. When the page loads, it makes an xdRequest to ESPN for the University of Florida Football schedule. I’m a die-hard Gator and UF alumnus, so that’s why I decided to start the demo with the UF football schedule. When you can click the hyperlinks in the demo, xdRequests are made to ESPN to load schedules for other teams, as well as the box scores for games. It’s a simple example, but the sky is the limit as to what you can do with xdRequest.

If you’re interested in learning more about the project, be sure to check out the xdRequest project page. If you’re a JavaScript geek and would like to join, just drop me a line and I’ll gladly add you to the project. I’m kind of a n00b when it comes to object-oriented programming with JavaScript, so I could definitely use some help to clean up the code, make enhancements, find bugs, and fix bugs. This is the first “official” Open Source project I’ve ever worked on, so I’m a n00b when it comes to working on OSS as well.

If you like xdRequest and build any cool web applications with it, let me know and I’ll link to it from the project page. If you have any feature suggestions or encounter any bugs, please report them in the xdRequest issue tracker.

About GeekLad

Geeklad is a technology enthusiast and programming hobbyist. Occasionally he will put together useful little bits of code (be it JavaScript or PHP) and share them with the world. He also enjoys creating and sharing howtos, describing how to do the things people want to do with their computers.
Tagged , , , . Bookmark the permalink.
  • http://www.daveoncode.com/ Davide Zanotti

    This project rocks! I would like to contribute, I can refactor the code and make
    some simplifications to use it in a more clear and practice way, like:
    var req = new xdRequest({
    method: “get”,
    url: “http://www.remotesite.com”,
    callback: function(data) {
    // handle data
    },
    headers: [
    {name: "headername1", value: "headervalue1"},
    {name: "headername2", value: "headervalue2"}
    ]
    });

    req.execute();

    …let me know!

  • http://geeklad.com GeekLad

    I'd be more than glad to add you to the project. Welcome aboard! :-D

  • http://yodisoft.com/ Kevin Gigiano

    Awesome job!

    About a year ago I tried to find a solution to do cross domain requests and was disappointed to find nothing worked.
    I understand the security concerns. In certain circumstances, it makes sense not to allow a clients browser to make requests outside the domain of whatever site they happen to be. I can see how a DoS attack could be accomplished to

    As a developer, not allowing cross domain scripting sucks. If I want to build a kick ass site that pulls data from multiple locations (many websites), I have to make my server(s) do all the work. At first this seems ok, but if I want that site to scale horizontally on a massive scale, I need lots of $$$ or a way to easily monetize the site (which isn't always obvious at first). Plus, those sites I use for data are eventually going to get pissed and ban the IP of my server(s).

    Massive low cost scaling can only be accomplished if the client is able to do some of the work. Plus, I don't have to worry about any of my servers getting blacklisted because all remote data request are handled by the client and appear as normal traffic.
    Your cross domain scripting solution is a major accomplishment, possibly an evolution of the web itself.

    I applaud and thank you for your work!
    You rock!

    Kevin

  • http://geeklad.com GeekLad

    Thanks for the kind words. Please be aware that the possibility of getting
    blacklisted still exists. The requests are made via dynamically inserted
    scripts to the YQL servers. YQL servers make the requests to the remote
    pages and then send the data back to the client, so it is possible that some
    servers may blacklist the YQL servers. YQL also provides an opt-out option
    for content providers: http://developer.yahoo.com/yql/provider/

  • http://yodisoft.com/ Kevin Gigiano

    I see your point.
    Correct me if I'm wrong, but the requests made to YQL from the client, when visiting a site using your code.
    YQL servers are going to see the IP of someone visiting a site, not the IP of the site itself?

    I see your point on the opt out as well. All the more reason to skip Yahoo entirely.
    Do you think there is a way to make remote requests completely on the client side.
    If you could figure that out, say hello web 3.0!

  • http://geeklad.com GeekLad

    The YQL servers will see the IP of the client. The remote site will see the
    IP of the YQL servers. YQL also does pass along some headers that include
    information about the client (for example, a header called X-Forwarded-For
    that contains the IP address of the client).

    Because of the security measures enforced by browsers, the only way you can
    make remote requests 100% within a client is with services that provide an
    API that provide JASONP responses. Otherwise, you have to use some sort of
    proxy that will fetch the data for you and provide a JSONP response, as YQL
    does. I think YQL is the best solution I've seen out there for doing this.

  • http://semlabs.co.uk/ David Hopkins

    Hi,

    Excellent project. Been looking for something like this for a while. I am having problems using this in conjunction with other JS files though, namely MooTools and not getting any error output.

    I think it may be some variable conflict. More project specific var and func names may help this issue.

  • http://semlabs.co.uk/ David Hopkins

    I found out why it doesn't work in conjunction with Mootools. There is a line:

    var natives = {'Array': Array, 'Date': Date, 'Function': Function, 'Number': Number, 'RegExp': RegExp, 'String': String};

    Which is on line 96 for me. For some reason xdRequest won't run unless you remove the ['Array': Array] part of that object. I'm not sure exactly why this is an issue though.

  • http://geeklad.com GeekLad

    Hmm, that's odd. I'm not very familiar with Mootools. I'm not sure why
    xdRequest would cause Mootools to break.

  • http://semlabs.co.uk/ David Hopkins

    It's a conflict with that in Mootools and the following in xdR:

    for(cookie in cookiejar) {
    if(cookiejar[cookie].match(url, this.secure())) {
    outputCookies.push(cookiejar[cookie]);
    }
    }

    Namely this bit:

    cookiejar[cookie].match(…

    It's saying that cookiejar[cookie] is an object when I guess it should be a string. However, if I bypass this, it just breaks.

    I think it could be to do with some Mootools meddligns with the natives of JS, probably the array since that is what is conflicting in Moo. However, I am very rusty with JS. Maybe its time to moove on :|

  • http://geeklad.com GeekLad

    Hmm, I wonder if the version I have in the download section is an older
    (damaged version). That for loop doesn't look as though it has been fixed
    to work in Internet Exploder. Are you using IE? You may want to try
    downloading the latest source in the trunk here:
    http://xdrequest.googlecode.com/svn/trunk/xdReq

  • http://semlabs.co.uk/ David Hopkins

    Thanks. No change though. I think the issue is with MooTools. I believe they have modified the Array object and I think that is causing the issue.

    I have posted a detailed report in the MooTools forum. If someone can find out what's wrong I will let you know.

  • Paul

    Hi – awesome script! I have one question though that I'm not clear on. The rate limit per IP. let's say I put this script on my site, every time it is triggered in someone's browser, is that instance attributed to my site's IP or the users? Also – if this page is calling data from the same remote site (say CNN.com), will CNN.com be blocked to users of my site after 10000 page views? Of course only if this occurs in an hour. I'm just trying to figure out exactly what IP they are talking about, the user, the site that host's the .js file, or the site that is being “scraped”?

    I hope I am making sense. Thanks again!

  • Paul

    As an example, in your demo app you created – what would need to happen in an hour for it to exceed the limit. Maybe that makes more sense as to what I am trying to ask! Thanks again.

  • http://geeklad.com GeekLad

    The rate limit is on the client IP. Since everything takes place in the
    browser window (the browser makes the calls to YQL), it would be based on
    the browser's IP address.

  • http://geeklad.com GeekLad

    In my demo app, you would have to click through and load enough pages for
    your browser to trigger the limit.

  • Gibbs

    Hi – I have been using this script, and it all of a sudden stopped working. It also does not work on your demo page anymore. Do you have any idea why? It's a great script and I would like to continue to make use of it, but as I said it all of a sudden went dead.

  • http://geeklad.com GeekLad

    Hmm, I'm not sure what broke it. It seems there's still a YQL response with
    data coming back. I haven't seen any changes in the code since November.
    Very odd indeed.

  • angl

    Hi, yes, me too. The callback function is never call, do you know whats the cause? thnks in advance, is a great project!

  • http://geeklad.com GeekLad

    No, it will take some debugging to find the root of the problem. Time which
    unfortunately I do not have.

  • angl

    Hi again, debugging the library , I notice that attribute diagnostics no exists!
    //attribute response.query.diagnostics no exists ?¿
    //newresult.error = response.query.diagnostics.url[1]["http-status-message"];

    Comment this line , works fine! but maybe is important to know which is the responsibility of this line.
    Temporaly with this patch works fine! I hope this be useful for us.

  • http://geeklad.com GeekLad

    Excellent, thanks for the tip! Are you interested in joining the project?
    You could update the code in the repository if you're interested.

  • angl

    Hi again, debugging the library , I notice that attribute diagnostics no exists!
    //attribute response.query.diagnostics no exists ?¿
    //newresult.error = response.query.diagnostics.url[1]["http-status-message"];

    Comment this line , works fine! but maybe is important to know which is the responsibility of this line.
    Temporaly with this patch works fine! I hope this be useful for us.

  • http://geeklad.com GeekLad

    Excellent, thanks for the tip! Are you interested in joining the project?

    You could update the code in the repository if you're interested.

  • Abdiel Technologies

    Hound  is a site which search job for you accross the world …hound directly connect with the employers  it means you apply  directly to employers. hound does not allow any banner to promote there bussiness 
    it only provide what the employer wants it is basically a good site which give us a great utility.its a amazing site everyone should try this site … it help us to search job world wide n it is very convenient . 
    in this sit e you did not need to search more just type your qualification and in which city do u want to do job that’s it …….
    executive job