Posterous theme by Cory Watilo

Cross domain API calls using Sinatra + pmxdr

You have an API server running and want to enable people to make cross domain XHR requests using javascript. How do you do that? There are bunch of methods that exist. I will present a technique to achieve this using HTML5's awesome postMessage API. I used pmxdr library, which is a pure javascript library that leverages HTML5's postMessage functionality without needing Flash or anything of that sort. Very neat!

Some setup details -

  1. My remote server is running on mylocal.name:9393 (mylocal.name -> localhosts in the /etc/hosts file). This server provides a POST operation on /api/token and returns a JSON object on a successful POST operation.
  2. My client is running on localhost:9494. It will make a cross domain request for the JSON token data from the remote server.
  3. Both my servers are powered by Sinatra.

Server settings

There are two things the server has to do before anything else.

  1. Have a valid route /pmxdr/api which loads the pmxdr-host.js file from the library. We need to add the approved domains to this file, but more on this later.
    get '/pmxdr/api' do
      p "got a get on pmxdr/api"
      erb :pmxdr
    end
    The view pmxdr.erb just loads the pmxdr-hosts.js file.
  2. The server must allow cross domain through X-Frame-Options header. For Sinatra, which automatically addes X-Frame-Options header, you can configure it by
  3. #x-frame-options allow sinatra
    configure do
      set :protection, :except => :frame_options
    end

These two things will get us started on the server side. The API token call is show here to give a context. 

post '/api/token' do
  p "post operation"
  content_type :json
    { :token => '9238928fkjkjkdjfser'}.to_json
end

In the pmxdr-hosts.js (which is rendered by pmxdr view in this example), allow localhost with any port to access its resources.

var alwaysTrustedOrigins = [ 

   /^[\w-]+:(?:\/\/)?(?:[\w\.-]+\.)?localhost\:\d+(?::\d+)?$/ // any origin on any protocol that has a domain that ends in localhost:anyport
  
    
  ];

If you don't do this, the logs in the server will show a POST operation, but pmxdr host will not return the data and give a disallowed-origin error.

{pmxdr: true, error: "DISALLOWED_ORIGIN", id:"123", status:200, statusText:"OK"}

Making API calls through Client

In this example, when you navigate to the client, an API call is made and the token is returned.

Load the pmxdr client code - <script src="pmxdr-client.js" type="text/javascript"></script>

The actual client side code is very simple.

var pmxdrInterface = new pmxdr("http://mylocal.name:9393/");
pmxdrInterface. function() {
    this.request([{
        method  : "post",
        uri     : "/api/token",
        data    : "",
        callback: handleResponse,
        headers : {},
        timeout : 30000 /* == 30 seconds*/
    }]);
 
}
function handleResponse(response) {
          //alert(response.data);
    // do stuff with response
                console.log(response);
                var code;
                codeContainer = document.createElement("pre");
                if ( !response.error ) {
         code += "\n\ndata:\n"+response.data;
       } else code += (errorIsGood? "(PASS) " : "(FAIL) ")+"Error: "+response.error;
       
       //codeContainer.appendChild(document.createTextNode(code));
       results.appendChild(document.createTextNode(code));
       //results.appendChild(document.createElement("hr"));
                //
    pmxdrInterface.unload();
}
pmxdrInterface.init();

This is what happens in the above code -

  1. When pmxdr is initialized, a temporary iFrame is setup, that has a source of http://mylocal.name:9393/pmxdr/api. Remember - When we were setting up the server, we made sure that the API end point was "gettable" and it loaded the host library code.
  2. After the iFrame is intialized, it sends a GET request with a uri parameter is http://mylocal.name:9393/api/token and a randomly generated id.
  3. After this request, the pmxdr host checks if the requesting window (in this case localhost:9494) is allowed to view the data. Remember - the entry we added to the hosts.js file.
  4. A temporary iFrame, which is setup gets deleted after a successful response or the timeout that is specified in the above code.

The great thing is that you can send any kind of custom headers that your API call should respond to (for authentication or anything like that).

You can download the entire code from my git repo https://github.com/saranyan/pmxdr-sinatra

 

Cloudfoundry, X.commerce, Ruby

I am right now at the awesome CloudFoundry open tour (Austin). I am hearing @ciberch talk about how one can use CloudFoundry to deploy ruby application, I decided to hack on something of my own. My first ruby/sinatra application using X.commerce platform and CloudFoundry.

In the last post, I posted an example in ruby about how two capabilities can compute a Fibonacci sequence between themselves. Recap - user initiates the computation by vising a url/ping, the capability computes the next number in the sequence and sends a message on /message/ping which gets handled by the other guy. He sends a message back with the next sequence on /message/pong. This goes on until 10 numbers in the sequence are computed after which the application terminates.

This is what I did to change my old code - The new code is found here

  1. Moved the client1 and client2 into different folders and created a Gemfile so that they can use Bundler instead.
  2. Changed my endpoints in X.commerce devportal (fibclient1-x.cloudfoundry.com, fibclient2-x.cloudfoundry.com)
  3. Pushed my apps to cloudfoundry. (Assuming you have setup your account already. more details here. create an account. Install the awesome vmc command line tool.)

There it is. My app works on CloudFoundry.

Fibonacci over Fabric

Today, I wrote the most complicated Fibonacci algorithm I have ever written. It reminds of the episode from "The Big Bang Theory", where Howard controls a light bulb in the room by routing the instructions over cables and satellites and back. (Check it out, very neat sequence http://www.youtube.com/watch?v=BW9FbjjkKo4). This is definitely not that complex and definitely has a strong use case in terms of conveying how to communicate with the X.commerce Fabric.

A refresher - An X.commerce capability is something that solves a commerce need. In simple terms, it can be visualized as an application that communicates with the X.commerce Fabric. While applications are generally limited to helping users perform some tasks, a capability can accomplish more in terms of helping businesses execute complex commerce operations. For instance, a capability might help a store front with order processing, inventory management, or even analytics. Unlike applications that usually communicate only with its end users, a capability can interact with other capabilities to accomplish its objectives.

In this example, we are going to rely on two capabilities to compute the Fibonacci sequence together. The computation begins with an intial sequence by GETting a URL /ping from capability 2. This in turn POSTs on /message/ping with a destination ID of the capability 1. Capability 1, which is already subscribed to the topic deserializes the message, computes the next number in the sequence and POSTs on /message/pong back on the Fabric with the destination ID as the original publisher ID. The pong action sends a message back on ping and this goes on. This example illustrates some great concepts about the X.commerce Fabric.

  • Communication is asynchronous. 
  • There should be a relationship (tenancy relationship) between a publisher and a listener. Though a listener is subscribed to receive on a topic, he will not get the message posted by the publisher on that topic unless the publisher is authorized to send messages to the listener.
  • If a capability wants the messages to be restricted/directed to one listener, they should specify the destination ID header (X-XC-DESTINATION-ID).
  • A receiving capability can respond just to the publishing capability by getting the destination ID from the incoming header X-XC-PUBLISHER (this will change in upcoming versions to X-XC-PUBLISHER-PSEUDONYM)

I created two capabilties in Ruby. The general flow and authorization follows the flow diagram provided below. Check out this link on how to build capabilities using the sandbox.

Fibonacci_flow

Here are some quick steps to get your capabilities configured correctly using the sandbox -

  • The sandbox end point is {https://api.sandbox.x.com/fabric}. 
  • Create a test merchant and make sure both your capabilities have authroized the same test merchant (this ensures that both the capabilities have a tenancy relationship with each other).
  • Use the tenant specific token as the authroization header when publishing to the fabric.
  • Use the fabric credentials to compare against the authorization header of an incoming message from the Fabric.

Fabric_snapshot

The code works as shown in the following diagram. 

Drawing1

With every ping pong message that gets sent to the Fabric, the respective capabilities add the next number of the Fibonacci sequence to the payload.

You can download the entire code for this sample from the X.commerce git repo.

 

 

A rails bundle for testing X.commerce capability

Using Rails 3.2 and Twitter bootstrap, I created a Ruby on Rails bundle that one can use to get started with X.commerce fabric. Using this bundle

  • you can send and receive messages to the Xfabric sandbox
  • you can create sample messages automatically for any valid schema hosted on the x.commerce cloud.

The entire code can be downloaded from github. The demo can be found at http://quiet-sword-2317.herokuapp.com/. Some steps you can follow to quickly get started using this bundle -

  • Clone the repository
  • Do a heroku create --stack cedar to create a heroku repo on the cedar stack
  • Go to https://devportal.x.com/ and register a new capability
  • Use the heroku application url as the end point
  • Register for an experimental topic /experimental/rails_bundle/test, which is the default topic that the message console uses. You can actually register any custom experimental topic.
  • Make sure your capability is subscribed to receive on that experimental topic you create.
  • Register a tenant and authorize your capability to send messages on behalf of that tenant. Copy the bearer token.
  • Now, you can use the bearer token and experimental topic in the message console to send and receive messages to the fabric. 

If you have any questions on using this, shoot a tweet @saranyan

magento extension to publish product data to x.commerce capability

In my continuing exploration of Magento, this is what I attempted - "Add a menu item in the admin dashboard, which when clicked, publishes the store product catalog to a X.commerce capability".

The goal was to understand what it takes to build this flow and roughly the things I learned from this exercise was the general structure of Magento folders to write custom extensions, and how to publish it to a X.commerce capability. As a refresher, X.commerce is the new open commerce platform pioneered by eBay. A capability is central to X.commerce technology, and in general, is anything technology or service that solves a commerce problem. A great overview is here

In a nutshell, a capability receives messages posted on a "topic" it is subscribed to. When it receives a message, it validates the schema using Avro contracts. I have written in detail about this in a previous blog post. In this blog post, we make use of all these numbing details, to publish a Magento catalog to a X.commerce topic (/cse/offers/create). The entire code can be downloaded from https://github.com/xcommerce/Sample-capabilities.

The first step was setting up the config files and folder structure (for extension). My folder structure looks like the following.

Screen_shot_2012-03-07_at_10

As you can see, the module name is Xcommerce_Cse. Under app/etc/modules I have created Xcommerce_All.xml file, which defines the Xcommerce module.

In Magento, config file plays a big part. The config files related to the extension we are defining contains all the details about blocks, helpers, events, etc. I created a sample config file like shown below, that can be extended to add more stuff -

The intention is to add a menu item it the dashboard. That part of the declaration can be seen in the config file under adminhtml/menu. When the user clicks this menu item, it should trigger an action that publishes the products from the catalog.

This triggers the code in Block/ListProducts.php

This code does the following things -

  1. Calls the helper function that encodes the list of products from the store in Avro binary.
  2. Posts to the X.commerce Fabric.

The helper function Helper/Data.php is shown below. Please include the avro php lib files into Magento's lib folder -

Screen_shot_2012-03-07_at_1

This publishes the product feed to the X.commerce fabric on the topic /cse/offers/create.

Screen_shot_2012-03-07_at_1

 

Using Ruby Instapaper Gem

https://github.com/spagalloco/instapaper is a decent instapaper gem for Ruby. The error handling can be better. If you are using this gem, this is what you need to do -

  1. From the README.md, it looks like, one needs to get an access token first through xAuth, by using Instapaper.access_token(username, password)
  2. Once you get the token, you should be able to configure the settings. 

 

Instapaper.configure do |config|
  config.consumer_key = YOUR_CONSUMER_KEY
  config.consumer_secret = YOUR_CONSUMER_SECRET
  config.oauth_token = YOUR_OAUTH_TOKEN
  config.oauth_token_secret = YOUR_OAUTH_TOKEN_SECRET
end

This does not work exactly this way, as instapaper expects a consumer key with the xAuth request. When I inspected the params, I saw that Consumer Key was not being sent. This makes sense, but the above code is slightly counter intuitive. You can get this working by setting a partial config before sending the access_token request.

Instapaper.configure do |config|
  config.consumer_key = YOUR_CONSUMER_KEY
  config.consumer_secret = YOUR_CONSUMER_SECRET
end

Now,  Instapaper.access_token(username, password) will work.

The challenge I was having was that, the error was not being caught properly in the gem, it was throwing a Hash argumenterror, which didn't make any sense, unless I started looking at the response body. I will probably update the gem and submit a pull request. But, it is an useful gem.

Magento Product Catalog data from GPS

Magento is interesting! As I dig deeper, I am more intrigued by its power, ambiguity and just the community around it. This is my first week into exploring Magento in depth and I already had to do some automation to get a fairly decent product catalog in place. 

I came across a blog post on creating products programmatically which was nice. I extended that script to pull data from Google shopping API and automatically populate my store running on localhost. 

Feel free to use the code. By the way, is there any thing that automatically sets my store with fake data? A plugin or something...? If there is an easy way, I want to hear about it.

Remeber to use your own Google API key and Magento installation locations before running the code.

mcrypt extension on snow leopard (php 5.3.8)

Modified from this article http://michaelgracie.com/2009/09/23/plugging-mcrypt-into-php-on-mac-os-x-snow...

  1. Download libmcrypt ( http://sourceforge.net/projects/mcrypt/files/)
  2. Compile mcrypt. Unpack the archive and execute  MACOSX_DEPLOYMENT_TARGET=10.6 CFLAGS='-O3 -fno-common -arch i386 -arch x86_64' LDFLAGS='-O3 -arch i386 -arch x86_64' CXXFLAGS='-O3 -fno-common -arch i386 -arch x86_64' ./configure --disable-dependency-tracking
  3. make -j6; sudo make install
  4. Download PHP 5.3.6 (This is not a typo. You need to download the source for 5.3.6) from http://us.php.net/get/php-5.3.6.tar.bz2/from/a/mirror
  5. goto ext/mcrypt directory and execute /usr/bin/phpize
  6. Execute  MACOSX_DEPLOYMENT_TARGET=10.6 CFLAGS='-O3 -fno-common -arch i386 -arch x86_64' LDFLAGS='-O3 -arch i386 -arch x86_64' CXXFLAGS='-O3 -fno-common -arch i386 -arch x86_64' ./configure --with-php-config=/Developer/SDKs/MacOSX10.6.sdk/usr/bin/php-config
  7. make -j6; sudo make install
  8. Add the extension to the php.ini file (typically located /etc/php.ini)
  9. Restart apache (sudo /usr/sbin/apachectl restart)

 

Sending and Receiving messages to the X.commerce Fabric

The post first appeared on x.com. https://www.x.com/developers/community/blogs/saranyan/sending-and-receiving-messages-fabric

Publishing/Sending

The X.commerce fabric expects capabilities to communicate via Avro binary messages. Avro is a data serialization system that relies on schemas. When Avro data is read, the schema used for writing should be available. It can be sometimes part of the message payload itself. However, in the case of X.commerce, the schema is not part of the message payload, but is made available through a reference URI. This is a conscious design decision that provides some significant advantages, few of which are listed below -

  • Better support for handling JSON and other formats that do not have an embedded schema.
  • Smaller message payloads and an ability to cache the schemas, thereby improving the performance.

As mentioned, an additional header is employed to provide a reference to the schema. There are two headers that we need to be aware of in this context. If a publishing capability is using contracts that are approved by the X.commerce team (https://github.com/xcommerce/X.commerce-Contracts), then they are publishing to known topics. In production environments, it is mandatory to publish only to approved topics. For instance, a capability might be using a marketplace contract and publishing to a topic like /marketplace/profile/delete. When such a known topic is used, the publishing capability needs to set a header called X-XC-SCHEMA-VERSION to the contract's version (eg., 1.0.0). This version number will be a value that corresponds to the schema definition found in the Open Commerce Language or X.commerce ontology.

The Fabric, in turn, will convert this and the topic information to a URI in the form of https://<host>/<namespace>/<topic>/<version> that uniquely identifies the that was used to create the message. The receiver of the message just has to look at the header called X-XC-SCHEMA-URI header to fetch the schema. 

In sandbox environment, the capabilities are allowed to experiment with custom topics and schemas. To send a message serialized through a custom schema, the capabilities can directly set the schema uri header. For instance, a capability can set the X-XC-SCHEMA-URI to a publicly hosted message contract and the Fabric will not overwrite it, but pass it on to the receiving capability. We need to remember that this is only allowed in development/sandbox mode.

Let us visit a simple example of a publishing capability that publishes a message to the Fabric. Let us use the marketplace contract and send a message to a known topic:  /marketplace/profile/delete. I will be showing both Ruby and PHP samples along the way. Let us assume that the capability has been registered with the sandbox and we have a bearer token for authorizing our messages. You can find all about registering capabilities here.

We post messages to the fabric url followed by topic name. The headers that are important to understand are

  • Content-Type: Set the content type to avro/binary, which is the type of message that the Fabric expects to communicate in.
  • Authorization: When a capability is registered and a tenant authorization is completed, a bearer token is generated. Use that bearer token. You are always communicating on behalf of a tenant. That tenant can sometimes be you, in that case, the SELF bearer token should be used. This is to prove that the publisher has a valid relationship with the tenant. This document talks in greater detail about capabilities and bearer tokens.  
  • X-XC-SCHEMA-VERSION: This is a numeric value that corresponds to the version of the contract being used

Before the message is posted, it needs to be encoded in Avro/binary using the schema

Error Message

When a capability is registered, it is automatically subscribed to the topic /message/failed. The capability cannot unsubscribe from this topic. The Fabric sends all message delivery errors back to a capability on this topic. Capabilities may not use this topic to deliver application-specific errors. However, if a capability wishes to send an error message to another capability, for instance, a bad field or invalid data, it needs to use X.commerce defined topics to communicate an error. In this example, if we send a message to delete a non-existent profile, the capability can publish back on a topic called /marketplace/profile/delete/failed which can used to indicate a business logic error.

Subscribing/Receiving

We saw a publishing capability send a message to the /marketplace/profile/delete topic, which was delivered to a marketplace capability, which was managing its profile. Logically, this capability needs to know someway if its request was successfully processed. It subscribes to a topic defined in the contract called /markeplace/profile/deleted and /marketplace/profile/delete/failed. When the marketplace capability, receives a message to delete the profile, assuming that the action was successful, it sends a message on /marketplace/profile/deleted. Now, the publisher capability from the previous example sees the following headers -

Using the schema URI, this message can be decoded from Avro binary. Sample code from PHP and Ruby is included -

Using Ruby FFI

This week, I had to work with a core C library and there was no reason for me to port it to Ruby. I came across the wonderful FFI gem/library. With FFI, calling external libraries is very easy. I think one of the real powers of FFI is to leverage native C libraries. However, it is also possible to port custom libraries or projects. There is a little bit of work to write the wrappers, which can be slightly tedius depending upon the size of the library/number of functions that we want to use, but still worth the time. There are few good documentation resources on the wiki page that are helpful - ffi examples was especially useful to get started fast.

These are the steps -

1. Create a shared library object for the C code that you want to port. On a unix based system the following commands will help. You might have to link additional libraries based on the code dependencies.

gcc -fPIC -c mylibrary.c
gcc -shared -o mylibrary.so mylibrary.o

2. Create a ruby wrapper based on your h file.

3. Start using your function calls. 

I have come up with a few use cases that covers some of the more used function call types. Let us see some test functions. For the lack of better name, mylibrary.h is defined as -

I have created four test functions, which show different argument flavors. Using the above commands, I have compiled my shared library object into say, a file called mylibrary.so. 

The ruby wrapper follows the structure outlined below -

module MyLibrary
  extend FFI::Library
  attach_function ...
end

In simple terms, attach_function call is used to redefine the function to enable its use in Ruby land. For the above code, my wrapper file looks as below -

The attach_function follows the pattern of name, input arg types, output arg type. It seems straight forward. (char *) resolves to string, (double *) is a pointer, (int) is an int...etc. You can find a list of types from the git repo of FFI. (https://github.com/ffi/ffi/blob/master/lib/ffi/types.rb)

Let us look at the sample code for calling these functions in both C and Ruby.

In C, using these functions are straight forward. In Ruby, most of them are straight forward except the last function test_function_4. This function takes an array and assigns the first few elements to some values. In Ruby, this array has to be declared FFI::MemoryPointer, which takes three arguments -

dd = FFI::MemoryPointer.new(:double,4, true)

I can pass this object to my function that expected the array. But, one thing that stumped me was that I was trying to access the values using get_double method (eg. dd.get_double(0), dd.get_double(1)...) it was not right. I found out that the right way to access this is using the get_array_of_double method.

The entire code can be downloaded from this git repo.