The post first appeared on x.com. https://www.x.com/developers/community/blogs/saranyan/sending-and-receiving-messages-fabric

Publishing/Sending

The X.commerce fabric expects capabilities to communicate via Avro binary messages. Avro is a data serialization system that relies on schemas. When Avro data is read, the schema used for writing should be available. It can be sometimes part of the message payload itself. However, in the case of X.commerce, the schema is not part of the message payload, but is made available through a reference URI. This is a conscious design decision that provides some significant advantages, few of which are listed below -

• Better support for handling JSON and other formats that do not have an embedded schema.
• Smaller message payloads and an ability to cache the schemas, thereby improving the performance.

As mentioned, an additional header is employed to provide a reference to the schema. There are two headers that we need to be aware of in this context. If a publishing capability is using contracts that are approved by the X.commerce team (https://github.com/xcommerce/X.commerce-Contracts), then they are publishing to known topics. In production environments, it is mandatory to publish only to approved topics. For instance, a capability might be using a marketplace contract and publishing to a topic like /marketplace/profile/delete. When such a known topic is used, the publishing capability needs to set a header called X-XC-SCHEMA-VERSION to the contract's version (eg., 1.0.0). This version number will be a value that corresponds to the schema definition found in the Open Commerce Language or X.commerce ontology.

The Fabric, in turn, will convert this and the topic information to a URI in the form of https://<host>/<namespace>/<topic>/<version> that uniquely identifies the that was used to create the message. The receiver of the message just has to look at the header called X-XC-SCHEMA-URI header to fetch the schema. 

In sandbox environment, the capabilities are allowed to experiment with custom topics and schemas. To send a message serialized through a custom schema, the capabilities can directly set the schema uri header. For instance, a capability can set the X-XC-SCHEMA-URI to a publicly hosted message contract and the Fabric will not overwrite it, but pass it on to the receiving capability. We need to remember that this is only allowed in development/sandbox mode.

Let us visit a simple example of a publishing capability that publishes a message to the Fabric. Let us use the marketplace contract and send a message to a known topic:  /marketplace/profile/delete. I will be showing both Ruby and PHP samples along the way. Let us assume that the capability has been registered with the sandbox and we have a bearer token for authorizing our messages. You can find all about registering capabilities here.

We post messages to the fabric url followed by topic name. The headers that are important to understand are

• Content-Type: Set the content type to avro/binary, which is the type of message that the Fabric expects to communicate in.
• Authorization: When a capability is registered and a tenant authorization is completed, a bearer token is generated. Use that bearer token. You are always communicating on behalf of a tenant. That tenant can sometimes be you, in that case, the SELF bearer token should be used. This is to prove that the publisher has a valid relationship with the tenant. This document talks in greater detail about capabilities and bearer tokens.  
• X-XC-SCHEMA-VERSION: This is a numeric value that corresponds to the version of the contract being used

Before the message is posted, it needs to be encoded in Avro/binary using the schema

Error Message

When a capability is registered, it is automatically subscribed to the topic /message/failed. The capability cannot unsubscribe from this topic. The Fabric sends all message delivery errors back to a capability on this topic. Capabilities may not use this topic to deliver application-specific errors. However, if a capability wishes to send an error message to another capability, for instance, a bad field or invalid data, it needs to use X.commerce defined topics to communicate an error. In this example, if we send a message to delete a non-existent profile, the capability can publish back on a topic called /marketplace/profile/delete/failed which can used to indicate a business logic error.

Subscribing/Receiving

We saw a publishing capability send a message to the /marketplace/profile/delete topic, which was delivered to a marketplace capability, which was managing its profile. Logically, this capability needs to know someway if its request was successfully processed. It subscribes to a topic defined in the contract called /markeplace/profile/deleted and /marketplace/profile/delete/failed. When the marketplace capability, receives a message to delete the profile, assuming that the action was successful, it sends a message on /marketplace/profile/deleted. Now, the publisher capability from the previous example sees the following headers -

Using the schema URI, this message can be decoded from Avro binary. Sample code from PHP and Ruby is included -

This week, I had to work with a core C library and there was no reason for me to port it to Ruby. I came across the wonderful FFI gem/library. With FFI, calling external libraries is very easy. I think one of the real powers of FFI is to leverage native C libraries. However, it is also possible to port custom libraries or projects. There is a little bit of work to write the wrappers, which can be slightly tedius depending upon the size of the library/number of functions that we want to use, but still worth the time. There are few good documentation resources on the wiki page that are helpful - ffi examples was especially useful to get started fast.

These are the steps -

1. Create a shared library object for the C code that you want to port. On a unix based system the following commands will help. You might have to link additional libraries based on the code dependencies.

gcc -fPIC -c mylibrary.c
gcc -shared -o mylibrary.so mylibrary.o

2. Create a ruby wrapper based on your h file.

3. Start using your function calls. 

I have come up with a few use cases that covers some of the more used function call types. Let us see some test functions. For the lack of better name, mylibrary.h is defined as -

I have created four test functions, which show different argument flavors. Using the above commands, I have compiled my shared library object into say, a file called mylibrary.so. 

The ruby wrapper follows the structure outlined below -

module MyLibrary
  extend FFI::Library
  attach_function ...
end

In simple terms, attach_function call is used to redefine the function to enable its use in Ruby land. For the above code, my wrapper file looks as below -

The attach_function follows the pattern of name, input arg types, output arg type. It seems straight forward. (char *) resolves to string, (double *) is a pointer, (int) is an int...etc. You can find a list of types from the git repo of FFI. (https://github.com/ffi/ffi/blob/master/lib/ffi/types.rb)

Let us look at the sample code for calling these functions in both C and Ruby.

In C, using these functions are straight forward. In Ruby, most of them are straight forward except the last function test_function_4. This function takes an array and assigns the first few elements to some values. In Ruby, this array has to be declared FFI::MemoryPointer, which takes three arguments -

dd = FFI::MemoryPointer.new(:double,4, true)

I can pass this object to my function that expected the array. But, one thing that stumped me was that I was trying to access the values using get_double method (eg. dd.get_double(0), dd.get_double(1)...) it was not right. I found out that the right way to access this is using the get_array_of_double method.

The entire code can be downloaded from this git repo.

Today, I had to read a binary file and transmit it over HTTP post. Here is what I did, which worked perfectly on my localhost (?) but had wierdness all over it on Heroku/Production.

contents = open("#{Rails.root}/tmp/#{fname}.data", "rb") {|io| io.read }
HTTParty.post("#{params["url"]}", {:body => contents, :headers => {'Content-Type' => 'avro/binary',  'Authorization' => params["token"]}})

On the other end, I was dumping this binary data into a file for post processing. In development, this worked fine. However, in production, I was getting a wierd error when I tried to read the data from the binary file. The error was something along the lines of -

ArgumentError ("negative length" -

After much digging, I traced it down to an encoding issue because the data was getting modified someway during the POST operation. The right way to do this is to encode the data using Base64.

contents = open("#{Rails.root}/tmp/#{fname}.data", "rb") {|io| Base64.encode64(io.read) }
HTTParty.post("#{params["url"]}", {:body => contents, :headers => {'Content-Type' => 'avro/binary',  'Authorization' => params["token"]}})

This preserves the data. This makes sense. I still am not sure, why it was working on my localhost. 

Avro is a data serialization system that is impressive in terms of the data structures it provides. Avro relies on schemas. An Avro data includes schemas during writing and the same schema is always available when reading or de-serializing this data. This makes it a really cool feature. This means that the serialized data is completely described as it includes its schema. In addition to this, the serialization can be very fast. There are few good examples and discussions on using Avro with Ruby. Check it out.

X.commerce uses Avro for defining message contracts, which makes it possible to describe and validate messages easily. However, the message contracts are defined as an Avro protocol and directly as a schema. Avro supports RPC, where both client and server exchange schemas with a handshaking protocol. But, we don't want to do that. We want to parse the X.commerce contracts as Avro schemas.

I was dabbling with the Avro ruby gem (https://github.com/apache/avro/tree/trunk/lang/ruby) to understand how it operates and how I can directly use that gem to serialize/de-serialize messages. I started with a sample schema -

SCHEMA = <<-JSON
[{ "type": "record",
  "name": "Product",
  "fields" : [
    {"name": "id", "type": "string"},
    {"name": "product_url", "type": "string"},
    {"name": "product_purchased", "type": "boolean", "default": "false"}
  ]},
  { "type": "record",
    "name": "Review",
    "fields" : [
      {"name": "id", "type": "string"},
      {"name": "review_url", "type": "string"},
      {"name": "review_verified", "type": "boolean", "default": "false"}
    ]}
JSON

To serialize this schema, we can do the following -

file = File.open('data.avr', 'wb')
schema = Avro::Schema.parse(SCHEMA)
writer = Avro::IO::DatumWriter.new(schema)
encoder = Avro::IO::BinaryEncoder.new(writer)


dw = Avro::DataFile::Writer.new(file, writer, schema)
dw << {"id" => "product123", "product_url" => "http://ebay.com/some_product", "product_purchased" => true}
dw << {"id" => "review123", "review_url" => "http://yelp.com/some_review", "review_verified" => false}
dw.close

If you look at the schema, I have intentionally used "id" as the name for both the records (Product and Review). This is to illustrate what I think is a bad practice. While, the fields are relative to the particular record, it might be better to have a proper identifier as this helps us easily correlate the data during de-serialization, which we might do the following way.

file = File.open('data.avr', 'r')
reader = Avro::IO::DatumReader.new(nil, Avro::Schema.parse(SCHEMA))
dr = Avro::DataFile::Reader.new(file, reader)
dr.each { |record| p record }

The output is -

$output -> 
 {"id"=>"product123", "product_url"=>"http://ebay.com/some_product", "product_purchased"=>true}
{"id"=>"review123", "review_url"=>"http://yelp.com/some_review", "review_verified"=>false}

If you try to serialize a data using improper schema, it gets flagged immediately. This is the error dump. A better way to display is to catch it and throw an error. Again, this is more for ilustrative purposes -

Avro::IO::AvroTypeError (The datum {"id"=>"foo", "value"=>"1001010"} is not an example of schema'

With X.commerce contracts, read them as schemas and not as RPC protocols. I will post a rails working example of a message console, that sends/receives an Avro message using the above approach.

require 'ostruct'

_user = {:first_name => "saran", :last_name => "vigraham"}
@user = OpenStruct.new _user

Now, you can access the @user hash by dot noation. @user.first_name, @user.last_name.

I was playing around with coffeescript, a language that compiles into JavaScript. Here is a code for getting Jquery Range Sliders to work in Coffee Script (slider-range is the div ID of my slider element) -

$ ->
        $("#slider-range").slider({
                range:true,
                min: 0,
                max: 100,
                values: [0, 20],
                slide: (event, ui) ->
                    #function here
        })

Instant Payment Notification (IPN) is PayPal's message service that sends a notification when a transaction takes place. When IPN is enabled in a seller's account,  the sellergetThere are bunch of tutorials online using Active Merchant gem for implementing IPN. However, most of these tutorials need some modification to be up and running. I created an IPN example using Rails 3.1, hosted on Heroku. Feel free to download the code and integrate it in your rails applications.

This post first appeared on x.com (https://www.x.com/developers/community/blogs/saranyan/hello-x.world)

To me, as a developer, "Hello world" is significant. It announces to the world the birth of a new developer, who is going to change the world with code. Code is a light saber that only a developer Jedi can wield. And "Hello world" is the baptism of a person with ideas to change the world to a responsible developer, who WILL change the world. After the magical "Hello world" appears on a screen and reflects the joy in the eyes of the developer, the real deal begins. It is just not baptism, it is baptism of fire. Questions start pounding the head - What is next after "hello world"? How do I build this now? This is too tough, is it worth it? The developer who successfully navigates this overwhelming technology and programming maze becomes the true master of "Hello world". He/She becomes part of the unique community of programming Jedi(s) who can build anything they want and in doing so, become the torch bearers and problem solvers.

I have been a developer all my life. From the time I got my PhD in Computer Science to dabbling with multiple startup ideas, I have worked with new technology and code all the time, and that means, several hello worlds. I am particularly thrilled and excited to be part of this awesome open commerce technology, which I believe is going to be a great leveler. There has never been a true open commerce platform that connects developers and merchants with each other and themselves. In addition to the competitive advantage this offers to merchants and developers, x.commerce introduces a "collaborative advantage". Now, merchants will have access to the best capabilities that will help them run their business better. Developers will have access to real people and real problems for which they can build solutions.

I met Nikelii during the x.commerce innovate developer's camp. He is an awesome person to talk to. I spent a long time chatting with him. Nikelii wants to build the next generation stock photography website. His idea is amazing. As a fitness instructor, he has attended camps and events where his astute photography skills are handy. People want to buy his photos, and he knows what is broken in the current stock photography market. For him, x.commerce is a powerful idea. He can get a store up and running without a lot of programming expertise. But, that does not satisfy him. He wants to build the "real thing". He wants to write capabilities that could completely transform the stock photography industry.  He is not a developer, he wants to be one. Because, Nikelii realizes that being a developer means being powerful enough to write the code that will change the world.

Dan is a great developer from San Francisco. He loves dabbling with new technology. He has written several iPhone and Android apps. He has been a programmer from the time he can remember. Dan was super excited about x.commerce. To him, this is like a commerce app store. He is looking forward to build some neat applications. He is a warrior who is looking to master yet another "hello world".

I believe that people Dan and Nikelii should connect and come together. It was not possible before. It is possible now with x.commerce, the world's first open commerce platform. I am a developer evangelist for x.commerce and I love how code and people can come together to build things that change the world in powerful ways. Keep watching this space for my thoughts on mastering the "hello world". If there is anything I can do to help you build cool things, do not hesitate to connect. My twitter handle is @saranyan. You can also write to me at saranyan@x.com

If you are seeing an error like this on Mac OS X lion -

===

dyld: Library not loaded: /System/Library/Frameworks/OpenCL.framework/Versions/A/Libraries/libclparser.dylib

  Referenced from: /opt/local/bin/convert

  Reason: image not found

====

You are missing this file "libclparser.dylib". You need to copy this file into this folder to fix the error. Thanks - http://apple.stackexchange.com/questions/21562/installing-imagemagick-leads-to-weird-error-involving-opencl