API Methods - Realtime Upload Feed

This is an in-testing feature! Service on this feature is not guaranteed and the specification may change without warning.

This is not an API method as such but allows external use of raw uploaded data by other applications.

Applications can make use of uploaded data which is transmitted in realtime via AMQP, a messaging and queueing system.

The distributed messages include all messages EVE Metrics receives and are not processed in any way before retransmission.

Usage

EVE Metrics operates an AMQP server at queue.eve-metrics.com using the default port of 5672.

To receives messages from the exchanges you should attach a queue to the desired exchange.

market_order_uploads - Market order data
market_history_uploads - Market price history data

Both these exchanges are fanout exchanges.

You should create your own queue(s) on the server for your application.

We recommend you instruct AMQP to delete your queue on consumer disconnect and to mark your queue as exclusive when creating it; this lowers demand on the EVE Metrics AMQP server.

Data Format and Content

All messages are YAML formatted. YAML parsers are available for a vast array of languages and the format allows low levels of overhead compared to others.

Each message includes the following fields:

Body - CSV formatted raw data
User ID - The unique ID of the user who uploaded this data
Developer Key ID - The unique ID of the key the upload program is authenticating itself with
Received At - The time this upload was received by EVE Metrics
Generated At - The time this uploaded data was generated, as reported by the upload client
Version - The version string of this upload client

Example Clients

The following client, written in Ruby, uses the AMQP gem and will simply spool out all received messages to standard system output.

require 'rubygems'
require 'amqp'
require 'mq'
AMQP.start(:host => 'queue.eve-metrics.com') do
  puts "Connected to AMQP server OK, listening for new messages"
  amq = MQ.new
  amq.queue('your_app_name_goes_here', :exclusive=>true, :auto_delete=>true).bind(amq.fanout('market_order_uploads')).subscribe do |msg|
    puts msg
  end
end

This client uses the Python library py-amqplib and again simply spools to stdout. Python users may also be interested in txAMQP which uses the Twisted framework for asynchronous I/O.

from amqplib import client_0_8 as amqp
conn = amqp.Connection(host="queue.eve-metrics.com:5672 ", userid="guest", password="guest", virtual_host="/", insist=False)
chan = conn.channel()
chan.queue_declare(queue="my_app_name", durable=False, exclusive=True, auto_delete=True)
chan.queue_bind(queue="my_app_name", exchange="market_order_uploads", routing_key="")
def recv_callback(msg):
  print msg.body
chan.basic_consume(queue="my_app_name", no_ack=True, callback=recv_callback, consumer_tag="testtag")
while True:
  chan.wait()
chan.basic_cancel("testtag")

Back to the land of Ruby, this is a slightly more complex example which spools all received messages down to text files to interface with existing raw log processor systems.

It is important to note the raw CSV data is from the _uploader_, not EVE. Upload clients should adhere to EVE standards and any additional columns should be added to the end of the row- EMU adds a column denoting the source as either log or cache, for example. In most implementations of CSV parsers these will not affect existing code.

require 'rubygems'
require 'yaml'
require 'amqp'
require 'mq'
Signal.trap('INT') { AMQP.stop{ EM.stop } }
Signal.trap('TERM'){ AMQP.stop{ EM.stop } }
AMQP.start(:host => 'queue.eve-metrics.com') do
  puts "Connected to AMQP server OK, listening for new messages"
  amq = MQ.new
  amq.queue('your_app_name_goes_here', :exclusive=>true, :auto_delete=>true).bind(amq.fanout('market_order_uploads')).subscribe do |msg|
    upload = YAML::load(msg)
    puts "Got new message uploaded at #{upload[:received_at]} by UID #{upload[:user_id]}/DKID #{upload[:developer_key_id]}"
    File.open("uploads/upload_#{upload[:received_at].to_i}.txt", "w"){|f| f << upload[:body] }
  end
end