The scenario for this example is pretty simple – let’s say you want to trace data on one machine and display it on another. Still the data should be up to date – so basically whenever a DTrace probe fires the data should be transmitted to the other hosts. This can be done with AMQP.

The examples here assume you have a RabbitMQ server running and have pika installed.

Within this example two Python scripts are involved. One for sending data (send.py) and one for receiving (receive.py). The send.py script will launch the DTrace consumer and gather data for 5 seconds:

thr = dtrace.DTraceConsumerThread(SCRIPT, walk_func=walk)
thr.start()
time.sleep(5)
thr.stop()
thr.join()

The DTrace consumer is given an callback function which will be called whenever the DTrace probes fire. Within this callback we are going to create a message ad pass it on to the AMQP broker:

def walk(id, key, value):
    channel.basic_publish(exchange='', routing_key='dtrace', body=str({key[0]: value}))

The channel has previously been initialized (See this tutorial on more details). Now AMQP messages are passed around with up-to-date information from DTrace. All that there is left is implementing a ‘receiver’ in receive.py. This is again straight forward and also works using a callback function:

def callback(ch, method, properties, body):
    print 'Received: ', data

if __name__ == '__main__':
    channel.basic_consume(callback, queue='dtrace', no_ack=True)
    try:
        channel.start_consuming()
    except KeyboardInterrupt:
        connection.close()

Start the receive.py Python script first. Than start the send.py Python script. You can even start multiple send.py scripts on multiple hosts to get an overall view of the system calls made by processes on all machines

The send.py script counts system calls and will send AMQP messages as new data arrives. You will see in the output of the receive.py script that data arrives almost instantly:

$ ./receive.py
Received:  python 264
Received:  wnck-applet 5
Received:  metacity 6
Received:  gnome-panel 15
[...]

Now you can build life updating visualizations of the data gathered by DTrace.

Now I just needed to write about 50 LOC to do something nice Those ~50 lines are the implementation of an WSGI app using Mako as a template engine. Embedded in the Mako templates are Google Charts. And those charts actually show information coming out of the Python consumer. Now all what is left, is to point my browser to my SmartOS machine and get up-to-date graphs! For example a piechart which shows system calls by process:

Click to enlarge

Or using quantize I can browse a nice read size distribution – aka: how much bytes do my processes usual read?:

Click to enlarge

With all this it is also possible to plot graphs on the latency of node.js apps :

Click to enlarge

Again documentation on writing DTrace consumers is almost non-existent. But with some ‘inspiration’ from Bryan Cantrill and the original C based consumer I was able to get it work.

I did not have much spare time lately but I got the a chance last night to test my Python based DTrace consumer on SmartOS, Solaris 11 and OpenIndiana – and can confirm that it runs on all 3.

To get it up and running on SmartOS you will first need to install some dependencies. Use the 3rd party repositories as described in the SmartOS wiki to get pkg up and running:

pkg install git gcc-44 setuptools-26 gnu-binutils

When that is done we will clone the consumer code and install cython (you could however also use ctypes) using pip:

easy_install pip
pip install cython
git clone git://github.com/tmetsch/python-dtrace.git
cd python-dtrace/
python setup.py install

Now since this is done we can do the obligatory ‘Hello World’ to get things going:

Python DTrace consumer on SmartOS (Click to enlarge)

For more examples refer to the examples folder within the python-dtrace repository.

To ensure the proper startup of nova-compute we will register it using SMF – this will also increase the dependability of the services. To start we will define a properties file for nova-compute – we will be using a glance and rabbitMQ host which run on a different host:

--connection_type=fake
--glance_api_servers=192.168.56.101:9292
--rabbit_host=192.168.56.101
--rabbit_password=foobar
--sql_connection=mysql://root:foobar@192.168.56.101/nova

We will store this contents in the file /data/workspace/smartos.cfg. We will also create an XML file with the manifest definition – it has some dependencies defined:

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='manifest' name='openstack'>
  <service name='openstack/nova/compute' type='service' version='0'>
    <create_default_instance enabled='true'/>
    <single_instance/>
    <dependency name='fs' grouping='require_all' restart_on='none' type='service'>
      <service_fmri value='svc:/system/filesystem/local'/>
    </dependency>
    <dependency name='net' grouping='require_all' restart_on='none' type='service'>
      <service_fmri value='svc:/network/physical:default'/>
    </dependency>
    <dependency name='zones' grouping='require_all' restart_on='none' type='service'>
      <service_fmri value='svc:/system/zones:default'/>
    </dependency>
    <exec_method name='start' type='method' exec='/usr/bin/nohup /data/workspace/nova/bin/nova-compute --flagfile=/data/workspace/smartos.cfg &amp;' timeout_seconds='60'>
      <method_context>
        <method_environment>
          <envvar name="PATH" value="/ec/bin/:$PATH"/>
        </method_environment>
      </method_context>
    </exec_method>
    <exec_method name='stop' type='method' exec=':kill' timeout_seconds='60'>
      <method_context/>
    </exec_method>
   <stability value='Unstable' />
  </service>
</service_bundle>

This can be imported using the svccfg command. After doing so we can use al the known commands to verify all is up and running:

[root@08-00-27-e3-a9-19 /data/workspace]# svcs -p openstack/nova/compute
STATE          STIME    FMRI
online         14:04:52 svc:/openstack/nova/compute:default
               14:04:51     3142 python

Now up to integrate vmadm…

After setting up nova on SmartOS you should be able to launch the nova-compute service – Andy actually managed to get the nova-compute instance up and running:

./bin/nova-compute --connection_type=fake --sql_connection=mysql://root:admin@192.168.0.189/nova --fake_rabbit
2012-02-15 10:26:32,205 WARNING nova.virt.libvirt.firewall [-] Libvirt module could not be loaded. NWFilterFirewall will not work correctly.
2012-02-15 10:26:32,374 AUDIT nova.service [-] Starting compute node (version 2012.1-LOCALBRANCH:LOCALREVISION)
2012-02-15 10:26:33,099 INFO nova.rpc [-] Connected to AMQP server on localhost:5672

Note that for now we will use a fake RabbitMQ connection. The connection to the MySQL server however is mandatory.

This is the first step in plan to get OpenStack running on SmartOS. Next step will be to look into integration vmadm as a driver in OpenStack. The overall Ideas we have can be found in this slideset on google docs.

This is the first post of hopefully a series of post about getting OpenStack up and running on SmartOS. There is a blueprint for this effort. For this first post we will focus on setting up SmartOS and try to get some needed packages up and running.

I used the the following SmartOS iso image which will you – when booted – guide you through a small setup routine: smartos-20120210T043623Z.iso. After this you need to configure the pkg repositories as described in this tutorial. I encourage you all to have two discs attached – One for the zones – and one for the data. So since the setup routine created the zones pool I will first create a data pool on the second disc:

zpool create data c0t2d0
zfs create data/ec
zfs set mountpoint=/ec data/ec
zfs mount data/ec
cd / && wget -O- -q http://svn.everycity.co.uk/public/solaris/misc/ \
                    pkg5-smartos-bootstrap-20111221.tar.gz | gtar -zxf-

One major dependency is python, gcc and some tools and libs – and we will also need git so we can get the source code of nova later on:

/ec/bin/pkg install pkg:/database/mysql-55/client@5.5.16-0.162 \
                    pkg:/library/python/mysqldb@1.2.3-0.162
/ec/bin/pkg install python26 git setuptools-26 gcc-44 libxslt gnu-binutils

Next step will be to create a workspace file-system using ZFS (Optional) – this will allow me to do rollbacks later on during the porting of OpenStack:

zfs create data/workspace
cd /data/workspace/

Let’s clone nova and install pip:

export PATH=/ec/bin:$PATH
easy_install pip
git clone git://github.com/tmetsch/nova.git
cd nova/tools
export CFLAGS="-D_XPG6 -std=c99"
pip install -r pip-requires

Now let’s see if we can get the test up and running:

cd ..
./run_tests.sh

This will do for the first steps – I’ll be going to get the dependencies of nova up and running next – Will than post the next blog post.

Percent done: 1%

So I developed a very minimalistic application which emulates a RESTful interface. Of course it takes the simplistic – all time favorite – Hello World approach for this. Client can create resources, which when queried will return the obligatory ‘Hello <resource name>’.

To start create a new java application in your OpenShift dashboard. When cloning the git repository – which was created – you will notice that you basically get a maven project with some templates in it. Now you can simple use maven to test and develop your application. When done do a git push and your application is ready to go.

The implementation is pretty straight forward:

/**
 * Simple REST Hello World Service.
 *
 * @author tmetsch
 *
 */
public class HelloServlet extends HttpServlet {
    [...]
    @Override
    protected final void doGet(final HttpServletRequest req,
            final HttpServletResponse resp) throws ServletException,
            IOException {
            [...]
    }
    [...]

So beside from extending the HttpServlet class all you need to do is implement the doGet and doPost methods. When done edit the ‘web.xml’ file in the folder src/main/webapp:

[...]
<servlet>
    <servlet-name>HelloWorld</servlet-name>
    <servlet-class>the.ultimate.test.pkg.HelloServlet</servlet-class>
</servlet>
<servlet-mapping>
    <servlet-name>HelloWorld</servlet-name>
    <url-pattern>/users/*</url-pattern>
</servlet-mapping>
[...]

This will make sure you the application can be found under the right route. If you like you can also add an index.html file in the same folder so people can read a brief introduction when visting the top layer of you application. In my case that would be http://testus-sample.rhcloud.com – The application itself can be found under: http://testus-sample.rhcloud.com/users/.

So I like the approach OpenShift takes here with maven. You can simply add you dependencies like jmock, junit etc. and during deployment it is taken care of that everything falls in place. Also if you write Unittest it’ll also ensure that you’re application will work – Non functional apps will not be deployed obviously if you write a Unittest and use maven. It’s pretty easy to write Unittest for the HttpServlet class if you use a mocking framework like jmock. You can get the source code at github.

As a first exercise I wanted to check that I was able to get a simple Task board up and running. A good tutorial to create such a simple application with Pyramid can be found here.

After signing into OpenShift you can create a new application in the dashboard. Make sure that before doing so you have your namespace setup and your public ssh key configured in OpenShift. After creating the application a git repository is created for you. Simple clone this so you have a local working copy on your machine.

Now have a look inside you local working copy. You will find the directories data, libs, wsgi and some files like a README and the python setuptools related setup.py. Let’s start with editing the setup.py file:

from setuptools import setup

setup(name='A simple TODO Application to test OpenShift Features.',
      version='1.0',
      description='An OpenShift App',
      author='Thijs Metsch',
      author_email='thijs.metsch@opensolaris.org',
      install_requires=['pyramid'],
      classifiers=[
                     'Operating System :: OS Independent',
                     'Programming Language :: Python',
                    ],
     )

Please note the install_required parameter. Using this parameter will ensure that during deployment of you app the necessary requirements get installed – in this case Pyramid.

The following steps are pretty straight forward. The file application in the wsgi directory is as the name states a simple WSGI app. What we need to do is edit this file and let it call the todo app:

#!/usr/bin/python

import os

virtenv = os.environ['APPDIR'] + '/virtenv/'
os.environ['PYTHON_EGG_CACHE'] = os.path.join(virtenv, 'lib/python2.6/site-packages')
virtualenv = os.path.join(virtenv, 'bin/activate_this.py')
try:
    execfile(virtualenv, dict(__file__=virtualenv))
except IOError:
    pass

from todo import application

Please note that the file has no ‘.py’ extension.

Now we are ready to create the finally application. Please have a look at this tutorial as the code will almost be identical.

Create a file todo.py and follow the tutorial as described. Make sure you application runs locally. Put the ‘*.mako’ files in the subdirectory static under the wsgi directory. You can place the schema.sql file in the data directory. We will be using a sqlite database which will also be place in there.

The final step is to alter the todo.py file and remove the __main__ part. Instead we will introduce the following application function:

here = os.environ['OPENSHIFT_APP_DIR']

def application(environ, start_response):
    '''
    A WSGI application.

    Configure pyramid, add routes and finally serve the app.

    environ -- the environment.
    start_response -- the response object.
    '''
    settings = {}
    settings['reload_all'] = True
    settings['debug_all'] = True
    settings['mako.directories'] = os.path.join(here, 'static')
    settings['db'] = os.path.join(here, '..', 'data', 'tasks.db')

    session_factory = UnencryptedCookieSessionFactoryConfig('itsaseekreet')

    config = Configurator(settings=settings, session_factory=session_factory)

    config.add_route('list', '/')
    config.add_route('new', '/new')
    config.add_route('close', '/close/{id}')

    config.add_static_view('static', os.path.join(here, 'static'))

    config.scan()

    return config.make_wsgi_app()(environ, start_response)

When done we can push the code to the repository. When done you can visit you application at the given URL.

With this I’ve been able to create the app http://todo-sample.rhcloud.com/ and I also deployed my OCCI example service: http://occi-sample.rhcloud.com/. As you can see pretty straight forward & fast and of course fun!

What would be fun is to create a simple unittesting framework for OpenShift. For now you can find the code pieces at github.

Based on the information you get from DTrace (using the Python consumer) you can draw life updating callgraphs of what is currently happening in the program. Not only is it possible to look at the callgraph but you can also look at the time it took to reach a certain piece of code to analyze bottle necks and the flow of the program:

$ pgrep python # get the pid of the process you want to trace
123456
$ ./callgraph.py 123456 # trace the program and create a callgraph

So if you would have the following Python code:

class A(object):

    def sayHello(self):
        return 'I am A'

class B(object):

    def __init__(self):
        self.a = A()

    def sayHello(self):
        return self.a.sayHello()

if __name__ == '__main__':
    print B().sayHello()

You would get the following life generated callgraph – the GUI can start, stop and restart tracing and get live updates as the DTrace probes fires:

Click to enlarge

The following screenshot was taken while looking into the printer manager:

Click to enlarge

DTrace for the win!

[Updated] Updated the screenshots.

Click to enlarge

Another example of Python as a DTrace consumer: This small GUI gives an up to date view of the number of syscalls made by an executable. Since this GUI is a live up to date view you can watch the circles appear, grow and become smaller again

Click to enlarge

Now on to other things…maybe creating live animated callgraphs as your program runs?