No Huddle Offense

"Individual commitment to a group effort-that is what makes a team work, a company work, a society work, a civilization work."

SmartStack = SmartOS + OpenStack (Part 3)

February 28th, 2012 • Comments Off on SmartStack = SmartOS + OpenStack (Part 3)

This is the third part os the blog post series. Previous parts can be found here: 1 2

To ensure the proper startup of nova-compute we will register it using SMF – this will also increase the dependability of the services. To start we will define a properties file for nova-compute – we will be using a glance and rabbitMQ host which run on a different host:

--connection_type=fake
--glance_api_servers=192.168.56.101:9292
--rabbit_host=192.168.56.101
--rabbit_password=foobar
--sql_connection=mysql://root:foobar@192.168.56.101/nova

We will store this contents in the file /data/workspace/smartos.cfg. We will also create an XML file with the manifest definition – it has some dependencies defined:

<?xml version='1.0'?>
<!DOCTYPE service_bundle SYSTEM '/usr/share/lib/xml/dtd/service_bundle.dtd.1'>
<service_bundle type='manifest' name='openstack'>
  <service name='openstack/nova/compute' type='service' version='0'>
    <create_default_instance enabled='true'/>
    <single_instance/>
    <dependency name='fs' grouping='require_all' restart_on='none' type='service'>
      <service_fmri value='svc:/system/filesystem/local'/>
    </dependency>
    <dependency name='net' grouping='require_all' restart_on='none' type='service'>
      <service_fmri value='svc:/network/physical:default'/>
    </dependency>
    <dependency name='zones' grouping='require_all' restart_on='none' type='service'>
      <service_fmri value='svc:/system/zones:default'/>
    </dependency>
    <exec_method name='start' type='method' exec='/usr/bin/nohup /data/workspace/nova/bin/nova-compute --flagfile=/data/workspace/smartos.cfg &amp;' timeout_seconds='60'>
      <method_context>
        <method_environment>
          <envvar name="PATH" value="/ec/bin/:$PATH"/>
        </method_environment>
      </method_context>
    </exec_method>
    <exec_method name='stop' type='method' exec=':kill' timeout_seconds='60'>
      <method_context/>
    </exec_method>
   <stability value='Unstable' />
  </service>
</service_bundle>

This can be imported using the svccfg command. After doing so we can use al the known commands to verify all is up and running:

[root@08-00-27-e3-a9-19 /data/workspace]# svcs -p openstack/nova/compute
STATE          STIME    FMRI
online         14:04:52 svc:/openstack/nova/compute:default
               14:04:51     3142 python

Now up to integrate vmadm…

SmartStack = SmartOS + OpenStack (Part 2)

February 15th, 2012 • 2 Comments

This is the second post in this series. Part 1 can be found here.

After setting up nova on SmartOS you should be able to launch the nova-compute service – Andy actually managed to get the nova-compute instance up and running:

./bin/nova-compute --connection_type=fake --sql_connection=mysql://root:admin@192.168.0.189/nova --fake_rabbit
2012-02-15 10:26:32,205 WARNING nova.virt.libvirt.firewall [-] Libvirt module could not be loaded. NWFilterFirewall will not work correctly.
2012-02-15 10:26:32,374 AUDIT nova.service [-] Starting compute node (version 2012.1-LOCALBRANCH:LOCALREVISION)
2012-02-15 10:26:33,099 INFO nova.rpc [-] Connected to AMQP server on localhost:5672

Note that for now we will use a fake RabbitMQ connection. The connection to the MySQL server however is mandatory.

This is the first step in plan to get OpenStack running on SmartOS. Next step will be to look into integration vmadm as a driver in OpenStack. The overall Ideas we have can be found in this slideset on google docs.

SmartStack = SmartOS + OpenStack (Part 1)

February 12th, 2012 • 4 Comments

[Update] – The first shot didn’t work – so I updated this post!

This is the first post of hopefully a series of post about getting OpenStack up and running on SmartOS. There is a blueprint for this effort. For this first post we will focus on setting up SmartOS and try to get some needed packages up and running.

I used the the following SmartOS iso image which will you – when booted – guide you through a small setup routine: smartos-20120210T043623Z.iso. After this you need to configure the pkg repositories as described in this tutorial. I encourage you all to have two discs attached – One for the zones – and one for the data. So since the setup routine created the zones pool I will first create a data pool on the second disc:

zpool create data c0t2d0
zfs create data/ec
zfs set mountpoint=/ec data/ec
zfs mount data/ec
cd / && wget -O- -q http://svn.everycity.co.uk/public/solaris/misc/ \
                    pkg5-smartos-bootstrap-20111221.tar.gz | gtar -zxf-

One major dependency is python, gcc and some tools and libs – and we will also need git so we can get the source code of nova later on:

/ec/bin/pkg install pkg:/database/mysql-55/client@5.5.16-0.162 \
                    pkg:/library/python/mysqldb@1.2.3-0.162
/ec/bin/pkg install python26 git setuptools-26 gcc-44 libxslt gnu-binutils

Next step will be to create a workspace file-system using ZFS (Optional) – this will allow me to do rollbacks later on during the porting of OpenStack:

zfs create data/workspace
cd /data/workspace/

Let’s clone nova and install pip:

export PATH=/ec/bin:$PATH
easy_install pip
git clone git://github.com/tmetsch/nova.git
cd nova/tools
export CFLAGS="-D_XPG6 -std=c99"
pip install -r pip-requires

Now let’s see if we can get the test up and running:

cd ..
./run_tests.sh

This will do for the first steps – I’ll be going to get the dependencies of nova up and running next – Will than post the next blog post.

Percent done: 1%

Coffee instead of snakes – Openshift fun (2)

February 8th, 2012 • Comments Off on Coffee instead of snakes – Openshift fun (2)

In my last post I demoed how OpenShift can be used to deploy WSGI based Python application. But since OpenShift also supports other languages including Java I wanted to give it a shot.

So I developed a very minimalistic application which emulates a RESTful interface. Of course it takes the simplistic – all time favorite – Hello World approach for this. Client can create resources, which when queried will return the obligatory ‘Hello <resource name>’.

To start create a new java application in your OpenShift dashboard. When cloning the git repository – which was created – you will notice that you basically get a maven project with some templates in it. Now you can simple use maven to test and develop your application. When done do a git push and your application is ready to go.

The implementation is pretty straight forward:

/**
 * Simple REST Hello World Service.
 *
 * @author tmetsch
 *
 */
public class HelloServlet extends HttpServlet {
    [...]
    @Override
    protected final void doGet(final HttpServletRequest req,
            final HttpServletResponse resp) throws ServletException,
            IOException {
            [...]
    }
    [...]

So beside from extending the HttpServlet class all you need to do is implement the doGet and doPost methods. When done edit the ‘web.xml’ file in the folder src/main/webapp:

[...]
<servlet>
    <servlet-name>HelloWorld</servlet-name>
    <servlet-class>the.ultimate.test.pkg.HelloServlet</servlet-class>
</servlet>
<servlet-mapping>
    <servlet-name>HelloWorld</servlet-name>
    <url-pattern>/users/*</url-pattern>
</servlet-mapping>
[...]

This will make sure you the application can be found under the right route. If you like you can also add an index.html file in the same folder so people can read a brief introduction when visting the top layer of you application. In my case that would be http://testus-sample.rhcloud.com – The application itself can be found under: http://testus-sample.rhcloud.com/users/.

So I like the approach OpenShift takes here with maven. You can simply add you dependencies like jmock, junit etc. and during deployment it is taken care of that everything falls in place. Also if you write Unittest it’ll also ensure that you’re application will work – Non functional apps will not be deployed obviously if you write a Unittest and use maven. It’s pretty easy to write Unittest for the HttpServlet class if you use a mocking framework like jmock. You can get the source code at github.

What is OCCI?

December 2nd, 2009 • 2 Comments

OCCI has been around for a while now, and for those just starting to fiddle with it here is a ‘introduction’. OCCI is more then just a document and specification…’

The OGF Open Cloud Computing Interface Working Group (OCCI-wg) is developing a clean, open API for ‘Infrastructure as a Service’ (IaaS) based Clouds. IaaS is one of three primary services: Infrastructure, Software, and Platform of the emerging Cloud industry.

OCCI–wg is a working group of the OGF established in March 2009. The group has active membership of over 200 individuals and is led by four chairs from industry, academia, service providers and end-users. Several members are from commercial service providers that are committed to implementing the OGF-OCCI specification.

Ignacio and me during the founding session.

Ignacio and me during the founding session.

The idea of cost reduction, GreenIT and on-demand resource planning lead to the idea of having elastic environments which scale on demand. Today’s demand for elastic environments in which public Clouds (like Amazon’s EC2 offerings) and private Clouds (locally installed resources) interoperate (Hybrid-,Federate- or InterClouds) show the need for standardization. Differentiation in the offerings is done by extensions and prices not by core functionalities. The need to interface all Cloud providers (private as well as public) in the same manner shows the need for standards for core features. Therefore main drivers behind the specification are to ensure Portability, Interoperability and Integration between Cloud service providers and end-users.

OCCI will be a very slim RESTful based API which can be easily extended. Without the overhead of many similar protocols the REST approach allows users to easily access their services. Every resource is uniquely addressed using a Uniform Resource Identifier (URI). Based on a set of operations – create, retrieve, update and delete (CRUD) – resources can be managed. Currently three types of resources are considered: storage, network and compute resources. Those resources can be linked together to form a virtual machine with assigned attributes. For example, it is possible to provision a machine which has 2GB of RAM, one hard disk and one network interface.

The Specification itself is setup to be modular. Therefore it is delivered as a document series. Each of the documents focuses on a certain topic and can be independently used from the others. Until now the OCCI specification consist of four modules: The Core specification, a description of how IaaS based resources should look like, a HTTP rendering for easy machine-to-machine management and a XHTML rendering for creation of human-to-machine interfaces.

The following diagram shows an example RESTful query towards an OCCI compliant service provider:

occi

OCCI overview diagram

An HTTP GET operation is performed on an unique URI representing a virtual machine of Solaris zone. The URI encapsulates the instance and provider id. The OCCI specification describes how operations can be performed upon resources (Compute, Storage and Network). Resources can be linked so they described e.g. a Virtual Machine. Each resource type has a certain set of attributes (e.g. CPU speed, Storage size and IP addresses).

The CRUD operations can be performed upon the resources. During the past months two reference implementations have emerged. Based on drafts of the specification a group of the University of Madrid (UCM – OpenNebula) and a group inside of the National Institute of Nuclear Physics (INFN) created reference implementations.

The idea which led to the creation of the OCCI group has been triggered and sponsored by the EU project RESERVOIR. RESERVOIR tries to establish a tool suite for Cloud computing middleware in which the interfaces should be standardized. One of these interface will be an implementation of the OCCI specification. The work done within the group has gained an significant impact in the Cloud community. Blogs, technology related websites, twitter (Tag #OCCI) and journals have driven and reported about OCCI thanks to the open process of the group.

Currently the OCCI group is finalizing and documenting the initial draft specification. After only half a year the OCCI group succeeded in delivering on of the first standards for the Cloud community. This could be achieved thanks to the big momentum the group has had. Due to the specification’s high extensibility the group will start to work on extensions after the first draft has been delivered. In future OCCI might become a provider of a complete Cloud computing interface stack which covers IaaS as well as PaaS based offerings.

OCCI plays a major role in today’s Cloud computing standardization efforts. The group is actively collaborating with other groups from SNIA for storage and DMTF for management standards. A white paper has been published describing how SNIA’s Cloud Data Management Interface and OCCI can work in conjunction. The work is also featured on the http://www.cloud- standards.org wiki where Cloud related standards are coordinated by the major Standards Development Organizations (SDOs).

For more information visit http://www.occi-wg.org/