July 16th, 2013 • Comments Off on Learn Service Behaviour
This started off as just an idea in my head a while ago and soon become my Master thesis. The idea is to apply Machine Learning/Data Analysis methodologies to Data derived from tools such as DTrace. Based on the learned knowledge it would then be possible to create adaptive/agile systems. For example to balance out your workload as a service provider overnight. Or move your compute close to your data. Or to learn how to best configure your system (by using Software Defined Networking). Or to simple tune your threshold values in your monitoring system etc. Or you name it – it might be doable.
After finally finishing my Master thesis – doing that next to work is quite a challenge I can tell now 🙂 – I can certainly say that some stuff can indeed be learned. During the work on my thesis I sketched out a, in browser programmable (Python of course :-)), Analytics as a Service (I just like the acronym for it :-)) which can be used learn from Data derived from DTrace (See 1).
As a first step it is useful to select the resources you want to look at. The should have a relevance to the behavior of the applications and services. The USE Method demoed by Brendan Gregg might be a good start. Once you know what you want to look for it is possible to gather some data. For example using the DTrace consumer for Python (See 2), like I did. Cool think is now that thanks to Python you can send it around (pika), store it (MongoDB, sqlite) and process it (scikit-learn) easily. Just add a few API for abstraction, a rich web application for creating python notebooks and you have the Analytics as a Service.
Now we can work through the data with some simple steps.
- Step 1 – Analyze the time series you got with some first methods. This could reach from calculating means, averages, etc over smoothing or regression analyses to looking into correlations in the data. Now you will have a good knowledge of which time series are interesting for further analysis.
- Step 2 – Cluster the applications/services based on the data points to get an first overview of how they fit together. Again simple k-means clustering can be an initial step. Remember sometimes the simplest methods are the best 🙂
- Step 3 – Based on what has been learned till now try to apply mechanism to analyze the covariance/correlation between the applications/services. Once done you get a nice graph which represents behavior of your overall environment.
- Step 4 – Go beyond the simple and try to build Bayesian networks on what you got now. Go look at decision trees if possible to try build that adaptive/agile system.
- Step 5 – Well as usual the sky might actually be the limit.
Results of each of the steps can be seen as models which can be compared with new incoming data from DTrace. Based on the ‘intelligence’ of the comparison of the new date with the learn model (knowledge) the adaptive/agile system can be build. Continuously updating your learned model in the learning process is key – we don’t want to ‘predict’ the future using a crystal ball. This is just the tip of the iceberg of all stuff worked on and discovered in my Master thesis – but as usual – so little time so much to do 🙂 But maybe I’ll find some time soon to sketch out the learning process and share some details of what I was able to let the computer learn…
Categories: Personal • Tags: Analytics, Data Science, DTrace, Machine Learning, OpenSolaris, Python, SmartOS • Permalink for this article
July 6th, 2011 • Comments Off on OpenIndiana zones & DTrace
Let’s assume we want to create a Solaris zone called foo on an OpenIndiana box. This post will walk you to all the steps necessary to bootstrap and configure the zone, so it’s ready to use without any user interaction. Also briefly discussed is how to limit the resources a zone can consume.
7 Steps are included in this mini tutorial:
- Step 1 – Create the zpool for your zones
- Step 2 – Configure the zone
- Step 3 – Sign into the zone
- Step 4 – Delete and unconfigure the zone
- Step 5 – Limit memory
- Step 6 – Use the fair-share scheduler
- Step 7 – Some DTrace fun
Step 1 – Create the zpool for your zones
First a pool is created and mounted to /zones. Deduplication is activated for this pool & a quota is set – so the zone has a space limit of 10Gb.
zfs create -o compression=on rpool/export/zones
zfs set mountpoint=/zones rpool/export/zones
zfs set dedup=on rpool/export/zones
mkdir /zones/foo
chmod 700 /zones/foo
zfs set quota=10g rpool/export/zones/foo
Step 2 – Configure the zone
A zone will be configured the way that it has the IP 192.168.0.160 (Nope – DHCP doesn’t work here :-)) and uses the physical device rum2. Otherwise the configuration is pretty straight forward. (TODO: use crossbow)
zonecfg -z foo "create; set zonepath=/zones/foo; set autoboot=true; \
add net; set address=192.168.0.160/24; set defrouter=192.168.0.1; set physical=rum2; end; \
verify; commit"
zoneadm -z foo verify
zoneadm -z foo install
Too ensure that when we boot the zone everything is ready to use without any additional setups, a file called sysidcfg is placed in the /etc of the zone. This will make sure that when we boot all necessary parameters like a root password, the network or the keyboard layout are automatically configured. Also the host’s resolv.conf is copied to the zone (this might not be necessary if you have a properly setup DNS server – than you can configure that DNS server in the sysidcfg file – mine does not know the hostname foo so that is why I do it this way) and the nsswitch.conf file is copied so it’ll use the resolv.conf file. Finally the zone is started…
echo "
name_service=NONE
network_interface=PRIMARY {hostname=foo
default_route=192.168.0.1
ip_address=192.168.0.160
netmask=255.255.255.0
protocol_ipv6=no}
root_password=aajfMKNH1hTm2
security_policy=NONE
terminal=xterms
timezone=CET
timeserver=localhost
keyboard=German
nfs4_domain=dynamic
" &> /zones/foo/root/etc/sysidcfg
cp /etc/resolv.conf /zones/foo/root/etc/
cp /zones/foo/root/etc/nsswitch.dns /zones/foo/root/etc/nsswitch.files
zoneadm -z foo boot
To create a password you can use the power of Python – the old way of copying the passwords from /etc/shadow doesn’t work on newer Solaris boxes since the value of CRYPT_DEFAULT is set to 5 in the file /etc/security/crypt.conf:
python -c "import crypt; print crypt.crypt('password', 'aa')"
Step 3 – Sign into the zone
Now zlogin or ssh can be used to access the zone – Note that the commands mpstat and prtconf will show that the zone has the same hardware configuration as the host box (zfs list – will show that disk space is already limited). In the next steps we want to limit those…
Step 4 – Delete and unconfigure the zone
First we will delete the zone foo again:
zoneadm -z foo halt
zoneadm -z foo uninstall
zonecfg -z foo delete
Step 5 – Limit memory
Following the steps above just change the configuration of the zone and add the capped-memory option. In this example it’ll limit the memory available to the zone. When running prtconf it’ll show that the zone only has 512Mb RAM – mpstat will still show all CPUs of your host box.
zonecfg -z foo "create; set zonepath=/zones/foo; set autoboot=true; \
add net; set address=192.168.0.160/24; set defrouter=192.168.0.1; set physical=rum2; end; \
add capped-memory; set physical=512m; set swap=512m; end; \
verify; commit"
Step 6 – Using Resource Pools
While using resources pools it is possible to create a resource pool for a zone which only has one CPU assigned. Use the pooladm command to configure a pool called pool1:
poolcfg -c 'create pset pool1_set (uint pset.min=1 ; uint pset.max=1)'
poolcfg -c 'create pool pool1'
poolcfg -c 'associate pool pool1 (pset pool1_set)'
pooladm -c # writes to /etc/pooladm.conf
To restore the old pool configuration run ‘pooladm -x‘ and ‘pooladm -s‘
Now just configure the zone to use and associate it with the pool:
zonecfg -z foo "create; set zonepath=/zones/foo; set autoboot=true; \
set pool=pool1; \
add net; set address=192.168.0.160/24; set defrouter=192.168.0.1; set physical=rum2; end; \
add capped-memory; set physical=512m; set swap=512m; end; \
verify; commit"
Running mpstat and prtconf in the zone will show only one CPU and 512Mb RAM.
Step 6 – Use the fair-share scheduler
Also if you have several zones running in one pool you want to modify the pool to use FSS – so a more important zone gets privileged shares:
poolcfg -c 'modify pool pool_default (string pool.scheduler="FSS")'
pooladm -c
priocntl -s -c FSS -i class TS
priocntl -s -c FSS -i pid 1
And during the zone configuration define the rctl option – This example will give the zone 2 shares:
zonecfg -z foo "create; set zonepath=/zones/foo; set autoboot=true; \
add net; set address=192.168.0.160/24; set defrouter=192.168.0.1; set physical=rum2; end; \
add capped-memory; set physical=512m; set swap=512m; end; \
add rctl; set name=zone.cpu-shares; add value (priv=privileged,limit=2,action=none); end; \
verify; commit"
Step 7 – Some DTrace fun
DTrace can ‘look’ into the zones – For example to let DTrace look at the files which are opened by process within the zone foo you can simply add the predicate ‘zonename == “foo”‘:
pfexec dtrace -n 'syscall::open*:entry / zonename == "foo" / \
{ printf("%s %s",execname,copyinstr(arg0)); }'
I was researching this stuff to create a Python module to configure and bootstrap zones so I can monitor the zones & their previously created SLAs.
Categories: Personal, Work • Tags: DTrace, OpenSolaris, Python, ZFS • Permalink for this article
July 1st, 2011 • Comments Off on Python & DTrace
The following DTrace script can be used to trace Python. It will show you the file, line-number, the time it took to get from the last line to the current one and the (indented) function it is in. This helps understanding the flow of your Python code and can help finding bugs & timing issues. As a little extra it is setup the way that it only shows the files of your code – not the once located in /usr/lib/*. This helps with readability and makes the output more dense since the site-packages are left out.
I now have this running in the background on a second screen whenever I code python. First screen holds my IDE – and next to all the other stuff I do it gives great up-to-date information on the stuff you’re coding. Again proof for the case that coding Python on Solaris is a great idea (Also thanks to DTrace). I know this is simple stuff – but sometimes the simple stuff helps a lot! Now it would only be nice to create a GUI around it 🙂 First the DTrace script is shown; example output is further below…
#pragma D option quiet
self int depth;
self int last;
dtrace:::BEGIN
{
printf("Tracing... Hit Ctrl-C to end.\n");
printf(" %-70s %4s %10s : %s %s\n", "FILE", "LINE", "TIME",
"FUNCTION NAME", "");
}
python*:::function-entry,
python*:::function-return
/ self->last == 0 /
{
self->last = timestamp;
}
python*:::function-entry
/ dirname(copyinstr(arg0)) <= "/usr/lib/" /
{
self->delta = (timestamp - self->last) / 1000;
printf(" %-70s %4i %10i : %*s> %s\n", copyinstr(arg0), arg2, self->delta,
self->depth, "", copyinstr(arg1));
self-> depth++;
self->last = timestamp;
}
python*:::function-return
/ dirname(copyinstr(arg0)) <= "/usr/lib/" /
{
self->delta = (timestamp - self->last) / 1000;
self->depth--;
printf(" %-70s %4i %10i : %*s< %s\n", copyinstr(arg0), arg2, self->delta,
self->depth, "", copyinstr(arg1));
self->last = timestamp;
}
Example output:
$ pfexec dtrace -s misc/py_trace.d
Tracing... Hit Ctrl-C to end.
FILE LINE TIME : FUNCTION NAME
[...]
/home/tmetsch/data/workspace/pyssf/occi/service.py 144 1380 : > put
/home/tmetsch/data/workspace/pyssf/occi/service.py 184 13 : > parse_incoming
/home/tmetsch/data/workspace/pyssf/occi/service.py 47 9 : > extract_http_data
/home/tmetsch/data/workspace/pyssf/occi/service.py 68 39 : < extract_http_data
/home/tmetsch/data/workspace/pyssf/occi/service.py 70 10 : > get_parser
/home/tmetsch/data/workspace/pyssf/occi/registry.py 60 36 : > get_parser
/home/tmetsch/data/workspace/pyssf/occi/registry.py 83 12 : < get_parser
/home/tmetsch/data/workspace/pyssf/occi/service.py 77 8 : < get_parser
/home/tmetsch/data/workspace/pyssf/occi/protocol/rendering.py 99 11 : > to_entity
/home/tmetsch/data/workspace/pyssf/occi/protocol/rendering.py 81 10 : > extract_data
/home/tmetsch/data/workspace/pyssf/occi/protocol/rendering.py 37 11 : > __init__
/home/tmetsch/data/workspace/pyssf/occi/protocol/rendering.py 41 11 : < __init__
/home/tmetsch/data/workspace/pyssf/occi/protocol/rendering.py 97 14 : < extract_data
[...]
This was inspired by the py_flowtime.d script written by Brendan Gregg.
Categories: Personal, Work • Tags: DTrace, OpenSolaris, Python • Permalink for this article
