The Start Of A New Term With Some Great People

I haven’t blogged in a while because it has taken literally weeks upon weeks of trying to bootstrap ARMv6HL for Fedora, pretty much by myself (plus hints from my boss Chris Tyler). I am not a programmer! But anyway, I think it’s coming along to some degree at least (remind me to always respect the complexity of gcc && glibc). In addition to this, I am planning on working with Seneca college for another year and also, this is officially the first time I’ve hit the gym!!! My friend and co-worker Andrew Greene, who has been working hard on various tasks, decided to show me around the machines today which was pretty cool and I already feel a bit better for myself after having gone (hopefully I’ll keep it up). I also have thank my co-workers Jordan and Paul for encouraging me to go in the first place, along with the other life advice they’ve given me. In addition, we got a new team member named Dmitry who’s been pretty much looking after our build farm and making sure all the machines remain up for all the remote users who depend on its availability.

And on a last note, my friend Leopold Antonio Martinez (who’s been like my best friend for life) and I are planning on looking for places to live. He’s helped me out constantly in the past and been there for me all the time. This next step should help me separate from my Italian parents and gain some independence! Thanks Lee 🙂

Anyway, enough of my ranting, here’s hoping for another productive year!

Johnny S.

The Start Of A New Term With Some Great People

Monitoring Our Build Farm — Visually

So now that I’m working full time hours, I’d like to monitor the ARM computers in our build farm more closely and in a nicer manner. These series of scripts (d3 js and Twitter Bootstrap) are based on a whole bunch of other scripts like: Koji Client Command, Icinga, and my own check_koji.pl nagios plugin. The result looks like the following:

farm.py - A python script that gets the needed info being displayed

#!/usr/bin/python

import getpass
import os
import re
import sys
import time
import subprocess
import urllib2, base64

sys.stdout.write("Username: ")
sys.stdout.flush()
user = sys.stdin.readline().strip()

spwd = getpass.unix_getpass()

while (1):
	indx = 0
	buil = {}
	node = ""
	link = ""
	
	node += ("{\"name\":\"Hosts\",\"size\":0,\"group\":0,\"info\":\"root\"}")
	link += ("{\"source\":0,\"target\":0}")
	indx += 1
	
	bsfs = base64.encodestring("%s:%s" % (user, spwd)).replace("\n", "")
	
	try:
		requ = urllib2.Request(sys.argv[1] + "/cgi-bin/status.cgi?host=all")
		requ.add_header("Authorization", "Basic %s" % bsfs)
		resu = urllib2.urlopen(requ)
		webp = resu.read().replace("\0","").replace("\t","").replace("\r","").replace("\n","")
	except:
		print("urllib...")
		time.sleep(60)
		continue
	
	webp = re.sub("<TABLE BORDER=0 CELLSPACING=0 CELLPADDING=0>", "\n<beg line='true'>", webp)
	
	for line in webp.split("\n"):
		if (re.match("^.*>Koji Stat<.*$", line, re.I)):
			host = re.sub("^.*host=([^&]+).*$", "\\1", line).strip()
			
			line = re.sub("^.+(OK|WARNING|UNKNOWN|CRITICAL).+[0-9]+[0-9]+-[0-9]+[0-9]+-[0-9]+[0-9]+[0-9]+[0-9]+[ ]+[0-9]+[0-9]+:[0-9]+[0-9]+:[0-9]+[0-9]+(.+)$", "<td>\\1</td>\\2", line).strip()
			line = line.replace("<TD", "\t<TD")
			part = line.split("\t")
			
			for x in range(0, len(part)):
				part[x] = re.sub("<[^>]*>", " ", part[x]).strip()
			
			stat = part[0]
			info = part[3]
			
			colr = 3
			
			if (stat == "CRITICAL"):
				colr = 3
			if (stat == "WARNING"):
				colr = 2
			if (stat == "OK"):
				colr = 1
				info = "OK"
			
			buil[host] = indx
			
			node += (",\n{\"name\":\"%s [%s]\",\"size\":1,\"group\":%d,\"info\":\"%s\"}" % (host, stat, colr, info))
			link += (",\n{\"source\":%d,\"target\":%d}" % (0, indx))
			indx += 1
	
	pipe = subprocess.Popen(["/usr/bin/koji", "-s", sys.argv[2], "list-tasks"], stdout=subprocess.PIPE)
	
	for line in pipe.stdout.readlines():
		line = line.replace("\t", " ").strip()
		line = re.sub(" [ ]+", " ", line)
		
		regx = re.match("^([^ ]+) .* (CANCELED|FAILED|OPEN|CLOSED) .* build[ ]+([^ ]+)[ ]+([^ ]+)$", line, re.I)
		
		if (regx):
			task = regx.group(4)
			host = regx.group(3).replace("[","").replace("]","")
			stat = regx.group(2)
			info = regx.group(1)
			colr = 4
			
			if (stat == "CANCELED"):
				colr = 3
			if (stat == "FAILED"):
				colr = 3
			if (stat == "OPEN"):
				colr = 5
			if (stat == "CLOSED"):
				colr = 1
			
			if (not host in buil.keys()):
				buil[host] = indx
				
				node += (",\n{\"name\":\"%s [%s]\",\"size\":1,\"group\":%d,\"info\":\"%s\"}" % (host, "REMOTE", 6, "N/A"))
				link += (",\n{\"source\":%d,\"target\":%d}" % (0, indx))
				indx += 1
			
			node += (",\n{\"name\":\"%s [%s]\",\"size\":1,\"group\":%d,\"info\":\"%s\"}" % (task, stat, colr, info))
			link += (",\n{\"source\":%d,\"target\":%d}" % (buil[host], indx))
			indx += 1
	
	fobj = open("force.json", "w")
	fobj.write("{\n\"nodes\":[\n%s\n],\n\"links\":[\n%s\n]\n}\n" % (node, link))
	fobj.close()
	
	print("sleeping...")
	time.sleep(45)

force.js - Some JavaScript that initializes the force directed graph

//var width = 640, height = 560, distance = 64;
var width = 1024, height = 900, distance = 128;
var sizes = [10, 10];
// blue, green, yellow, red, purple, orange, blue
var colors = ["rgba(30, 120, 180, 1)", "rgba(0, 155, 0, 1)", "rgba(250, 195, 0, 1)", "rgba(206, 22, 32, 1)", "rgba(100, 0, 155, 1)", "rgba(255, 160, 0, 1)", "rgba(30, 120, 180, 1)"];
var nodes = "", force = null, svg = null, mous = {x:0,y:0};

function over()
{
	try { var l = force.nodes(); }
	catch(err) { var l = []; }
	
	for (var i in l)
	{
		var r = sizes[l[i].size];
		var a = (l[i].x - r);
		var b = (l[i].x + r);
		var c = (l[i].y - r);
		var d = (l[i].y + r);
		
		if ((a <= mous.x) && (mous.x <= b) && (c <= mous.y) && (mous.y <= d))
		{
			document.getElementById("info").innerHTML = ("<b>" + l[i].name + "</b> :: " + l[i].info);
		}
	}
	
	setTimeout("over()", 0.15 * 1000);
}

function move(e)
{
	var obj = document.getElementsByTagName("svg");
	
	if (obj.length > 0)
	{
		var pt = obj[0].createSVGPoint();
		var scr = obj[0].getScreenCTM();
		var inv = scr.inverse();
		
		pt.x = e.clientX; pt.y = e.clientY;
		mous = pt.matrixTransform(inv);
	}
}

function tick()
{
	try { force.start(); }
	catch(err) { console.log("tick"); }
	setTimeout("tick();", 3 * 1000);
}

function loop()
{
  d3.json("force.json", function(json) {
    var stri = JSON.stringify(json);

    if (stri == nodes)
    {
      return;
    }
    //console.log(nodes+"!="+stri);

    nodes = stri;
    document.getElementById("chart").innerHTML = "";

    force = d3.layout.force()
      .charge(-512)
      .linkDistance(distance)
      .size([width, height]);

    svg = d3.select("#chart").append("svg")
      .attr("width", width)
      .attr("height", height);

    force
      .nodes(json.nodes)
      .links(json.links)
      .start();

    var l = force.nodes();
    document.getElementById("list").innerHTML = "";

    for (var i in l)
    {
      if (l[i].name.match(/^.*cdot.*(WARNING|CRITICAL|UNKNOWN).*$/i))
      {
        document.getElementById("list").innerHTML += ("<div class='alert alert-error' style='margin: 5px;'><b>" + l[i].name + "</b> :: " + l[i].info + "</div>");
      }
    }

    var link = svg.selectAll("line.link")
      .data(json.links)
      .enter().append("line")
      .attr("class", "link")
      .style("stroke-width", function(d) { return Math.sqrt(4); });

    var node = svg.selectAll("circle.node")
      .data(json.nodes)
      .enter().append("circle")
      .attr("class", "node")
      .attr("r", function(d) { return sizes[d.size]; })
      .style("fill", function(d) { return colors[d.group]; })
      .call(force.drag);

    force.on("tick", function() {
      link.attr("x1", function(d) { return d.source.x; })
        .attr("y1", function(d) { return d.source.y; })
        .attr("x2", function(d) { return d.target.x; })
        .attr("y2", function(d) { return d.target.y; });

      node.attr("cx", function(d) { return d.x; })
        .attr("cy", function(d) { return d.y; });
    });
  });

  setTimeout("loop();", 15 * 1000);
}

loop();
tick();
over();
window.onmousemove = move;

force.html - The main page that puts everything together

<html>
	<head>
		<title>Farm Visual</title>
		<script type="text/javascript" src="d3.v2.js"></script>
		<style>
			circle.node
			{
				stroke: #000;
				stroke-width: 2.0px;
			}
			
			line.link
			{
				stroke: #999;
				stroke-opacity: 0.75;
			}
			
			svg
			{
				border: 1px solid black;
			}
			
			#info, #list
			{
				white-space: nowrap;
			}
		</style>
		<link href="/scripts/bootstrap/css/bootstrap.css" rel="stylesheet">
		<link href="/scripts/bootstrap/css/bootstrap-responsive.css" rel="stylesheet">
		<script src="/scripts/jquery.js"></script>
		<script src="/scripts/bootstrap/js/bootstrap.js"></script>
	</head>
	
	<body>
		<center>
			<table width="100%">
				<tr><td colspan="2"><div id='info' class='alert alert-info' style='margin: 5px;'><b>Build Farm Status</b></div></td></tr>
				<tr><td><div class="gallery" id="chart" style="margin: 5px;"></div></td>
				<td valign="top" width="100%"><div id="list"></div></td></tr>
			</table>
		</center>
		<link href="force.css" rel="stylesheet" type="text/css" />
		<script src="force.js" type="text/javascript"></script>
	</body>
</html>

Monitoring Our Build Farm — Visually

Correcting The Automation Of Live-Media Creation Tools

I helped to create a simple script which could assist in the automation and creation of ARM-based images after standardized tools like kickstart, lorax, and livemedia-creator have been used. The problem is that some ARM devices (like OMAP) require there to be a FAT partition and filesystem listed first which contains proper binaries to assist with the initial booting process. Since x86 machines don’t have this requirement, their creation tools don’t include an option to help set this up. The python script listed below helps to take an outputted image and just modify it slightly to meet these requirements for us automatically. Here is an example of the output along with its source code.

# fdisk -l F18-panda-20121218.img
...
                 Device Boot      Start         End      Blocks   Id  System
F18-panda-20121218.img1   *        2048      411647      204800    6  FAT16
F18-panda-20121218.img2          411648     1435647      512000   82  Linux swap / Solaris
F18-panda-20121218.img3         1435648     5941247     2252800   83  Linux

# ./livemedia-modifier --image=F18-panda-20121218.img --platform=omap
...
loop deleted : /dev/loop0

Modification completed successfully!

# fdisk -l F18-panda-20121218.img
...
                 Device Boot      Start         End      Blocks   Id  System
F18-panda-20121218.img1   *        2048      411647      204800    c  W95 FAT32 (LBA)
F18-panda-20121218.img2          411648     1435647      512000   82  Linux swap / Solaris
F18-panda-20121218.img3         1435648     5941247     2252800   83  Linux

# 

Source Code (Git Hub)
Binary Program (Fedora People)
Fedora Wiki

Correcting The Automation Of Live-Media Creation Tools

End Of Summer — Improving Online Dating!

So my friends have been encouraging me to try and meet new people on the Internet through some various online dating sites. Creating a profile was a good start but I quickly found myself getting really distracted by all the profile pictures when I should be reading the profile text instead. I decided to create a simple little grease-monkey script to remove all of the pics on the site and therefore force me to read one’s profile instead. Here’s the result:

// ==UserScript==
// @name        okcp
// @namespace   gm-dev
// @include     http://www.okcupid.com/*
// @version     1
// ==/UserScript==

var pics = document.getElementsByTagName("img");

for (var pici in pics)
{
	try
	{
		var wide = pics[pici].width;
		var high = pics[pici].height;
		pics[pici].src = "http://142.204.133.82/tmp/images/qmt.png";
		pics[pici].onmouseover = "";
		pics[pici].width = wide;
		pics[pici].height = high;
	}
	catch(e)
	{
		console.log("pic error?");
	}
}

var side = document.getElementById("left_bar");

if (side)
{
	side.style.width = "200px";
}
End Of Summer — Improving Online Dating!

Icinga Koji Monitoring Page + Greasemonkey Script

Following up on my work with the Koji Icinga/Nagios based monitoring system for our Fedora ARM machines: A screenshot of how it looks along with a GM script that helped to make it look like that. The Perl script I made in an earlier post checks for some of the following errors that could cause a Koji builder to fail. And on an semi-related note, if you’d like to follow our build progress, I’m currently testing a Twitter script posting here: http://twitter.com/fedoraarmstats

 

  • Any non-rw mount points
  • Permissions and sizes for directories: [/, /var, /tmp, /var/{lib,cache}/mock]
  • Running kojid systemd/systemctl cgroup process numbers, commands
  • Koji config file, cert date, system time checks
  • Koji host, enabled, ready, not-ready+tasks, update-time, task-times

 

// ==UserScript==
// @name        icinga
// @namespace   icinga-dev
// @include     http://hongkong.proximity.on.ca/icinga/cgi-bin/status.cgi?host=all*
// @version     1
// ==/UserScript==

var cnmatch = false, globflag = false;
var tablelist = document.getElementsByTagName("table");
var bodylist = [], rowlist = [], collist = [];
var regxobjc = new RegExp("Koji Stat", "g");

for (var tableitem in tablelist)
{
	try { cnmatch = tablelist[tableitem].className.match(/^.*status.*$/); }
	catch(erro) { cnmatch = false; }
	
	if (cnmatch)
	{
		bodylist = tablelist[tableitem].childNodes;
		
		for (var bodyitem in bodylist)
		{
			try { cnmatch = (bodylist[bodyitem].nodeName.toLowerCase() == "tbody"); }
			catch(erro) { cnmatch = false; }
			
			if (cnmatch)
			{
				var lastclas = "", lastvalu = "";
				
				rowlist = bodylist[bodyitem].childNodes;
				
				for (var rowitem in rowlist)
				{
					var rowflag = true;
					var colnumb = 0, colobjc = null;
					
					try { cnmatch = (rowlist[rowitem].nodeName.toLowerCase() == "tr"); }
					catch(erro) { cnmatch = false; }
					
					if (cnmatch)
					{
						collist = rowlist[rowitem].childNodes;
						
						for (var colitem in collist)
						{
							try { cnmatch = (collist[colitem].nodeName.toLowerCase() == "td"); }
							catch(erro) { cnmatch = false; }
							
							if (cnmatch)
							{
								if (colnumb == 0)
								{
									if (collist[colitem].innerHTML != "")
									{
										lastvalu = collist[colitem].innerHTML;
									}
									
									else
									{
										colobjc = collist[colitem];
									}
								}
								
								if (colnumb == 1)
								{
									if (!collist[colitem].innerHTML.match(regxobjc))
									{
										rowflag = false;
									}
								}
								
								lastclas = collist[colitem].className;
								++colnumb;
							}
						}
						
						if ((colobjc != null) && (colnumb > 3))
						{
							lastvalu = lastvalu.replace(/class=['"]status[^'"]+['"]/g, lastclas);
							colobjc.innerHTML = lastvalu;
							colobjc.className = lastclas;
						}
					}
					
					if ((rowflag == false) || ((0 < colnumb) && (colnumb < 4)))
					{
						try { rowlist[rowitem].style.display = "none"; }
						catch(erro) { cnmatch = false; }
					}
				}
			}
		}
	}
}

Icinga Koji Monitoring Page + Greasemonkey Script

My First Icinga/Nagios/Perl Monitoring Script

Hey all, I made a monitoring module that one could use to more closely monitor Koji builder participating in a build farm. I enjoyed working on this script because it pretty much required me to learn the basics of a new language (Perl) which is a bit different than what I’m used to (Python). It was fun and hopefully it proves to be useful for our team.

for x in {1..12}-{1..5} ; do echo "[$x]" ; /usr/lib64/nagios/plugins/check_nrpe -H $x -c check_koji -t 60 ; done
[1-1]
Mounts [Options OK] - Dirs [Perms OK] - Dirs [Sizes OK] - System [Time OK] - Koji [Config OK] - Koji [Cert OK] - Koji [2 Procs OK] - Koji [Host OK] - Koji [Enabled OK] - Koji [Ready OK] - Koji [0 Tasks OK]
[1-2]
Mounts [Options OK] - Dirs [Perms OK] - Dirs [Sizes OK] - System [Time OK] - Koji [Config OK] - Koji [Cert OK] - Koji [10 Procs OK] - Koji [Host OK] - Koji [Enabled OK] - Koji [Ready OK] - Koji [1 Tasks OK]
[1-3]
Mounts [Options OK] - Dirs [Perms OK] - Dirs [Sizes OK] - System [Time OK] - Koji [Config OK] - Koji [Cert OK] - Koji [19 Procs OK] - Koji [Host OK] - Koji [Enabled OK] - Koji [Ready OK] - Koji [1 Tasks OK]
[1-4]
Mounts [Options OK] - Dirs [Perms OK] - Dirs [Sizes OK] - System [Time OK] - Koji [Config OK] - Koji [Cert OK] - Koji [2 Procs OK] - Koji [Host OK] - Koji [Enabled OK] - Koji [Ready OK] - Koji [0 Tasks OK]
[1-5]
Mounts [Options OK] - Dirs [Perms OK] - Dirs [Sizes OK] - System [Time OK] - Koji [Config OK] - Koji [Cert OK] - Koji [1 Procs OK] - Koji [Host OK] - Koji [Enabled OK] - Koji [Ready OK] - Koji [0 Tasks OK]
...

Source Code (Git Hub)
Source Code (Fedora People)

My First Icinga/Nagios/Perl Monitoring Script

Really Really Random – I graduated yay! (but not with honours blah…)

Basically just a break-type post (like taking a 5 min break to go outside which all computer people should do btw… stretch and relax those eye balls). Anyway, I just want to show appreciation for the various people who have helped me tremendously along the way!

The first is Chris Tyler (my boss) who is a really really smart and nice guy who has a general yet complete knowledge of how all of this stuff works at levels I can’t even understand (also he can write some incredible SQL statements, I think it’s because of his previous work experience with programming and PostgreSQL… ewwwwww!). Also, anyone reading this should definitely buy his book which covers concepts I still suck at like software RAIDs and LVMs in Linux. You can find it here here or here. Really good stuff!

I’d like to thank Jon Masters who is an insanely smart Linux Kernel Engineer. This guy honestly puts people like me to shame regarding how much in-depth knowledge he has about how computers run at the lowest-levels imaginable. Seriously crazy, but also, he mentioned me in a Linux Weekly News article which made me day, heh! If you need more links just Google his name, his stuff is everywhere pretty much.

On a semi-related note, my friend Drew (who’s a DJ) helped me greatly back in the day when a certain someone (she shall remain nameless) broke my heart :). This guy is note only amazing socially but also technically too. At any rate, you should definitely check out his stuff like Twitter SoundCloud FaceBook. Thanks for your help Drew!

Lastly, the only reason I chose to quit working @ Zehrs and start a new life @ Seneca was because of my friend Sam. Sadly, this guy (like me), doesn’t really participate much in the online social world so I can’t really plug anything for him but I thought I’d give him the last mention in this post. Thanks for guiding me in life man, I owe you!

Really Really Random – I graduated yay! (but not with honours blah…)

Possible Raspberry Pi Splash Screen

Using the source code already available and provided to everyone in a package named raspberrypi-vc-demo-source and running yum groupinstall "Development Tools" "Development Libraries", I was able to take a Raspberry Pi GIF animation, split it up into frames (bmp) via http://www.xoyosoft.com/gs/, convert them into to raw file formats (rgb), and place them as an array of textures on a cube face via OpenGL in C located under /usr/share/raspberypi-vc-demo-source/hello-pi.

Running On A F17 Rasp Pi

Automated bmp-to-rgba Converter (Py)

import sys
f=open(sys.argv[1],"r")
g=open(sys.argv[2],"w")
h=f.read(54)
l=0
m=0
while(1):
	d=f.read(3)
	a=chr(255)
	l+=len(d)
	if (not d):
		break
	rgb=(hex(ord(d[0]))+hex(ord(d[1]))+hex(ord(d[2])))
	if (rgb == "0x250x210xd"):
		e=(chr(0)+chr(0)+chr(0))
		a=chr(0)
	e=(d[2]+d[1]+d[0])
	sys.stderr.write(rgb+"\n")
	o=(e+a)
	m+=len(o)
	g.write(o)
g.close()
f.close()
print(l,":",m)

Binary Program (Fedora People)
Source Code (Git Hub)

Possible Raspberry Pi Splash Screen