[update] Koji Hosts Page

There is a bug in the code-change I wrote to modify the look of the Koji Build System Hosts page. If the last-update timestamp was blank, it would cause the embedded py script to crash and freeze the page. Below is how the “better” code should read…

Before:

After:

      #for $host in $hosts
        <tr class="$util.rowToggle($self)">
          <td>$host.id</td>
        #if ($util.formatTime($host.last_update) != ""):
          #if (($time.time() - $time.mktime($time.strptime($util.formatTime($host.last_update),'%Y-%m-%d %H:%M:%S'))) < (15 * 60)):
            <td><a href="hostinfo?hostID=$host.id"><font color="green">$host.name</font></a></td>
          #else
            <td><a href="hostinfo?hostID=$host.id"><font color="red">$host.name</font></a></td>
          #end if
        #else
          <td><a href="hostinfo?hostID=$host.id">$host.name</a></td>
        #end if
          <td>$host.arches</td>
          <td class="$str($bool($host.enabled)).lower()">#if $host.enabled then $util.imageTag('yes') else $util.imageTag('no')#</td>
          <td class="$str($bool($host.ready)).lower()">#if $host.ready then $util.imageTag('yes') else $util.imageTag('no')#</td>
          <td>$util.formatTime($host.last_update)</td>
        </tr>
      #end for

[update] Koji Hosts Page

The Web Part Of Styrene

I was requested to blog more about my actual work instead of the huge flood of off-topic posts that I usually do. So here is a post about Styrene’s web interface end of things. As you may see, one is able to perform a range of package/build “actions” and Styrene is able to handle multiple users as well.



How to use the command line:
– Make a config file like the example provided below
– Load a DB with the required package set and dependencies
– Mark the DB initially to get the latest build information
– Run the koji-que loop in one bash shell to auto build packages
– Run the mark-db every 1+ hours to refresh the DB information

Here is how the back-end of things look:

Usage: styrene <command>	[--db=<db name>] [--release=<release number>] [--kojihub=<koji url>] [--kojitag=<tag name>]
				[--binrepo=<bin repo>] [--srcrepo=<src repo>] [--srcarch=<src arch>]
				[--pkg=<pkg name>] [--state=<pkg state>] [--comment=<pkg comment>] [--override=<envr override>]
				[--num=<general number>] [--time=<time in seconds>]

Commands:

(Db Operations)
			load-db			Create and load a local database with packages from a specific architecture
			update-db		Append a local database with packages from a specific architecture

(Pkg Info)
			list-unbuilt		Display a list of currently unbuilt packages
			list-ready		List any packages that are ready to be built (all dependencies satisfied)
			pkg-info		Display various package info/stats from the local db
			mark-pkg		Mark a pkg state in the local db

(Koji Info)
			koji-info		Get various info/stats for a pkg from koji
			koji-prev		Print the latest built version of a package if one exists
			koji-equiv		Check to see if koji has a pkg version that is >= to this local db version
			koji-que		Send a package to koji for a build attempt
			koji-tasks		List the current active tasks for this user

(Auto Building)
			mark-db			foreach (list-unbuilt) use (koji-info) to (mark-pkg)
			que-ready		foreach (list-ready) use (koji-tasks) to throttle (koji-que)

(Web Interface)
			cgi-bin			Generate a CSS/HTML/JavaScript (web-based) interface

(Pkg Fixing)
			fix-spec		Attempt to auto-fix a non-building SRPM by auto-merging old spec file changes

(Misc Cmds)
			show-vars		Print the different variable values for info purposes
			show-deps		Look up the related dependencies for a given pkg
			build-model		Run a continuous loop to simulate a build order until the pkgs aren't satisfied (modifies the model field)

(Example)
			~/.styrene.conf		db = /tmp/ps.db
						kojitag = dist-f13-updates
						release = 13
						binrepo = updates
						srcrepo = updates-source

The Web Part Of Styrene

[random] 2x Off-Topic Ideas

So Google employee’s are required to take a break from work and experiment with anything they’re interested in (projects like gmail arose from this!). That was just a random side-note which may or may not be related to what’s below. This will probably be one of my more random posts on this blog.

RSA Animate – Drive: The surprising truth about what motivates us
(Full Version)

Anyway, here’s some tmp stuff I don’t really need but I guess this can be considered a backup, who knows…

rm -fv /tmp/f15.*.list
(curl -s 'http://git.fedorahosted.org/git/?p=arm.git;a=tree;f=stage3/armv7hl/SRPMS;hb=refs/heads/armv7hl' | tr -d '\t\r\n' | awk '{ gsub(/<tr/, "\n<tr"); gsub(/<\/tr/, "\n"); print; }' | grep -i '<td class="mode">' | sed -e 's/<[^>]*>/ /g' | awk '{ print $3 }') | tee /tmp/f15.git.src.list
(curl -s 'http://ausil.us/fedora-arm/mock/' | sed -e 's/<[^>]*>/ /g' | awk '{ print $1 }' | grep -i '.*/$' | while read line ; do test=`curl -s "http://ausil.us/fedora-arm/mock/$line" | sed -e 's/<[^>]*>/ /g' | awk '{ print $1 }' | grep -i '\.src\.rpm$'` ; if [ "$test" != "" ] ; then echo "$test" ; fi ; done) | tee /tmp/f15.dg.src.list
(curl -s 'http://ausil.us/fedora-arm/SRPMS/' | sed -e 's/<[^>]*>/ /g' | awk '{ print $1 }' | grep -i '\.src.rpm$') | tee /tmp/f15.dg.alt.list
(curl -s 'http://ausil.us/fedora-arm/noarch/' | sed -e 's/<[^>]*>/ /g' | awk '{ print $1 }' | grep -i '\.noarch.rpm$') | tee /tmp/f15.dg.noar.list
gunzip < /home/jchiappetta/Desktop/arm/styrene/misc-files/f15.db.gz > /tmp/f15.db
(cat /tmp/f15.*.list | sed -e 's/^\([^\.]*-[0-9]*\.[0-9]*\)[0-9\.]*.*$/\1*/g' | ./styrene mark-pkg --state=1) | tee /tmp/f15.mark.log

<?php
	function webs($urls)
	{
		$fobj = fopen($urls, "r");
		$page = "";
		while ($fobj && !feof($fobj))
		{
			$page .= fgets($fobj, 4096);
		}
		fclose($fobj);
		
		$page = str_replace("\t", "", $page);
		$page = str_replace("\r", "", $page);
		$page = str_replace("\n", "", $page);
		$page = preg_replace("/>[ ]*</", "><", $page);
		
		return $page;
	}
	
	$urls = $_GET["page"];
	$pref = preg_replace("/^http:\/\//", "", $urls);
	$urls = ("http://".$pref);
	$root = ("http://".preg_replace("/\/.*$/", "", $pref));
	
	$dsts = webs($urls);
	$dsts = preg_replace("/href=(['\"])\//", "href=\\1".$root."/", $dsts);
	$dsts = str_replace("</li></ul></td></tr></table>", "</li></ul></td></tr></table>\n", $dsts);
	
	$dstl = explode("\n", $dsts);
	
	if (count($dstl) > 1)
	{
		$rsss = webs("http://reddit.com/.rss");
		$rsss = str_replace("<item", "\n<item", $rsss);
		$rsss = str_replace("</item", "</item\n", $rsss);
		
		$rssl = explode("\n", $rsss);
		
		foreach ($rssl as $line)
		{
			if (preg_match("/^<item.*$/", $line))
			{
				$head = $line;
				$head = preg_replace("/^.*<title>/", "", $head);
				$head = preg_replace("/<\/title>.*$/", "", $head);
				
				$desc = $line;
				$desc = preg_replace("/^.*<description>/", "", $desc);
				$desc = preg_replace("/<\/description>.*$/", "", $desc);
				
				$dstl[0] .= ("<h2><span class=\"mw-headline\">".$head."</span></h2>");
				
				$desc = str_replace("&lt;", "<", $desc);
				$desc = str_replace("&gt;", ">", $desc);
				$desc = str_replace("&quot;", "\"", $desc);
				
				$dstl[0] .= $desc;
			}
		}
		
		echo $dstl[0].$dstl[1]."\n";
	}
?>

[random] 2x Off-Topic Ideas

[final] Patching The Koji Hosts Page

So you edit this file

/usr/share/koji-web/scripts/hosts.chtml

change the displaying if-block

    #if $len($hosts) > 0
      #for $host in $hosts
        #if (($time.time() - $time.mktime($time.strptime($util.formatTime($host.last_update),'%Y-%m-%d %H:%M:%S'))) < (15 * 60)):
          #continue
        #end if
        <tr class="$util.rowToggle($self)">
          <td>$host.id</td>
          <td><a href="hostinfo?hostID=$host.id"><b><font color="red">$host.name</font></b></a></td>
          <td>$host.arches</td>
          <td class="$str($bool($host.enabled)).lower()">#if $host.enabled then $util.imageTag('yes') else $util.imageTag('no')#</td>
          <td class="$str($bool($host.ready)).lower()">#if $host.ready then $util.imageTag('yes') else $util.imageTag('no')#</td>
          <td>$util.formatTime($host.last_update)</td>
        </tr>
      #end for
      #for $host in $hosts
        #if (($time.time() - $time.mktime($time.strptime($util.formatTime($host.last_update),'%Y-%m-%d %H:%M:%S'))) >= (15 * 60)):
          #continue
        #end if
        <tr class="$util.rowToggle($self)">
          <td>$host.id</td>
          <td><a href="hostinfo?hostID=$host.id"><b><font color="green">$host.name</font></b></a></td>
          <td>$host.arches</td>
          <td class="$str($bool($host.enabled)).lower()">#if $host.enabled then $util.imageTag('yes') else $util.imageTag('no')#</td>
          <td class="$str($bool($host.ready)).lower()">#if $host.ready then $util.imageTag('yes') else $util.imageTag('no')#</td>
          <td>$util.formatTime($host.last_update)</td>
        </tr>
      #end for
    #else

– OR THE HARDER WAY –

edit

/usr/share/koji-web/scripts/includes/header.chtml

add this line right before the ending </head> tag

#include "includes/hosts.js"

and you create a file in the same directory named hosts.js with

<script>
	var hoststat = 0;
	
	function conmonth(monthstr)
	{
		var x, l, m;
		var monthary = ["January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"];
		
		l = monthary.length;
		m = monthstr.length;
		
		for (x = 0; x < l; ++x)
		{
			if (monthary[x].substring(0, m) == monthstr)
			{
				return (x + "");
			}
		}
		
		return (0 + "");
	}
	
	function procdown(rowdlist)
	{
		var x, l = rowdlist.length, z;
		var coldlist, objclist;
		var locldate, loclutct;
		var hostdate, hostutct;
		var logiinfo = document.getElementById("loginInfo");
		var dateobjc = logiinfo.innerHTML.replace(/[\t\r\n]/g, "").match(/^.*[^ ]+ ([^ ]+) ([^ ]+) ([^ ]+) ([^ ]+) UTC.*$/);
		
		locldate = (dateobjc[3] + ":" + conmonth(dateobjc[2]) + ":" + dateobjc[1] + ":" + dateobjc[4]).split(":");
		loclutct = new Date(locldate[0], locldate[1], locldate[2], locldate[3], locldate[4], locldate[5]);
		
		for (x = 0; x < l; ++x)
		{
			coldlist = rowdlist[x].getElementsByTagName("td");
			
			if (coldlist.length < 6)
			{
				continue;
			}
			
			hostdate = coldlist[5].innerHTML.replace(/^[^0-9]*/g, "").replace(/[^0-9]*$/g, "").replace(/[^0-9]/g, ":").split(":");
			
			if (hostdate.length < 6)
			{
				continue;
			}
			
			objclist = coldlist[1].getElementsByTagName("a");
			hostutct = new Date(hostdate[0], parseInt(hostdate[1].replace(/^[^1-9]*/g, "")) - 1, hostdate[2], hostdate[3], hostdate[4], hostdate[5]);
			
			if (objclist.length > 0)
			{
				if ((hostutct.getTime() / 1000) <= ((loclutct.getTime() / 1000) - (15 * 60)))
				{
					objclist[0].innerHTML = ("<b><font color=\"red\">" + objclist[0].innerHTML + "</font></b>");
				}
				
				else
				{
					objclist[0].innerHTML = ("<b><font color=\"green\">" + objclist[0].innerHTML + "</font></b>");
				}
			}
		}
	}
	
	function redotabl()
	{
		var x, l;
		var downlist = [], rowdlist = ["row-odd", "row-even"];
		var tabllist = document.getElementsByClassName("data-list");
		var tablrowd;
		
		if (tabllist.length > 0)
		{
			l = tabllist[0].rows.length;
			
			for (x = 0; x < l; ++x)
			{
				if (tabllist[0].rows[x].innerHTML.replace(/[\t\r\n]/g, "").match(/^.*font.*color.*red.*$/))
				{
					downlist.unshift([x, tabllist[0].rows[x].innerHTML]);
				}
			}
			
			l = downlist.length;
			
			for (x = 0; x < l; ++x)
			{
				tabllist[0].deleteRow(downlist[x][0]);
			}
			
			for (x = 0; x < l; ++x)
			{
				tablrowd = tabllist[0].insertRow(3);
				tablrowd.innerHTML = downlist[x][1];
			}
			
			l = tabllist[0].rows.length;
			
			for (x = 3; x < l; ++x)
			{
				tabllist[0].rows[x].className = rowdlist[(x - 3) % 2];
			}
		}
	}
	
	function hostloop()
	{
		var oddrlist = document.getElementsByClassName("row-odd");
		var evenlist = document.getElementsByClassName("row-even");
		
		if ((oddrlist.length > 0) || (evenlist.length > 0))
		{
			procdown(oddrlist);
			procdown(evenlist);
			redotabl();
			
			hoststat = 1;
		}
		
		if (hoststat == 0)
		{
			setTimeout("hostloop();", 0.5 * 1000);
		}
	}
	
	if (location.href.match(/^.*hosts.*$/))
	{
		hostloop();
	}
</script>

[final] Patching The Koji Hosts Page

[arm-koji] Modified Hosts Page In PHP

So the other day, I published some JS code to change the look of the ARM-Koji Hosts page. It assumed that each client’s time was set to Eastern time zone settings (really very bad practice!). Since I have access to a PHP-based Apache server, I wanted to see how many lines it took me to implement the same.

<?php
	# get the koji host-status web page
	$u = "http://arm.koji.fedoraproject.org/koji/hosts";
	$hostpage = file_get_contents($u);
	$hostpage = str_replace("\t", "", $hostpage);
	$hostpage = str_replace("\r", "", $hostpage);
	$hostpage = str_replace("\n", "", $hostpage);
	# prep the page to be processed for host names and times
	$tabldata = $hostpage;
	$tabldata = str_replace("<tr", "\n<tr", $tabldata);
	$tabldata = str_replace("</tr>", "</tr>\n", $tabldata);
	$tabldata = explode("\n", $tabldata);
	# store the host names and times in a new list
	$hostlist = array();
	$begindex = -1;
	for ($x = 0; $x < count($tabldata); $x += 1)
	{
		if (preg_match("/^.*<a href=\"hostinfo\?hostID=([0-9]+)\">([^<]+)<\/a>.*>([0-9]+-[0-9]+-[0-9]+ [0-9]+:[0-9]+:[0-9]+).*$/", $tabldata[$x], $machlist))
		{
			$hosttime = preg_replace("/[^0-9]/", "", $machlist[3]);
			$templist = array(0, $machlist[1], $machlist[2], $machlist[3], $hosttime);
			array_push($hostlist, $templist);
		}
		if (preg_match("/^.*<a href=\"hostinfo\?hostID=[0-9]+\">[^<]+<\/a>.*$/", $tabldata[$x]))
		{
			$begindex = $x;
			$tabldata[$x] = "";
		}
	}
	# loop thru the list and mark/sort the ones that up/down
	$timesecs = time();
	$utcsdate = gmdate("YmdHis", $timesecs);
	$utcstime = (intval($utcsdate) - (15 * 60));
	for ($x = 0; $x < count($hostlist); $x += 1)
	{
		if (intval($hostlist[$x][4]) <= $utcstime)
		{
			$hostlist[$x][0] = 0;
			$hostlist[$x][2] = ("<b><font color='red'>".$hostlist[$x][2]."</font></b>");
		}
		else
		{
			$hostlist[$x][0] = 1;
			$hostlist[$x][2] = ("<b><font color='green'>".$hostlist[$x][2]."</font></b>");
		}
	}
	sort($hostlist);
	$headlist = array("row-odd", "row-even");
	for ($x = 0; $x < count($hostlist); $x += 1)
	{
		$c = $headlist[$x % 2];
		$i = $hostlist[$x][1];
		$n = $hostlist[$x][2];
		$t = $hostlist[$x][3];
		$tabldata[$begindex] .= ("<tr class='".$c."'><td>".$i."</td><td><a href='".$u."/hostinfo?hostID=".$i."'>".$n."</a></td><td>.</td><td>.</td><td>.</td><td>".$t."</td></tr>\n");
	}
	echo (join("\n", $tabldata)."\n");
?>


[arm-koji] Modified Hosts Page In PHP

[quick] Preparing For The F15 Build Process

So before we can run styrene for the F15 build process, we first must load/initialize it with a seed package set. This package set has to be manually built by the Fedora-ARM members. Since we are storing this package set in git, I needed a way to retrieve a list of what is currently built and load it into styrene. Here is the command below I used to do so:

curl -s 'http://git.fedorahosted.org/git/?p=arm.git;a=tree;f=stage3/armv7hl/SRPMS;hb=refs/heads/armv7hl' | tr -d '\t\r\n' | awk '{ gsub(/<tr/, "\n<tr"); gsub(/<\/tr/, "\n"); print; }' | grep -i '<td class="mode">' | sed -e 's/<[^>]*>/ /g' | awk '{ print $3 }'
[quick] Preparing For The F15 Build Process

Hightlighting Downed Hosts In JS

So if you visit

http://arm.koji.fedoraproject.org/koji/hosts

And you enter this into your URL bar

javascript:
function custom_pad(n)
{
	if (n < 10)
	{
		return ("0" + n);
	}
	return ("" + n);
}
function custom_chg(k, l)
{
	var x, indxlist = [], rowdlist = [];
	for (x = 0; x < l.length; ++x)
	{
		var t = new Date();
		var s = parseInt(t.getTime() / 1000);
		var r = new Date((s + (4 * 60 * 60) - (30 * 60)) * 1000);
		var q = (r.getFullYear()+""+custom_pad(r.getMonth() + 1)+""+custom_pad(r.getDate())+""+custom_pad(r.getHours())+""+custom_pad(r.getMinutes())+""+custom_pad(r.getSeconds()));
		var c = l[x].getElementsByTagName("td");
		if (c.length < 6)
		{
			continue;
		}
		var d = c[5].innerHTML.replace(/[^0-9]/g, "");
		var u = "green";
		if (parseInt(d) <= parseInt(q))
		{
			u = "red";
		}
		var o = c[1].getElementsByTagName("a");
		for (var y = 0; y < o.length; ++y)
		{
			var n = o[y].innerHTML.replace(/<[^>]+>/g, "");
			o[y].innerHTML = ("<b><font color='" + u + "'>" + n + "</font></b>");
			o[y].href = ("http://arm.koji.fedoraproject.org/koji/" + o[y].href.replace(/^.*\//g, ""));
		}
		if (u == "red")
		{
			indxlist.push(x);
			rowdlist.push(l[x].innerHTML);
		}
	}
	for (x = (indxlist.length - 1); x > -1; --x)
	{
		k.deleteRow(indxlist[x]);
	}
	for (x = (rowdlist.length - 1); x > -1; --x)
	{
		var v = k.insertRow(1);
		v.innerHTML = rowdlist[x];
	}
	var rowdtype = ["row-odd", "row-even"];
	for (x = 1; x < l.length; ++x)
	{
		l[x].className = rowdtype[(x - 1) % 2];
	}
}
if (location.href.match(/^.*(fedoraproject.org|hongkong).*hosts.*/))
{
	var h, i, j;
	h = null;
	while (!h)
	{
		h = document.getElementById("header");
	}
	h.parentNode.removeChild(h);
	h = null;
	while (!h)
	{
		h = document.getElementById("mainNav");
	}
	h.parentNode.removeChild(h);
	h = null;
	while (!h)
	{
		h = document.getElementById("loginInfo");
	}
	h.parentNode.removeChild(h);
	i = null;
	while (!i)
	{
		i = document.getElementsByTagName("h4");
	}
	i[0].parentNode.removeChild(i[0]);
	i = null; j = null;
	while (!i && !j)
	{
		i = document.getElementsByClassName("data-list");
		j = document.getElementsByTagName("tr");
	}
	i[0].deleteRow(0);i[0].deleteRow(0);
	custom_chg(i[0], j);
}

You get a coloured list of downed hosts!

Hightlighting Downed Hosts In JS

Dual File Compare in py

#!/usr/bin/python
import os
import sys

filelist = sys.argv[1:]
fileleng = len(filelist)

maxlnumb = 0
linelist = []

for filename in filelist:
	fobjitem = file(filename)
	fobjdata = fobjitem.readlines()
	fobjitem.close()
	
	linenumb = len(str(len(fobjdata)))
	maxlnumb = max(maxlnumb, linenumb)
	linelist.append(fobjdata)

spacereq = (maxlnumb + 1 + 4)
maxsline = ((int(os.environ["COLUMNS"]) - (spacereq * fileleng)) / fileleng)
x = 1

while (1):
	quitflag = 1
	linenumb = str(x)
	outpstri = ""
	
	while (len(linenumb) < maxlnumb):
		linenumb = ("0" + linenumb)
	
	for lineitem in linelist:
		linestri = ""
		
		if (len(lineitem) > 0):
			linestri = lineitem.pop(0)
			quitflag = 0
		
		linestri = linestri.replace("\t", "    ")
		linestri = linestri[0:maxsline].rstrip()
		
		while (len(linestri) < maxsline):
			linestri += " "
		
		if (outpstri != ""):
			outpstri += " | "
		
		outpstri += (linenumb + ":" + linestri)
	
	if (quitflag == 1):
		break
	
	print(outpstri)
	x += 1

Dual File Compare in py

[arm-team] Freeing Possibly Stuck Tasks

Since I’ve finally been experimenting with mass queue builds, there have been tasks and builders which have become “stuck” over a long period of time.

To get your latest tasks:

arm-koji list-tasks --mine | grep -i '.*open.*noarch.* build .*' | awk '{ print $1 }' > tasks.log

To free tasks from their builders that were created more than 2 days ago:

cat tasks.log | while read tasknum ; do taskday=`arm-koji taskinfo "$tasknum" | grep -i '^created' | awk '{ print $4 }'` ; today=`date "+%d"` ; let today="$today - 2" ; if [ $taskday -le $today ] ; then echo "beg [$tasknum]" ; arm-koji free-task "$tasknum" ; echo "end [$tasknum]" ; fi ; sleep 1 ; done

[arm-team] Freeing Possibly Stuck Tasks

Finally Getting To Really Test Styrene!

So after much scripting, I am finally getting to test and see if this thing actually works. It is currently attempting to automate the building of 2000+ Fedora 13 Update packages. I am completely aware and fully expecting there to be many packages that fail requiring manual, user intervention. I am hoping tho, however, that this tool helps to clear a bulk of the packages out of the way for our team. In addition, I will continue working on the visual component of it hopefully allowing our team to manually edit and customize builds. Maybe in the future this tool can attempt to automatically learn what specific fixes were applied to past packages and auto-apply them to future ones, who knows. Anyway, if you want to track the builds of any of our team members, you can run the following bash script below.

$ ./bcomp.sh fossjon 20110628

name:[liferea-1.6.5-1.fc13] date:[20110628]
name:[gwibber-3.0.0.1-2.fc13] date:[20110628]
name:[doxygen-1.7.3-1.fc13] date:[20110628]
name:[dovecot-1.2.17-1.fc13] date:[20110628]
name:[cups-1.4.6-1.fc13] date:[20110628]
name:[bash-completion-1.2-4.fc13] date:[20110628]
name:[NetworkManager-0.8.4-1.fc13] date:[20110628]
name:[dhcp-4.1.2-4.ESV.R2.fc13] date:[20110628]
...


#!/bin/bash
if [ "$1" == "" ]
then
	echo "Usage: $0 <account name> [start date]"
	exit 0
fi
x=0
s=0
if [ "$2" != "" ]
then
	s="$2"
fi
while true
do
	o=`curl -s "http://arm.koji.fedoraproject.org/koji/builds?userID=$1&order=-build_id&state=1&start=${x}" | tr -d '\t\r\n' | awk '{ gsub(/<tr/, "\n<tr"); print; }' | grep -i 'buildinfo?buildID=[0-9]*'`
	if [ "$o" == "" ]
	then
		break
	fi
	echo "$o" | while read l
	do
		p=`echo "$l" | sed -e 's@.*<td><a href="buildinfo?buildID=[0-9][0-9]*">\([^<][^<]*\)</a></td>.*@\1@g'`
		d=`echo "$l" | sed -e 's@.*<td>\([0-9][0-9][0-9][0-9]\)-\([0-9][0-9]\)-\([0-9][0-9]\) [0-9][0-9]:[0-9][0-9]:[0-9][0-9]</td>.*@\1\2\3@g'`
		q="name:[$p] date:[$d]"
		if [ $d -ge $s ]
		then
			echo "$q"
		fi
	done
	let x="$x + 50"
done

Finally Getting To Really Test Styrene!

Python Function Grep In Python

I am replacing this post (which was just normal grep in Python) with a script that will look for a search term and print out the last py method (def) it was seen in.

#!/usr/bin/python

import re
import sys

linenumb = 1
srchterm = ""
srchfunc = ""

lastfnum = 0
lastfunc = ""

if (len(sys.argv) < 2):
	print("Usage: %s <regex search term> [<function name>]" % (sys.argv[0]))
	sys.exit(0)

if (len(sys.argv) >= 2):
	srchterm = sys.argv[1]

if (len(sys.argv) >= 3):
	srchfunc = sys.argv[2]

while (1):
	lineread = sys.stdin.readline()
	
	if (not lineread):
		break
	
	lineread = lineread.rstrip()
	
	if (re.match("^def .*$", lineread)):
		lastfnum = linenumb
		lastfunc = lineread
	
	if (re.match("^[\t ]+.*" + srchterm + ".*$", lineread)):
		if ((lastfunc == "") or (re.match("^def " + srchfunc + ".*$", lastfunc))):
			if (lastfunc != ""):
				print("[*] [%d] %s" % (lastfnum, lastfunc))
				
				lastfnum = 0
				lastfunc = ""
			
			print("[-] [%d] %s" % (linenumb, lineread))
	
	linenumb += 1

Python Function Grep In Python

Remote IRC Viewer In Py

Here is a script (which probably contains bugs still) that allows one to host an “IRC like proxy” server in which multiple IRC clients can connect to one IRC server. Basically, the IRC session is simply relayed to all clients while the server maintains just one IRC connection. Anyway, the code is below in case you would like to play around with it.

import os
import random
import re
import select
import stat
import socket
import sys
import time


class loccom:
    def rand(self, numb):
        outp = ""
        
        for x in range(0, numb):
            outp += random.choice("0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz")
        
        return outp
    
    def clrf(self):
        try:
            fobj = open(self.file, "w")
            fobj.write("")
            fobj.close()
        
        except:
            pass
    
    def tmpc(self):
        dlst = os.listdir(self.fold)
        
        for item in dlst:
            if (not re.match("^%s.*$" % (self.pref), item)):
                continue
            
            sobj = os.stat("%s/%s" % (self.fold, item))
            
            psec = time.time()
            asec = sobj.st_atime
            msec = sobj.st_mtime
            stle = (60 * 60 * 1)
            
            if (((psec - asec) >= stle) or ((psec - msec) >= stle)):
                os.unlink("%s/%s" % (self.fold, item))
    
    def __init__(self):
        toke = self.rand(16)
        
        self.fold = "/tmp"
        self.pref = "loccom"
        self.file = ("%s/%s.%s.txt" % (self.fold, self.pref, toke))
        
        self.lock = 0
        self.wait = 0.1
        
        self.clrf()
        
        self.tmpc()
    
    def sloc(self):
        if (not os.path.isfile(self.file)):
            raise IOError("File does not exist.")
        
        while (self.lock != 0):
            time.sleep(self.wait)
        
        self.lock = 1
    
    def rloc(self):
        self.lock = 0
    
    def send(self, stri):
        self.sloc()
        
        try:
            fobj = open(self.file, "a")
            fobj.write(stri)
            fobj.close()
        
        except:
            pass
        
        self.rloc()
    
    def recv(self):
        self.sloc()
        
        try:
            fobj = open(self.file, "r")
            outp = fobj.read()
            fobj.close()
        
        except:
            outp = ""
        
        self.clrf()
        
        self.rloc()
        
        return outp


def resi(srch, repl, inpt):
    while (1):
        regx = re.match("^.*(%s).*$" % srch, inpt, re.I | re.S)
        
        if (not regx):
            break
        
        inpt = re.sub(regx.group(1), repl, inpt)
    
    return inpt


def ircpmake():
    ircphost = "irc.freenode.net"
    ircpport = 6667
    
    ircpobjc = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    
    ircpobjc.connect((ircphost, ircpport))
    
    return ircpobjc


statdata = ""

def readline(inptstri=""):
    global statdata
    
    outpstri = ""
    
    if (inptstri != ""):
        statdata += inptstri;print("setting["+inptstri+"]")
    
    elif (statdata != ""):
        statlist = statdata.split("\n")
        
        if (len(statlist) > 1):
            outpstri = (statlist[0] + "\n")
            tempdata = ""
            
            for x in range(1, len(statlist)):
                if (statlist[x] == ""):
                    continue
                
                if (tempdata != ""):
                    tempdata += "\n"
                
                tempdata += statlist[x]
            
            if (tempdata != ""):
                if (statdata[-1] == '\n'):
                    tempdata += "\n"
            
            statdata = tempdata
    
    return outpstri


# irc client object

ircpobjc = ircpmake()

ipaddr = ircpobjc.getsockname()
ipaddr = ipaddr[0]

imptlist = []
chanlist = []
sendlist = []
recvlist = []

# parent server object

hostname = ""
portnumb = 6060

sockobjc = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

sockobjc.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sockobjc.bind((hostname, portnumb))
sockobjc.listen(1)

while (1):
    sendremo = []
    recvremo = []
    (readlist, writelst, errolist) = select.select([sockobjc, ircpobjc], [], [], 0.1)
    
    if (sockobjc in readlist):
        (clieobjc, clieaddr) = sockobjc.accept()
        print("parent accepted [%s]" % (clieaddr[0]))
        
        servclie = loccom()
        clieserv = loccom()
        forkpidn = os.fork()
        
        if (forkpidn == 0):
            try:
                ircpobjc.close()
                sockobjc.close()
                sendlist = []
                recvlist = []
            
            except:
                pass
            
            break
        
        clieobjc.close()
        sendlist.append(servclie)
        recvlist.append(clieserv)
        
        for ircpitem in imptlist:
            servclie.send("FORK INIT\r\n%s\r\n" % (ircpitem))
        
        for chanitem in chanlist:
            servclie.send("FORK INIT\r\n%s JOIN %s\r\n" % (chanitem[0], chanitem[1]))
    
    if (ircpobjc in readlist):
        try:
            ircpdata = ircpobjc.recv(1024)
            
            if (ircpdata):
                readline(inptstri=ircpdata)
            
            else:
                ircpobjc = ircpmake()
        
        except:
            ircpobjc = ircpmake()
    
    while (1):
        ircpdata = readline()
        
        if (not ircpdata):
            break
        
        ircpline = ircpdata.strip()
        
        regxobjc = re.match("^:[^ ]+ 001 .+$", ircpline, re.I)
        
        if (regxobjc):
            imptlist.append(ircpline)
        
        regxobjc = re.match("^(:[^ ]*%s[^ ]*) join ([^ ]+).*$" % (ipaddr), ircpline, re.I)
        
        if (regxobjc):
            chanlist.append([regxobjc.group(1), regxobjc.group(2)])
        
        regxobjc = re.match("^(:[^ ]*%s[^ ]*) part ([^ ]+).*$" % (ipaddr), ircpline, re.I)
        
        if (regxobjc):
            remochan = []
            
            for chanitem in chanlist:
                if (chanitem[1] == regxobjc.group(2)):
                    remochan.append(chanitem)
            
            for remoitem in remochan:
                chanlist.remove(remoitem)
        
        print("ircp -> serv -> clie [%s]" % (ircpline))
        
        for senditem in sendlist:
            try:
                senditem.send(ircpdata)
            
            except:
                sendremo.append(senditem)
    
    x = 0
    
    for recvitem in recvlist:
        try:
            recvdata = recvitem.recv()
            
            if (recvdata):
                if (x > 0):
                    recvdata = resi("\nNICK ", "\nTICK ", "\n" + recvdata)
                
                print("clie -> serv -> ircp [%s]" % (recvdata.strip()))
                
                try:
                    ircpobjc.send(recvdata)
                
                except:
                    ircpobjc = ircpmake()
        
        except:
            recvremo.append(recvitem)
        
        x += 1
    
    for remoitem in sendremo:
        sendlist.remove(remoitem)
    
    for remoitem in recvremo:
        recvlist.remove(remoitem)

if (forkpidn != 0):
    sys.exit(0)

# child server object

print("child accepted [%s]" % (clieaddr[0]))

while (1):
    (readlist, writelst, errolist) = select.select([clieobjc], [], [], 0.1)
    
    if (clieobjc in readlist):
        cliedata = clieobjc.recv(1024)
        
        if (cliedata):
            cliedata = re.sub("^WHO ", "NAMES ", cliedata)
            print("user -> clie -> serv [%s]" % (cliedata.strip()))
            clieserv.send(cliedata)
    
    servdata = servclie.recv()
    
    if (servdata != ""):
        print("serv -> clie -> user [%s]" % (servdata.strip()))
        clieobjc.send(servdata)


Remote IRC Viewer In Py

Recursive GIT Tree Traverser In Python

Here’s a little script I wrote which will traverse through an online git repo and hopefully display the file tree. There’s probably a command to do this but I thought I’d try to write it anyway to test my recursion skills.

#!/usr/bin/python

import re
import urllib

def mt(tn):
    o = ""
    
    for x in range(0, tn):
        o += "    "
    
    return o

def rr(pd, tn):
    webobj = urllib.urlopen("http://git.fedorahosted.org/git/?p=arm.git;a=tree;f=" + pd)
    webstr = webobj.read()
    
    webstr = webstr.replace("\t", "")
    webstr = webstr.replace("\r", "")
    webstr = webstr.replace("\n", "")
    
    webstr = re.sub("<tr", "\n<tr", webstr)
    
    weblist = webstr.split("\n")
    
    dirlist = pd.split("/")
    
    while (len(dirlist) > 1):
        dirlist.pop(0)
    
    print(mt(tn) + dirlist[0] + "/")
    
    for webitem in weblist:
        if (re.match("^<tr", webitem)):
            webitem = re.sub("<[^>]*>", " ", webitem)
            webitem = webitem.strip()
            webitem = re.sub("  [ ]*", " ", webitem)
            
            itemlist = webitem.split(" ")
            
            if (len(itemlist) < 3):
                continue
            
            if (itemlist[2] == ".."):
                continue
            
            if (not re.match("^d.*", itemlist[0])):
                print(mt(tn + 1) + itemlist[2])
            
            else:
                rr(pd + "/" + itemlist[2], tn + 1)

rr("styrene", 0)

also to get the last check-in name/time:

#!/bin/bash

webpoutp=`curl -s 'http://git.fedorahosted.org/git/?p=arm.git;a=shortlog'`
webpoutp=`echo "$webpoutp" | tr -d '\t\r\n'`
webpoutp=`echo "$webpoutp" | awk '{ gsub(/<td/, "\n<td"); gsub(/<\/td/, "\n</td"); print; }'`
webpoutp=`echo "$webpoutp" | grep -i '^<td'`
webpoutp=`echo "$webpoutp" | head -n 2`
webpoutp=`echo "$webpoutp" | sed -e 's/<[^>]*>//g'`

fnaloutp=""

echo "$webpoutp" | while read lineread
do
	if [ "$fnaloutp" != "" ]
	then
		fnaloutp="${fnaloutp} - "
	fi
	
	fnaloutp="${fnaloutp}${lineread}"
	echo "Last git push: [$fnaloutp]" > /tmp/git.log
done

cat /tmp/git.log

Recursive GIT Tree Traverser In Python

[weekend] Real-Time BASH To HTTP Viewer

So for some reason, I decided to make python call up a bash shell while outputting everything from the terminal to a web page. I don’t think this is supposed to even work but with FF4 it seems to. Basically the python script starts by setting stdin to raw (canon) i/o mode. Then it forks a child process (bash) in which it can write to it’s stdin and read from it’s stdout/stderror. It then fires up a simple TCP socket HTTP server which sends any viewers the output of the shell’s stdout. It uses basic alarm interrupts to prevent blocking calls and basically acts as a service-in-the-middle. In English, this thing lets web viewers monitor your command line input/output. I don’t know why you would even want or need this but I guess it’s a cool hack/proof of concept.

#!/usr/bin/python

import os
import re
import signal
import socket
import sys
import time

import tty
import termios

def cleanstr(inptstri):
	backspce = (chr(8) + chr(27) + "[K")
	outpstri = inptstri
	
	outpstri = outpstri.replace(backspce, "^H")
	outpstri = outpstri.replace("&", "&amp;")
	outpstri = outpstri.replace("<", "&lt;")
	outpstri = outpstri.replace(">", "&gt;")
	outpstri = outpstri.replace(" ", "&nbsp;")
	outpstri = outpstri.replace("\t", "&nbsp;&nbsp;&nbsp;&nbsp;")
	outpstri = outpstri.replace("\r", "")
	outpstri = outpstri.replace("\n", "<br /> \n")
	
	fnalstri = ""
	
	for c in outpstri:
		d = ord(c)
		if ((d != 10) and (d < 32)):
			fnalstri += ("[%d]" % (d))
		else:
			fnalstri += c
	
	return fnalstri

# define a dummy alarm handler

def alarmhan(signanum, framenum):
	pass

signal.signal(signal.SIGALRM, alarmhan)

# spawn a new child shell

(shelcpid, shelcdes) = os.forkpty()

if (shelcpid == 0):
	os.execve("/bin/bash", [], os.environ)
	sys.exit(0)

# init some socket variables for our server

hostname = ""
portnumb = 8080
sockobjc = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

sockobjc.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sockobjc.bind((hostname, portnumb))
sockobjc.listen(5)
sockobjc.setblocking(0)

# set stdin i/o to raw (canon) mode

stdidesc = sys.stdin.fileno()
stdiolds = termios.tcgetattr(stdidesc)
tty.setraw(stdidesc)

# become a man-in-the-middle : user <-> us <-> shell -> us -> viewer

timeouta = 0.05
socklist = []

while (1):
	signal.setitimer(signal.ITIMER_REAL, timeouta)
	try:
		c = sys.stdin.read(1)
		if (ord(c) == 4):
			break
		if (ord(c) == 13):
			c = chr(10)
		os.write(shelcdes, c)
	except:
		pass
	signal.alarm(0)
	
	try:
		(clieconn, clieaddr) = sockobjc.accept()
		clieconn.send("HTTP/1.1 200 OK\r\nContent-Type: text/html\r\n\r\n")
		clieconn.send("<script>function stob() { window.scrollTo(0, document.body.scrollHeight); } setInterval('stob();', 250);</script> \n")
		for x in range(0, 64):
			clieconn.send(" &nbsp; <br /> \n ")
		socklist.append(clieconn)
	except:
		pass
	
	signal.setitimer(signal.ITIMER_REAL, timeouta)
	try:
		r = os.read(shelcdes, 1024)
		sys.stdout.write(r)
		sys.stdout.flush()
		l = []
		s = cleanstr(r)
		for sockitem in socklist:
			try:
				sockitem.send(s)
			except:
				l.append(sockitem)
		for i in l:
			socklist.remove(i)
	except:
		pass
	signal.alarm(0)

termios.tcsetattr(stdidesc, termios.TCSADRAIN, stdiolds)
print("")

[weekend] Real-Time BASH To HTTP Viewer