network scanner with perl + Tk

perl Tk ip scanner application
perl Tk ip scanner application

I enhanced my command line ip scanner by adding a Tk gui and adding a few more features.  Above, I am scanning a range of contiguous addresses on my home network.

It can also accept a list of hosts from a csv file.

list of hosts in csv format
list of hosts in csv format

Here are the results from the scan of the above list.

perl Tk ip scanner results using csv file input
perl Tk ip scanner results using csv file input

I am forced against my will to use Windows 10 on my PC at work.  I also lack admin privileges, so I cannot install a free ip scanner application.  I do, however, have ActivePerl installed on my box, and it has tons of modules so I can write my own utilities.

I have several classes of ip devices (routers, switches, battery chargers, etc.) that I want to monitor and have created individual csv files for each.  This simple script is highly effective at discovering devices on a subnet, or to give a quick glance at what devices on a list are online.

 
     1	#!/usr/bin/perl -w
     2	use strict;
     3	use Tkx;
     4	use Tkx::Scrolled;
     5	use Net::Ping;
     6	use Time::HiRes qw(gettimeofday);
       
     7	#main window
     8	my $mw = Tkx::widget->new(".");
     9	$mw->g_wm_title("nnmap network scanner");
    10	$mw->g_wm_geometry("800x600");
       
    11	#global vars
    12	my $ip_range = '';
    13	my $tdelay = 250; 
       
    14	#create ping object
    15	my $p = Net::Ping->new('icmp');
    16	#hi res time
    17	$p->hires();
       
    18	#a single ip address
    19	my $host;
       
    20	#up / down devices
    21	my @online_devices; 
    22	my @offline_devices; 
       
       
    23	#main content frame
    24	my $mfrm = $mw->new_ttk__frame(-padding => "5 10");
    25	$mfrm->g_grid(-column => 1, -row => 0, -sticky => "news");
    26	$mfrm->new_ttk__label(-text => "ip range")->g_grid(-column => 1, -row => 1, -sticky => "nw", -padx => 3, -pady => 1);
    27	$mfrm->new_ttk__entry(-textvariable => \$ip_range, -width => 24)->g_grid(-column => 1, -row => 2, -sticky => "nw", -padx => 3, -pady => 1);
    28	$mfrm->new_ttk__label(-text => "delay msec")->g_grid(-column => 1, -row => 3, -sticky => "nw", -padx => 3, -pady => 1);
    29	$mfrm->new_ttk__spinbox(-from => 1, -to => 900, -width => 5, -textvariable => \$tdelay)->g_grid(-column => 1, -row => 4, -sticky => "nw", -padx => 1, -pady => 2);
    30	$mfrm->new_ttk__button(-text => "scan", -command => sub {scan_net($ip_range);})->g_grid(-column => 1, -row => 5, -sticky => "nw", -padx => 1, -pady => 2);
       
    31	my $ta = $mfrm->new_tkx_Scrolled(
    32	    'text',
    33	    -scrollbars => "e",
    34	    -width => 70,
    35	    -height => 33,
    36	    -state => "normal"
    37	);
    38	$ta->g_grid(-column => 2, -row => 1, -sticky => "e", -padx => 5, -pady => 5, -rowspan => 50);
       
    39	#colors
    40	$ta->tag_configure("success", -foreground=>"white", -background=>"green");
    41	$ta->tag_configure("failure", -foreground=>"white", -background=>"red");
    42	#fonts
    43	$ta->tag_configure("lgtxt", -font =>"r18", -relief=>"raised", -underline=>1);
       
    44	sub scan_net{
    45		my $func_name = (caller(0))[3];
    46		print "Called $func_name on line ". __LINE__."\n";
    47		
    48		print "ip range: $ip_range\n";
       
    49		#parse ip range
    50		#only for class c or smaller subnets
    51		#?'s in regex account for a single ip address to scan
    52		$ip_range =~ /(\d+\.\d+\.\d+)\.(\d+)-?(\d+)?/;
    53		my $network = $1;
    54		my $start_ip = $2;
    55		my $end_ip = $3;
    56		
    57		print "end_ip: $end_ip\tstart ip: $start_ip\n";
       
    58		#account for just one ip to scan
    59		if($end_ip eq ""){
    60			$end_ip = $start_ip; 
    61		}#end if
       
    62		#convert time delay
    63		my $ping_delay = $tdelay * 0.001;
       
    64		#empty text area
    65		$ta->delete("1.0", "end");
       
    66		#get t0 for benchmark
    67		my $t0 = gettimeofday();
       
    68		#line counter
    69		my $line_n = 1; 
       
    70		for(my $i=$start_ip;$i<=$end_ip;$i++) { 71 $host = "$network.$i"; 72 print "scanning host: $host\n"; 73 #list context, returns duration 74 my ($ret, $dur, $ip) = $p->ping($host, $ping_delay);
    75			
    76			#format time
    77			$dur = sprintf("%.6f", $dur); 	
       
    78			#results
    79			if($ret){
    80				#print WHITE ON_GREEN "$host is up  latency: $dur seconds", RESET;
    81				#form response string
    82				my $response = "$host is up  $dur seconds";
    83				my $res_len = length($response); 
    84				if($res_len < 70){#fill up ta rows with color
    85					my $n_sp = 70 - $res_len; 
    86					for(my $x=0;$x<$n_sp;$x++){ 87 $response = $response." "; 88 }#end for add sp 89 }#end if 90 $response = $response."\n"; 91 #print "res_len: $res_len\n"; 92 #print "\n"; 93 $ta->insert("$line_n.0", $response);
    94				$ta->tag_add("success", "$line_n.0", "$line_n.0 lineend");
    95				push @online_devices, $host;
    96			}#end if
    97			else{
    98				my $response = "$host down";
    99				my $res_len = length($response); 
   100				if($res_len < 70){
   101					my $n_sp = 70 - $res_len; 
   102					for(my $x=0;$x<$n_sp;$x++){ 103 $response = $response." "; 104 }#end for add sp 105 }#end if 106 $response = $response."\n"; 107 #print "res_len: $res_len\n"; 108 #print WHITE ON_RED "$host down", RESET; 109 #print "\n"; 110 $ta->insert("$line_n.0", $response);
   111				$ta->tag_add("failure", "$line_n.0", "$line_n.0 lineend");
   112				push @offline_devices, $host;
   113			}#end else		
       
   114			#increment line number
   115			$line_n++;
   116		
   117		}#end for
       
   118		#results
   119		#benchmarking results
   120		my $t1 = gettimeofday();
   121		my $elapsed = $t1 - $t0; 
   122		$elapsed = sprintf("%6f", $elapsed);
   123		print "\ntime elapsed: $elapsed....\n";
   124		my $up_sz = @online_devices;
   125		my $down_sz = @offline_devices;
   126		print "$up_sz devices online\n$down_sz devices offline\n";
       
   127		#form results
   128		my $results = "Results\ntime elapsed: $elapsed\n\n$up_sz devices online\n$down_sz devices offline\n";
   129		
   130		#insert results
   131		$ta->insert("$line_n.0", $results);
   132		$ta->tag_add("lgtxt", "$line_n.0", "$line_n.0 lineend");
   133	}#end scan_net
       
   134	sub openCSV{
   135		my $func_name = (caller(0))[3];
   136		print "Called $func_name on line ". __LINE__."\n";
       
   137		my $fn = Tkx::tk___getOpenFile();
   138		print "FILE: $fn\n";
       
   139		#open csv file
   140		open HF, $fn, or die $!;
       
   141		#empty text area
   142		$ta->delete("1.0", "end");
       
   143		#ping delay
   144		my $ping_delay = $tdelay*0.001;
       
   145		#get t0 for benchmark
   146		my $t0 = gettimeofday();
       
   147		#line number counter
   148		my $line_n = 1;
       
   149		while(){
   150			my $ipdevice = $_;
   151			my @ipdev = split(',', $ipdevice); 
   152			my $description = $ipdev[1]; 
   153			$host = $ipdev[2]; 		
   154			$host =~ s/\s+//;
   155			$description =~ s/\n//;
   156			
   157			#list context, returns duration
   158			my ($ret, $dur, $ip) = $p->ping($host, $ping_delay);
       
   159			print "scanning host: $host\n";
   160			
   161			#format time
   162			$dur = sprintf("%.6f", $dur); 	
   163			
   164			my $host_desc = $host."\t\t".$description;
   165			
   166			#results
   167			if($ret){
   168				#print WHITE ON_GREEN "$host $description is up  latency: $dur seconds", RESET;
   169				#print "\n";
   170				my $response = "$host $description is up  $dur sec";
   171				my $res_len = length($response); 
   172				if($res_len < 70){#fill up ta rows with color
   173					my $n_sp = 70 - $res_len; 
   174					for(my $x=0;$x<$n_sp;$x++){ 175 $response = $response." "; 176 }#end for add sp 177 }#end if 178 $response = $response."\n"; 179 $ta->insert("$line_n.0", $response);
   180				$ta->tag_add("success", "$line_n.0", "$line_n.0 lineend");
   181				push @online_devices, $host_desc;
   182			}#end if
   183			else{
   184				#print WHITE ON_RED "$host $description down", RESET;
   185				#print "\n";
   186				push @offline_devices, $host_desc;
   187				my $response = "$host $description down";
   188				my $res_len = length($response); 
   189				if($res_len < 70){#fill up ta rows with color
   190					my $n_sp = 70 - $res_len; 
   191					for(my $x=0;$x<$n_sp;$x++){ 192 $response = $response." "; 193 }#end for add sp 194 }#end if 195 $response = $response."\n"; 196 $ta->insert("$line_n.0", $response);
   197				$ta->tag_add("failure", "$line_n.0", "$line_n.0 lineend");
   198			}#end else
       
   199			$line_n++;
   200		}#end while
   201		
   202		close HF; 
       
   203		#results
   204		#benchmarking results
   205		my $t1 = gettimeofday();
   206		my $elapsed = $t1 - $t0; 
   207		$elapsed = sprintf("%6f", $elapsed);
   208		print "\ntime elapsed: $elapsed....\n";
   209		my $up_sz = @online_devices;
   210		my $down_sz = @offline_devices;
   211		print "$up_sz devices online\n$down_sz devices offline\n";
       
   212		#form results
   213		my $results = "Results\ntime elapsed: $elapsed\n\n$up_sz devices online\n$down_sz devices offline\n";
   214		
   215		#insert results
   216		$ta->insert("$line_n.0", $results);
   217		$ta->tag_add("lgtxt", "$line_n.0", "$line_n.0 lineend");
   218	}#end openCSV
       
   219	#menu
   220	Tkx::option_add("*tearOff", 0);
   221	$mw->configure(-menu => mk_menu($mw));
   222	sub mk_menu{
   223		my $mw = shift;
   224		my $menu = $mw->new_menu();
   225		my $file = $menu->new_menu( -tearoff => 0);
   226		$menu->add_cascade(
   227			-label => "File", 
   228			-underline => 0, 
   229			-menu => $file
   230		);
   231		$file->add_command(
   232			-label => "Open CSV File", 
   233			-underline => 0, 
   234			-command => sub {openCSV();}
   235		);
   236	          $file->add_command(
   237	              -label   => "Exit",
   238	              -underline => 1,
   239	              -command => [\&Tkx::destroy, $mw],
   240	          );
   241		return $menu;
   242	}#end mk_menu
       
   243	Tkx::MainLoop();

This is one of those utilities of mine that will probably continue to grow in complexity.

Fret Not! guitar fretboard tutor program

I got a nice acoustic guitar for Christmas.  I haven’t played in years, really since my eldest son was born 17 years ago.  At the time I was really into Mississippi River Delta Blues, and finger-style playing in alternate tunings.  I mostly read tablature.  I knew some pentatonic shapes to play up and down the fretboard, but not the actual note names I was playing.  This time around,  I want to learn how to read notes on a staff, and that means knowing the fretboard unconsciously.  To help,  I wrote a very simple perl script that I call ‘Fret Not!’

Fret Not! guitar fretboard tutoring perl script
Fret Not! guitar fretboard tutoring perl script teaches notes on the fretboard

It uses the ascii art program figlet for some simple command line graphics.  It basically is an endless loop asking you to name the note on a particular string at a particular fret.  It really is helping me to memorize the note positions on a fretboard.

 
     1	#!/usr/bin/perl -w
     2	#use Data::Dumper;
     3	use strict;
     4	$| = 1;	#flush output
       
     5	#display heading
     6	print "#####################################################################################\n";
     7	system('figlet "FretNot!" -c');
     8	print "#####################################################################################\n";
       
     9	#guitar string arrays
    10	my @e1 = ('E','F','F#','G','G#','A','A#','B','C','C#','D','D#','E');
    11	my @a = ('A','A#','B','C','C#','D','D#','E','F','F#','G','G#','A');
    12	my @d = ('D','D#','E','F','F#','G','G#','A','A#','B','C','C#','D');
    13	my @g = ('G','G#','A','A#','B','C','C#','D','D#','E','F','F#','G');
    14	my @b = ('B','C','C#','D','D#','E','F','F#','G','G#','A','A#','B');
    15	my @e2 = ('E','F','F#','G','G#','A','A#','B','C','C#','D','D#','E');
       
    16	#hashref of all strings
    17	my $stringref = [\@e1, \@a, \@d, \@g, \@b, \@e2];
       
    18	#testing; worked
    19	#print Dumper($stringref);
       
    20	for(;;){
    21		my $str_num = 1 + int rand(6);
    22		my $fret_num = 1 + int rand(12);
    23		print "What is the note on string $str_num at fret $fret_num ?:";
    24		chomp(my $answer = );
    25		$str_num-=1;
    26		#$fret_num-=1;
    27		my $note = ${$stringref}[$str_num][$fret_num]; 
    28		print "note: $note\tans: $answer\n";
       
    29		if($note eq $answer){
    30			system('figlet "correct!"');
    31		}#end if
    32		else{
    33			system('figlet "LOSER!"');
    34		}#end else
    35	}#end for

As you can see, I created a data structure in the form of a reference of array references.  It randomly chooses a string at a particular fret.  A mere 35 lines of perl, but very effective for it’s purpose.  To use this,  you will have to install figlet on your linux box.

ssh to a raspberry pi zero over usb

connecting to a rapsberry pi zero via usb
connecting to a raspsberry pi zero via usb

I am working on a data logging project based on the raspberry pi zero.  NOT a pi zero w,  just the pi zero.  Adding software packages to the pi zero is a little more difficult, because it has no wifi or ethernet connection.  I have been taking the sd card out ant putting it in a pi 3b+ to get packages that I need then putting it back in the pi zero.  I have tried and failed to use minicom and zmodem to transfer deb files over UART.

I found this great tutorial that explains how to ssh into a pi via the usb port on the pi.  I did everything it said and was still not able to connect to the pi.  I did an ifconfig on the pi, and saw that it had a APIPA on it’s usb network interface like so:

usb network interface on pi zero
usb network interface on pi zero

I had a usb network interface appear on my laptop, but no ipv4 address:

usb network interface on ubuntu machine
usb network interface on ubuntu machine

I finally got it to work by issuing an ifconfig command to my laptop’s usb interface and assigning an IP address in the same class B subnet :

ifconfig on usb network interface APIPA
ifconfig on usb network interface APIPA

Another benefit to a standard pi zero having ip connectivity via usb is that you can use piscope for troubleshooting.  Here is piscope examining i2c frames polling an HTU21D relative humidity sensor:

examining an HTU21D with piscope on a pi zero with usb network interface
examining an HTU21D with piscope on a pi zero with usb network interface

dht22 indoor use only

In my empirical observations experimenting with a dht22 temperature / humidity sensor,  I have come to the conclusion that they are most definitely for INDOOR USE ONLY!  Any time I placed one outside, they quickly max out at 99.9% humidity,  and don’t recover until I dry them out, and place them indoors.  I have ordered a HTU21D-F sensor to replace the dht22 in my rf propagation / humidity experiment.

The dht22 does work pretty well indoors.  I took over 14K samples over a 30 hr period and got clean results.

dht22 sensor temperature and humidity graph
dht22 sensor temperature and humidity graph

The temperature is in red and the humidity in green.  For most of my data logging projects using a pi, I sore the results in CSV format.  I used perl’s CSV database functions to query results and create graphs.  This is the sql query I use on the to csv tables to get the above graph:


select data.index, data.temp, temp_humidity.humidity from data outer join temp_humidity on data.index = temp_humidity.index

piscope is totally AWESOME with perl!

this is a screenshot of piscope running on my laptop analyzing the gpio lines of a raspberry pi over IP that is running my 24-port battery load test analyzer
this is a screenshot of piscope running on my laptop analyzing the gpio lines of a raspberry pi over IP that is running my 24-port battery load test analyzer program

I would assume many makers are familiar with using a logic analyzer in conjunction with sigrok + pulseview.  I love these resources.  They allow you to analyze precisely what is happening on your digital IO pins on whatever microcontroller you are using whether it be arduino, raspberry pi, etc.  They can also analyze signals at the protocol level such as i2c and are so inexpensive every maker should be equipped with these tools.

Pulseview SPI
Pulseview SPI scan

I do not like to reinvent the wheel in most cases.  I wanted to use some dht22 temp / humidity sensors on an RF signal strength project I am still working on.  As I have mentioned in earlier posts, I chose to use the RPi::PIGPIO::Device::DHT22 module on cpan to read my sensor.  This required the pigpiod daemon to be running on the raspberry pi.  I am very impressed with PIGPIO.  It allows you to very easily read / write to a raspberry pi ‘s GPIO lines over TCP/IP.  Just think of the possibilities.

The author of PIGPIO also offers an incredible logic analyzer for the raspberry pi called piscope.  I may never use a standard logic analyzer on a pi ever again.  You can invoke piscope on any linux computer once you have installed it to analyze the gpio on a remote pi.

invoking piscope to monitor the gpio lines on a raspberry pi
invoking piscope to monitor the gpio lines on a raspberry pi

After this, launch piscope:

run piscope logic analyzer
run piscope logic analyzer

I am just beginning to experiment with piscope, but so far it is very user friendly.  This is a trace of the SDA and SCL lines on the pi reading an MCP9808 temperature sensor.

piscope logic analyzer zoomed in on the i2c lines of an MCP9808 temperature sensor
piscope logic analyzer zoomed in on the i2c lines of an MCP9808 temperature sensor

This trace was taken over the net.  I didn’t have to get out my logic analyzer and connect any test leads.  Here is a trace of a poll and response from a dht22 sensor connected to gpio 24.

piscope logic analyzer reading a dht22 via perl
piscope logic analyzer reading a dht22 via perl

You can see it go low, then the sensor sends its reading, and goes high again.  pigpiod is definitely a resource hog, but that is hardly a consideration for my uses of the pi in my projects.  I will definitely be incorporating piscope into my future projects.

piscope by default uses port 8888 on the pi you are monitoring.  Out of curiosity,  I scanned the incoming frames with tcpdump.

analyzing piscope frames on port 8888 using tcpdump
analyzing piscope frames on port 8888 using tcpdump

It sends a lot of traffic over the network.

piscope network traffic on linux mint's system monitor
piscope network traffic on linux mint’s system monitor

Here is a video of me launching piscope, and live traffic from a pi 3B+.

data logging rssi on a GE MDS9710 with perl, dht22, and raspberry pi

pi zero w data logging temperature, rssi, and humidity
pi zero w data logging temperature, rssi, and humidity

I work on 900MHz point to multi-point data radio systems.  Humidity can greatly affect the propagation of rf waves at this frequency.  I wanted to do some data logging and analysis of how moisture in the air affected the received signal strength indication over time.

The GE MDS 9710 P70 radio system has an analog voltage test point that corresponds to the rssi on it’s interface board.  I used a DHT22 to measure the temperature and humidity,  an MCP3004  10-bit analog to digital converter to read the rssi voltage, and a pi zero W for the brains.

arduino nano every and DHT22. Readings displayed on 16X2 LCD
arduino nano every and DHT22. Readings displayed on 16X2 LCD

I originally implemented this project with an arduino nano every.   It worked great.  I was going to log the data by connecting the nano every to a pi zero via UART while a screen session was running on the pi with the screen output (STDOUT) saved to a file .  This method would require a logic level converter which I did not have on hand at the time, so I decided to implement the whole project on my pi zero.

It is much easier to use a DHT22 sensor with arduino using adafruit’s unified sensor library.

adafruit unified sensor library for DHT22
adafruit unified sensor library for DHT22

Here is the source:

 
// REQUIRES the following Arduino libraries:
// - DHT Sensor Library: https://github.com/adafruit/DHT-sensor-library
// - Adafruit Unified Sensor Lib: https://github.com/adafruit/Adafruit_Sensor

#include "DHT.h"
#include 
#define DHTPIN 2     // Digital pin connected to the DHT sensor

// Uncomment whatever type you're using!
//#define DHTTYPE DHT11   // DHT 11
#define DHTTYPE DHT22   // DHT 22  (AM2302), AM2321
//#define DHTTYPE DHT21   // DHT 21 (AM2301)

DHT dht(DHTPIN, DHTTYPE);

const int rs = 12, en = 11, d4 = 10, d5 = 9, d6 = 8, d7 = 7;
LiquidCrystal lcd(rs, en, d4, d5, d6, d7);

//number of samples
int sn = 0;

void setup() {
  Serial.begin(9600);
  Serial1.begin(9600);
  // set up the LCD's number of columns and rows:
  lcd.begin(16, 2);
  // Print a message to the LCD.
  lcd.print("MDS RSSI Data   Logger");
  delay(2000);
  dht.begin();
}

void loop() {
  // Wait a few seconds between measurements.
  delay(2000);

  //increment sample number
  sn++;

  // Reading temperature or humidity takes about 250 milliseconds!
  // Sensor readings may also be up to 2 seconds 'old' (its a very slow sensor)
  float h = dht.readHumidity();
  // Read temperature as Fahrenheit (isFahrenheit = true)
  float f = dht.readTemperature(true);

  // Check if any reads failed and exit early (to try again).
  if (isnan(h) || isnan(f)) {
    Serial.println(F("Failed to read from DHT sensor!"));
    Serial1.println(F("Failed to read from DHT sensor!"));
    return;
  }

  //read rssi voltage
  int bv = analogRead(A0);
  float v = (5 * bv) / 1023.0;
  float rssi = v - 1.5;
  rssi = (rssi/0.8) * 20 - 120; 
  
  // Compute heat index in Fahrenheit (the default)
  float hif = dht.computeHeatIndex(f, h);

  //print to console
  Serial.print(F("Humidity: "));
  Serial.print(h);
  Serial.print(F("%  Temperature: "));
  Serial.print(f);
  Serial.print(F("°F  Heat index: "));
  Serial.print(hif);
  Serial.println(F("°F"));

  //print to pi
  Serial1.print("Humidity: ");
  Serial1.print(h);
  Serial1.print(F(" Temp: "));
  Serial1.print(f);
  Serial1.print("F RSSI: ");
  Serial1.print(rssi);
  Serial1.print("dBm");
  Serial1.print("\n");

  lcd.clear();
  lcd.print("Humidity:");
  lcd.print(h);
  lcd.print("%");
  lcd.setCursor(0,1);
  lcd.print(f);
  lcd.print("F ");
  //lcd.print(v);
  //lcd.print("V");
  lcd.print(rssi);
  lcd.print("dBm");
}

To use perl to interface with the DHT22 on my pi required the RPi::PIGPIO::Device::DHT22 module on cpan.  Before I could use this,  I needed to install PIGPIO and run it as sudo before executing my script.  PIGPIO is a VERY interesting library for controlling a pi’s gpio’s locally or remotely via socket, and I will hopefully find the time to explore more possibilities in the near future.  It is, however, somewhat of a resource hog.

pigpiod daemon cpu resources on a raspberry pi
pigpiod daemon cpu resources on a raspberry pi

Once all the modules are loaded on the pi and the PIGPIO daemon is fired up,  I just had to translate the ardino C code into perl.

 
#!/usr/bin/perl 
use strict;
require "/home/pi/gpio/MCP3004.pl";
use RPi::PIGPIO;
use RPi::PIGPIO::Device::DHT22;

=pod
loggs the temperature, humidity, and rssi of a
control point radio using a DHT22.  requires
PIGPIO daemon to be started first.  from home dir, 
cd/PIGPIO.  then, sudo pigpiod.  cd back into
gpio dir, and sudo perl CTC_datalogger.pl.  
=cut

my $pi = RPi::PIGPIO->connect('127.0.0.1');
my $dht22 = RPi::PIGPIO::Device::DHT22->new($pi,23);

#ADC channel
my @ch0 = (1,1,0,0,0);

#init ADC
init3004(3,4,5,6);

#open log file
open RD, ">", "ctc_data.log" or die $!;
#print table description
print RD "index,epoch,date,humidity,temperature,voltage,rssi\n";

my $i = 0;
for(;;){
	#formatted date string 
	my $time_str = `date`;
	print "$i\n";
	print $time_str;
	chomp($time_str);

	$dht22->trigger();
	my $deg_f = ($dht22->temperature * (9/5)) + 32;
	#$deg_f = sprintf("%.1f", $deg_f);
	$deg_f = int($deg_f);

	print "Temp: $deg_f \n";
	my $humidity = $dht22->humidity;
	$humidity = sprintf("%.1f", $humidity);
	print "Humidity: ".$dht22->humidity."%\n";
	int($deg_f);

	#take 5 samples from ADC and avg
	my $v_avg = 0;
	my $v_accum = 0;
	for(my $s=0;$s<5;$s++){
		my ($reading, $binval, $voltage ) = read3004(\@ch0, 50, 4.966);
		#print "binval: $binval\tvoltage: $voltage vdc\n";
		#print RD "$s\n$voltage\n";
		usleep(1000);

		$v_accum += $voltage; 
	}#end for
	$v_avg = $v_accum / 5;
	$v_avg = sprintf("%.2f", $v_avg);
	#print "avg: $v_avg\n";
	
	#calculate RSSI
	my $rssi = 0;
	$rssi = $v_avg - 1.5;
	$rssi = ($rssi / 0.8) * 20;
	$rssi -= 120;
	
	#format rssi
	$rssi = sprintf("%.2f", $rssi);
	print "rssi: $rssi dBm \n";
	print "-----------------------------------\n";

	#print to LCD
	`./lcd "Humidity: $humidity%     $deg_f F $rssi dBm"`;

	#print to log file
	my $log_str = '';
	my $uts = time;
	my $hr_time = localtime();
	$log_str = "$i,$uts,$hr_time,$humidity,$deg_f,$v_avg,$rssi\n";	
	select((select(RD), $|=1)[0]);
	print RD $log_str;

	sleep(2);
	
	$i++;
}#end for

close RD;

So far, it’s working great on the test bench.

pi zero rssi humidity data logger
pi zero rssi humidity data logger

Forgot to mention.  I used a DS3231 real time clock to keep time on the pi.  It works great and was easy to set up using this tutorial.

DS3231 RTC on pi zero
DS3231 RTC on pi zero

I will be deploying the data logger in the field this week and will post the results.  My bench test results are as follows.  Here is my humidity graph after nearly 9K samples.

dht22 humidity readings with pi zero w
dht22 humidity readings with pi zero w

And here is the rssi over the same dataset.  You can definitely see a higher rssi when the humidity is lower as expected.

rssi readings using raspberry pi
rssi readings using raspberry pi

perl + curl hyperlink parser

I go through seasons where I struggle to find ideas for projects that I am interested in.  Now is not one of those times.  I am finding it hard to make time to work on my ideas.  I like that.  I wanted to see how hard it would be to implement my own web crawler.  At the core of this idea is the ability to find all the hyperlinks in a given web page.  After some trial and error (not googling)  I have a pretty decent hyperlink finder.  My script involves getting the content using curl.  It then redirects the output of curl to a text file.  I use perl to parse the text file and a regular expression to locate the hyperlinks.  At first I was using anchor tags to find links.  It was somewhat effective, unless there were multiple links on a single  line of text.  I scrapped this approach because I used the ‘.+’ method of getting the inner contents of the anchor. It had the undesirable (but predictable) effect of mushing multiple links in a single line together.   I had more luck looking for the inner text of the href=”” attributes of the anchor tags.

Here is a hyperlink finder script.

 
#/usr/bin/perl -w
use strict;

my $url = $ARGV[0];

print "looking up $url...\n";

`curl $url > pl.txt`;

print "analyzing $url...\n";

my $wc = `wc  pl.txt`;

print $wc;

open XL, "pl.txt" or die $!;
my $nlink = 0;
while(){
	while($_ =~ /\shref="(\S+)"/g){
		$nlink++;
		print "$nlink\t$1\n";
	}#end if	
}#end while

print "num links on page: $nlink\n";

close XL;

Here is a sample output using my homepage.

hyperlink parser using cURL and perl regular expressions
hyperlink parser using cURL and perl regular expressions

experimenting with a DHT11 temperature & humidity sensor with perl and raspberry pi zero w

using perl and a pi zero to interface with a dht11 temperature and humidity sensor
using perl and a pi zero to interface with a dht11 temperature and humidity sensor

I personally think that the raspberry pi zero w is the best choice for data logging projects.  You can use any programming language,  file IO is siimple,  it’s easy to send your data to the cloud, and best of all: it is very inexpensive.  For example an adafruit feather M0 is $20, and you are restricted to programming in C.

I am developing a data logging project that will use a dht11 temperature + humitidy sensor to measure the effects of temperature and humidity on the RF propagation of a 900MHz radio system.  The dht11 will log the temp / humidity, and I will measure the RSSI (received signal strength indication)  with an MCP3004 analog to digital converter.  The rssi is rendered as a DC voltage; the higher the voltage, the greater the signal strength.  Frequencies at 900MHz are highly susceptible to moisture in the atmosphere and I want to analyze the effect of humidity on the rf path of this radio system .

Im in the beginning stages, and just getting started with the dht11.  I am using the  awesome RPi::DHT11 cpan module by Steve Bertrand.

#!/usr/bin/perl -w
use strict;
use RPi::DHT11;

my $env = RPi::DHT11->new(18);

for(my $i=0;$i<20;$i++){ print "sample $i\t"; my $temp = $env->temp('f');
	my $humidity = $env->humidity;

	print "temp: $temp\thumidity: $humidity \n";
	print "----------------------------------------------\n";

	sleep(2);
}#end for

Can’t get much easier. This script samples the dht11 every 2 seconds and prints the results to STDOUT.

dht11 perl script output
dht11 perl script output

When the pi drives the output line of the dht11 low for 20ms, it outputs a 40-bit data string that contains the readings (most of the time).

dht11 communications process
dht11 communications process

Here is a close up of the process using PulseView.

dht11 40-bit data
dht11 40-bit data

And here is a close up of the data above:

dht11 40-bit data close up
dht11 40-bit data close up

pine64 guinea pig habitat comfort control

pine64 and perl guinea pig habitat environment automation
pine64 and perl guinea pig habitat environment automation

We’ve moved our guinea pig Monster out to the garage.  He’s in no danger of overheating (trust me, he is spoiled rotten!) but it does get a little warm out there.  I used my pine64 to turn on a 20W fan when it rises above a chosen temperature.  Temperature samples are taken with an analog TMP36 sensor which is read by an MCP3004 analog to digital converter.  It also gets pitch dark with the lights out at night, so there is also a photo resistor that is read by the MCP3004 ADC.  When it drops below a certain threshold, it turns on an LED night light for Monster.  I think he likes it.  He has been laying in front of the fan.

Once the ADC reading is taken of the TMP36 sensor, this is how it is converted into degrees Fahrenheit.

my $temperature = ((($voltage_t * 1000) - 500) / 10) * 1.8 + 32;

I am running armbian on my pine64 and have written my own gpio control script in perl.  Here is the perl script that controls Monster’s fan and night light.

#!/usr/bin/perl 
use strict;
require "./MCP3004.pl";

gpio_enable(17, 'out');
gpio_enable(13, 'out');

#get thermostat arg
my $thermostat = $ARGV[0];
if($thermostat eq ''){
	$thermostat = 85;
}#end thermostat


#channels: first bit is set to single, not differential
#5 bits because the first is the start bit
my @ch0 = (1,1,0,0,0);
my @ch1 = (1,1,0,0,1);
my @ch2 = (1,1,0,1,0);
my @ch3 = (1,1,0,1,1);
#differential
my @ch01 = (1,0,0,0,0);

init3004(18,19,24,25);

#loop global variables
my $tmp10 = 0;
my $tavg = 0;
my $bin10 = 0;
my $binavg = 0;
my $fan_f = 0;
my $lt_f = 0;

for(;;){
	$tmp10 = 0;
	$tavg = 0;
	$bin10 = 0;
	$binavg = 0;

	for(my $n=0;$n<10;$n++){ #photoresisitor reading------------------------------------------- my ($reading, $binval, $voltage ) = read3004(\@ch0, 50, 5.01); print "binval: $binval\t$voltage vdc"; $bin10 += $binval; #print RD "$s\n$voltage\n"; #photocell end---------------------------------------------------- #tmp36 temperature reading---------------------------------------- print "\n---------------------------------------\n"; my ($reading_t, $binval_t, $voltage_t ) = read3004(\@ch1, 50, 5.01); #print "binval_tmp: $binval_t\tvoltage_tmp: $voltage_t vdc\n"; my $temperature = ((($voltage_t * 1000) - 500) / 10) * 1.8 + 32; $temperature = sprintf("%5.2f", $temperature); print "temperature: $temperature deg F\n"; #accumulate temp $tmp10 +=$temperature; print "---------------------------------------\n"; #tmp36 end-------------------------------------------------------- #0.5 pause between samples usleep(500000); }#end for 10 #calculate temperature averages $tavg = $tmp10 / 10; if($tavg > $thermostat){
		#turns fan on
		gpio_write(17, 1);
		$fan_f = 1;
	}#end if
	else{
		gpio_write(17, 0);
		$fan_f = 0;
	}#end else

	#calculate light intensity averages
	$binavg = $bin10 / 10;
	if($binavg > 610){
		#turns light on
		gpio_write(13, 0);
		$lt_f = 1;
	}#end if
	else{
		gpio_write(13, 1);
		$lt_f = 0;
	}#end else

	print "\n-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-\n";
	print "avg temp: $tavg stat:$fan_f\tavg light: $binavg stat:$lt_f";
	print "\n-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-*-\n";

	#wait 1 minute
	sleep(60);
}#end outer for

That’s a lot of processing power just to control a light and a fan.  I plan to migrate the function to an arduino nano soon.

pine64 and perl guinea pig habitat environment controls
pine64 and perl guinea pig habitat environment controls
pine64 guinea pig night light
pine64 guinea pig night light
per & pine64 guinea pig automation
perl & pine64 guinea pig automation

logging data with adafruit io using raspberry pi and cURL

adafruit io data logging
adafruit io data logging

adafruit.com is my preferred provider for prototyping supplies, raspberry pi devices, and most of all  breakout boards.  I just recently starting using the free version of their data logging service and I am very impressed.

I have an MCP9808 i2c temperature sensor connected to a pi3 set up in my garage.  I wrote a perl module to get the temperature from the MCP9808:

package MCP9808;
use strict;
use RPi::I2C;

=pod
experimenting with MCP9808 temperature sensor
ambient temp register returns 2's complement

ambient temp register is 0x05, returns 16-bit result;
bits 15-13 must be masked out, bit 12 is sign bit, 
12-bit result is 11-0b.  Lower byte 0-7b, upper byte
8-15b.  Conversion to decimal:
Temp = (upperbyte * 2^4) + (lowerbyte * 2-4)
=cut

sub new {
	#constructor, taked I2C addr as arg
	my $self = { addr => $_[1]  };
	bless ($self, "MCP9808");
	#print "addr:\t", $self->{addr}, "\n";
	return $self;
}#end new

sub read_temp{
	my $self = shift;

	my @result = ();
	my $tempdev = RPi::I2C->new($self->{addr});

	#read 2 bytes from temp register 0x05
	#returns array, upper byte, lower byte
	my @reading = $tempdev->read_block(2, 0x05);

	#convert reading to decimal temp
	my $ubyte = $reading[0];
	#strip 15-13b
	if($ubyte > 127){$ubyte-=128;}
	if($ubyte > 63){$ubyte-=64;}
	if($ubyte > 31){$ubyte-=32;}
	
	my $celsius = ($ubyte * 16) + ($reading[1] * 0.0625);
	my $farenheit = ($celsius * 1.8) + 32;
	
	push @result, $celsius;
	push @result, $farenheit;

	return @result;
}#end read_temp

1;

I had an existing script to get the current temperature from the mcp9808 and display the time and temperature on a 4×20 LCD display.    I only had to change one line of code to get it to post my readings to adafruit io

#!/usr/bin/perl
use strict;
use Time::HiRes qw(usleep);
use MCP9808;
#require "/home/pi/gpio/MAX7219.pl";

my $t1 = MCP9808->new(0x18);

#init timer
my $begin = time();
my $hr = 0;
my $min = 0;
my $sec = 0;
my $time_val = $hr."-".$min."-".$sec;

for(;;){
	my $now = time();
	my $tsec = $now - $begin;

	my @res = $t1->read_temp();
	my $deg_f = sprintf("%2.1f", $res[1]);

	#calc min / sec
	$hr  = ($tsec / 60) / 60 ;
	$hr = int($hr);
	$min = $tsec / 60;
	$min = int($min);
	$sec = $tsec%60;
	$sec = int($sec);
	#print length($sec)."\n";
	if($sec == 60){
		$sec = 0;
	}#end if
	if($min >= 60){
		$min = $min - ($hr * 60);
	}#end if

	if(length($min) eq 1){ $min = "0".$min; }
	if(length($sec) eq 1){ $sec = "0".$sec; }
	if(length($hr) eq 1){ $hr = "0".$hr; }

	#linux date command output
	my $time_val1 = `date`; 
	print $time_val1;
	chomp($time_val1);

	if($tsec > 0){
		$time_val = $hr.":".$min.":".$sec;
		#print_sentence($time_val, 500, 1);
		#`./lcd "Temp: $deg_f F        Time: $time_val"`;
		`./lcd "Temp: $deg_f F        $time_val1"`;
		#send data to adafruit.io
		curl -H 'X-AIO-Key: 2347yfudnvuefh378myaiokeygoeshere2h8f' -H 'Content-Type: application/json' -X POST -d '{"datum":{"value": "$deg_f"}}' https://io.adafruit.com/api/v2/icom032/feeds/garage-temperature/data
	}#end if

	usleep(10000000);
}#end for

this is the line of code that posts my temperature measurements to adafruit io:

curl -H 'X-AIO-Key: fudnvuefh378myaiokeygoeshere2h8f' -H 'Content-Type: application/json' -X POST -d '{"datum":{"value": "$deg_f"}}' https://io.adafruit.com/api/v2/icom032/feeds/garage-temperature/data

I changed my key, of course.  curl can be used for development boards that run a full linux distribution.  There are other libraries for devices like arduino.  I am amazed at how simple they made it.

The charts look cool, scales automatically, updates in near real time, and has cool mouse-over effects.

adafruit.io graph of MCP9808 temperature sensor in my garage connected to raspberry pi 3
adafruit.io graph of MCP9808 temperature sensor in my garage connected to raspberry pi 3

I downloaded my data in csv format and graphed it using my own perl tk csv database analysis tool.  It matches up pretty well, but the adafruit graphs are way cooler!

perl Tk csv database analysis tool
my own perl Tk csv database analysis tool graphing the same data I downloaded

I have a lot of ideas I want to try and integrate into adafruit IO.  Here is three days worth of readings from my temperature sensor in the garage->

adafruit IO interactive real-time graph
adafruit IO interactive real-time graph