Greqo 4 Released

Greqo is a PHP project I started a couple years back for Google’s services. It was an extension of a Blogger API I had written in PHP. I extended it to include Gmail and Google Analytics. I took some time this year to clean it up and recently released Greqo 4. It’s not really the 4th version. More like the 2nd. The 4 signifies full compatability with PHP 4.

For more information on usage, I recommend the unit and acceptance tests that come packaged with the source code.

There are probably better PHP libraries for Google’s Data APIs (e.g. the Zend Framework.) PHPMailer, upon which my Gmail library was built, now supports Gmail natively. Google Analytics now has an export API. And PHP 4, of course, has reached the end of its life.

But Greqo 4 runs on PHP 5. And if you need a PHP 4 implementation of these Google APIs, well then here you are:

Download: Greqo 4 Release for Google Mail, Blogger, and Analytics

Going forward, the Greqo library will abandon complete PHP 4 compatibility and focus on PHP 5 and, eventually, PHP 6.

FormEncode (Pylons): What Is State?

If you’re using Pylons, the Python framework, you’re probably using FormEncode. And if you’re using FormEncode, you’ve probably have noticed, and blithely ignored, the state argument that’s the last argument to a number of the validator class methods.

This is the official FormEncode explanation of state:

All the validators receive a magic, somewhat meaningless state argument (which defaults to None). It’s used for very little in the validation system as distributed, but is primarily intended to be an object you can use to hook your validator into the context of the larger system.

For instance, imagine a validator that checks that a user is permitted access to some resource. How will the validator know which user is logged in? State! Imagine you are localizing it, how will the validator know the locale? State! Whatever else you need to pass in, just put it in the state object as an attribute, then look for that attribute in your validator.

Hmmm. Maybe an example would help. Here’s the best example of its usage I could find on the Pylons site:

  this_schema = TheAppropriateSchema()
  try:
    state = dict() # this won't actually work. you'll need an object that formencode can hang things on.
    state["useful_state_information"] = something_useful
    form_result = this_schema.to_python(postvars, state)

    # validation is successful - form_result contains good data for you to consume

    redirect_to("somewhere")

  except Invalid, e:
    defaults = request.params
    errors = e.error_dict
    unfilled_html = a_function_that_draws_your_form_page_html()
    filled_html = htmlfill.render(unfilled_html, defaults, errors)
    return filled_html

Notice the comment: # this won’t actually work. Thanks for the warning.

So what is state and why would you want to use it?

Let’s answer the second question first, because at work, where we’re using the Pylons framework, I came up with an excellent situation which helped me figure out what it is and why I would want to use it.

The situation, briefly: our application has two separate forms. One is a form where a user can add new records in a multi-row table form. Each row is an individual record. The second is a review form, which looks just like the new record form, but is filled in from data uploaded in csv file (from another form).

Since DRY is a guiding principle of development in my office, the goal is to use the same underlying code for the form, controller, validation, etc. But the problem is that data originate in two different formats: one is as POST data from the add form, the second as the contents of a csv file. So how can we normalize these two data formats so that we can use the same underlying code? All together now: State! Now you’re talking!

If we could just tell our FormEncode validator where the data was coming from, then we could create two separate normalization methods in our validation class that would transform the data into a form that the validator and template could deal with. Anyway, enough verbiage. Find below a representative generalized version of the FormEncode validator I created and the controller it’s used in.

One more note before I finish. Notice the FormencodeState object comment. That explains why the example from the Pylons wiki does not work (in most cases). My first impulse would be to use a dict, too, as the state object. But the internals of the FormEncode validator class require an object. This is what gets passed in the validator method state argument and it is what passes info from the general framework environment to the validator.

Pylons Controller Code (with State Object)

class FormencodeState(object):
    """
    State class for formencode
    Although NOT well documented, to use the state argument in the to_python
    method in the context of schema that does complex, multistep validation,
    the state argument must be an object that formencode can hang additional
    attributes from, else you get errors like:
    Module formencode.schema:114 in _to_python
    >>  state.full_dict = value_dict
    <type 'exceptions.AttributeError'>: 'dict' object has no attribute 'full_dict'
    """
    pass

# ... within actual Controller class method
Validator = SourceDataPreValidator()
ControllerState = FormencodeState()
ControllerState.source = 'csv'
Validator.to_python(DataValue, ControllerState)     # <- state object in action!

FormEncode Validator Code

class SourceDataPreValidator(formencode.validators.FormValidator):
    """normalize data from either a form submission or a csv file upload and validate"""
    validate_partial_form = True
    def _to_python(self, value, state):
        """normalize csv input"""
        if state.source == 'csv':
            value['normal_data'] = self._normalize_csv_data(value.get('csv_import_file'), state)
        elif state.source == 'form':
            pass
        return value

    def _normalize_import_data(self, csv_file_string, state):
        NormalDataList = []
        ImportedLines = csv_file_string.split('\n')
        for i in range(len(ImportedLines)):
            ColValues = ImportedLines[i].split(',')
            fpre = 'csv_import-' + str(i)
            NormalDataList.append((fpre + '.id', str(ColValues[1])))
            NormalDataList.append((fpre + '.amount', str(ColValues[2])))
            NormalDataList.append((fpre + '.date',
                self._importdate_to_python_date(ColValues[4])))
        return NormalDataList

    def _importdate_to_python_date(self, datestr):
        """some code to convert a date string to object"""
        pass

I hope that sheds a little light on this powerful mystery. Questions or comments welcome.

Comparing Big Numbers in PHP

Problem

What if I need to compare really big numbers in PHP? Like comparing 2^66 > 3^53?

Overview

This will work:

print (int) (pow(2,66) > pow(3,53));

PHP will convert the integers to scientific notation. But this script illustrates the limitation of normal operational syntax (i.e.: pow(2,66) > pow(3,53) vs. bccomp(bcpow(2,66), bcpow(3,53),1) > 0):

$max = 1000;
foreach ( range(1,$max) as $x )
{
    if (
        ! ( pow(2,$x)+1 > pow(2,$x) ) ||
        ! ( pow(2,$x)-1 < pow(2,$x) ) ||
        ! ( pow(2,$x-1)*2 == pow(2,$x) )
    ) break;
}

$maxpow = $x-1;
$maxint = bcpow(2, $maxpow);
$Result['op'] = "max supported int: 2^$maxpow or $maxint\n";

foreach ( range(1,$max) as $x )
{
    if (
        ! ( bccomp(bcadd(bcpow(2,$x),1), bcpow(2,$x)) == 1 ) ||
        ! ( bccomp(bcsub(bcpow(2,$x),1), bcpow(2,$x)) == -1 ) ||
        ! ( bccomp(bcmul(bcpow(2,$x-1),2), bcpow(2,$x)) == 0 )
    ) break;
}

$maxpow = $x-1;
$maxint = bcpow(2, $maxpow);
$Result['bc'] = "max supported int >= 2^$maxpow or $maxint\n";

printf("<pre>%s</pre>", print_r($Result,1));

Solution

Use bccomp. The script above can be found on the klenwell code site:

http://code.google.com/p/klenwell/source/browse/trunk/pastebin/bc_demo.php

Results will vary depending on the processing capacity of the system. Here’s the results on my machine:

Array
(
[op] => max supported int: 2^52 or 4503599627370496

[bc] => max supported int >= 2^999 or 5357543035931336604742125245300009052807024058527668037218751941851755255624680612465991894078479290637973364587765734125935726428461570217992288787349287401967283887412115492710537302531185570938977091076523237491790970633699383779582771973038531457285598238843271083830214915826312193418602834034688
)

Command Line Email on Ubuntu (mailx version)

This guide explains how to configure Ubuntu so that you can send email from the command line using mailx with your Gmail account for delivery. This updates a guide I originally posted on the ubuntu forums for mailx, which is part of the current Ubuntu repositories (versions 7 and 8). To use nail, for which this guide was originally drawn up, see this post.

In 5 Fairly Easy Steps

1. Install the needed programs

$ sudo apt-get install msmtp
$ sudo apt-get install mailx

2. Install Thawte certificates for Gmail

EDIT: verisign.com apparently no longer issues certs at the address below. So the ‘wget’ step will not work. adkein, in a comment below (click here), links to another page when you can apparently get the needed cert. I cannot vouch for this as my cert still works. But you might try downloading that and putting it your ~/etc/.certs directory if you’re having issues finding the cert.

This is necessary for Gmail. (Thanks to laurentbois.com.)

$ mkdir -p ~/etc/.certs
$ chmod 0700 ~/etc/.certs
$ cd ~/etc/.certs
$ wget https://www.verisign.com/support/thawte-roots.zip –no-check-certificate
$ unzip thawte-roots.zip
$ cp ‘Thawte Server Roots/ThawtePremiumServerCA_b64.txt’ ThawtePremiumServerCA.crt

3. Configure msmtp
This will open up a new msmtp configuration file

$ gedit ~/.msmtprc

Copy the following lines. Replace UPPERCASE text with your personal settings:

# config options: http://msmtp.sourceforge.net/doc/msmtp.html#A-user-configuration-file
defaults
logfile /tmp/msmtp.log

# gmail account
#account gmail
auth on
host smtp.gmail.com
port 587
user YOURNAME@gmail.com
password YOURPASSWORD
from YOURNAME@gmail.com
tls on
tls_trust_file /home/USER/etc/.certs/ThawtePremiumServerCA.crt

# set default account to use (not necessary with single account)
#account default : gmail

Change permission on this file or msmtp will complain:

$ chmod 600 ~/.msmtprc

4. Configure mailx

$ gedit ~/.mailrc

Copy the following lines and replace UPPERCASE text with your personal settings:

# set smtp for nail
# ref: http://ubuntuforums.org/showpost.php?p=4531994&amp;amp;amp;amp;amp;amp;amp;amp;amp;postcount=6

# gmail account (default)
# $ mailx -s "subject line" -a /path/file recipient@email.com < /path/body.txt
set from="YOURNAME@gmail.com (YOURNAME)"
set sendmail="/usr/bin/msmtp"
set message-sendmail-extra-arguments="-a gmail"

5. Run a test
Send a test message to your gmail account:

$ echo -e “testing email from the command line” > /tmp/test_email
$ mailx -s “nail gmail test” YOURNAME@gmail.com < /tmp/test_email

Check your gmail account and you should have a message from yourself. You can also check your log:

$ gedit /tmp/msmtp.log

I personally prefer nail because it more easily accommodates multiple accounts. But if you don’t have a deliberate need to use multiple accounts and are using a later version of Ubuntu, use this.

Command Line Email on Ubuntu (nail version)

This will be the first of two posts showing you how to configure the more recent versions of Ubuntu to send email from the command line as simply as humanly possible using your Gmail account for delivery. These instructions were originally posted on my old blog and on the Ubuntu forums site.

This guide covers using the nail program, which is my client of choice but is no longer in the main repositories. Find a guide for using mailx, which is part of the current Ubuntu repositories, here.

In 6 Fairly Easy Steps

1. Add the breezy repositories containing nail to your source file
Edit your sources list

$ sudo gedit /etc/apt/sources.list

Add the following lines at bottom. These will enable you to apt-get nail

# breezy repositories (added to install nail)
# see http://old-releases.ubuntu.com/releases/ for more info
deb http://old-releases.ubuntu.com/ubuntu/ breezy universe

Don’t forget to update

$ sudo apt-get update

2. Install the needed programs

$ sudo apt-get install msmtp
$ sudo apt-get install nail

3. Install Thawte certificates for Gmail

EDIT: verisign.com apparently no longer issues certs at the address below. So the ‘wget’ step will not work. According to this comment, you can downloaded the needed cert from http://www.cs.utexas.edu/~suriya/UT-wireless/ThawtePremiumServerCA_b64.txt. I cannot vouch for this as my cert still works. But you might try downloading that and putting it your ~/etc/.certs directory if you’re having issues finding the cert.

This is necessary for Gmail. (Thanks to laurentbois.com.)

$ mkdir -p ~/etc/.certs
$ chmod 0700 ~/etc/.certs
$ cd ~/etc/.certs
$ wget https://www.verisign.com/support/thawte-roots.zip –no-check-certificate
$ unzip thawte-roots.zip
$ cp Thawte Server Roots/ThawtePremiumServerCA_b64.txt ThawtePremiumServerCA.crt

4. Configure msmtp
This will open up a new msmtp configuration file

$ gedit ~/.msmtprc

Copy the following lines. Replace UPPERCASE text with your personal settings:

# config options: http://msmtp.sourceforge.net/doc/msmtp.html#A-user-configuration-file
defaults
logfile /tmp/msmtp.log

# isp account
account isp
auth login
host SMTP.YOURISP.COM
port 25
user YOURNAME@ISP.COM
from YOURNAME@ISP.COM
password YOURPASSWORD

# gmail account
account gmail
auth on
host smtp.gmail.com
port 587
user YOURNAME@gmail.com
password YOURPASSWORD
from YOURNAME@gmail.com
tls on
tls_trust_file /home/USER/etc/.certs/ThawtePremiumServerCA.crt

# set default account to use (from above)
account default : gmail

Change permission on this file or msmtp will complain:

$ chmod 600 ~/.msmtprc

5. Configure nail

$ gedit ~/.mailrc

Copy the following lines and replace UPPERCASE text with your personal settings:

# set smtp for nail
# ref: http://ubuntuforums.org/showpost.php?p=4531994&amp;amp;amp;amp;amp;postcount=6

# gmail account (default)
# $ nail -s "subject line" -a /path/file recipient@email.com < /path/body.txt
set from="YOURNAME@gmail.com (YOURNAME)"
set sendmail="/usr/bin/msmtp"
set message-sendmail-extra-arguments="-a gmail"

# isp account (add -A option to command line)
# $ nail -A isp -s "subject line" -a /path/file recipient@email.com < /path/body.txt
account isp {
set from="YOURNAME@ISP.COM"
set sendmail="/usr/bin/msmtp"
set message-sendmail-extra-arguments="-a isp"
}

6. Run a test
Send a test message to your gmail account:

$ echo -e “testing email from the command line” > /tmp/test_email
$ nail -s “nail gmail test” YOURNAME@gmail.com < /tmp/test_email

Check your gmail account and you should have a message from yourself. You can also check your log:

$ gedit /tmp/msmtp.log

That’s it. You should now be ready to incorporate email into your bash scripts, which is great for making backups. I did successfully test this on an old laptop had installed Ubuntu 8.04 this weekend.

If you have any problems or questions, I’d recommend posting to the ubuntu forums thread as you’ll probably get a quicker response there.