When I struck gold and didn’t want to cobble together my files yet again (IE: every time I reinstall) I decided to begin versioning my dotfiles. I began in the most obvious of ways: I made my home directory a git repository, ignored all my files, and only explicitly added all the files I really wanted in there. This worked for a while, but the files that I truly wish WERE versioned were not - my SSH and GPG keys, and my SSH configuration. Additionally, I had a few bash variables which contained private information that I didn’t want to make public.

The obvious solution was to put these into another repository hosted on my own server but integrating that repository into my home directory is what infuriated me.

Symlinks

The first and common method people seem to be doing is using tools like dotty or homesick to manage the multiple repositories and setup symlinks automatically.

These tools worked fine for a little while, but they had a severe downside: neither of them are designed to be easy to add files back in. This made it very difficult to maintain or find files I had forgotten to import; it was mostly a guess-and-check kind of situation.

They created a reasonable interface to working with themselves, but when it came down to using git like it is supposed to be, it was just an enormous headache.

Unfortunately the problem with using symlinks in this situation is inherent: git doesn’t know what you’re doing, and does its best. I cannot blame git.

A Proposed Solution

My biggest frustration with these tools were they didn’t let git do what it is good at: manage my dammed files. They tried to be smarter than git, and really quite poorly. It pissed me off a good deal.

My ideal solution was letting git manage itself like I know it can, but I just needed it to support multiple repositories in one directory.

If only…

If only I could have my git repository files located in, say, ~/.git-repos/private, but tell it to check my files out into ~/ that would do it, right?

OH WAIT!

Git already provides the tools to do exactly to this!

• $GIT_DIR - This tells Git where to look for the .git directory. In this way you could be in /tmp/whatever and manage your git repository located in /home/grahamc/my-awesome-files.
• GIT_WORK_TREE - This tells Git where the files are that it is tracking. Using this your .git directory could be in /tmp/whatever and actually have all your checked out files be put in to /home/grahamc/my-awesome-files.

Now we’re getting somewhere!

The Sexy Solution: git multi

That’s right, I built it out. Using those two shell variables it was pretty trivial, didn’t require crazy dependencies, or any nasty symlinks.

Go ahead and get these shell scripts, drop them in to your ~/bin (or path, I suppose) and be enlightened.

Adding a Repository

git multi add public git@github.com:youruser/dotfiles.git

Blammo. Done. That’s it. That even checks out the repository and everything.

Working on a Repository

git multi work public

This command opens up a bash terminal with $GIT_DIR and $GIT_WORK_TREE already set and everything. Your git commands work like they were native. Wait. Mmmm… it’s because they are.

When you’re done and want to commit or push? Just do that. git commit; git push origin master.

For bonus points, add grahamc/git-multi (or your own fork) as an additional repository to your home directory.

Hit me up with a pull request or comments if you have issues.

It was pretty easy, git init --bare secure.git to initialize a repository at ./secure.git. The problem with this, is even if you set the umask to 0077, the files will become readable by all users after you push. You could re-mask them to be 0700, but next time you push it’ll store new files too permissively.

The solution is fairly easy, but it took a little bit of googling:

git init --bare --shared=0700 secure.git

This causes all files in this repository to only be readable by the user who owns the directory. If you want your files to be secure, make sure you initialize your repository with this command, otherwise everyone will be able to read your PGP keys.

Now, pretending your server is your-ssh-server.com, and the username is user, you would add it as a remote to your repository as such: git remote add origin user@your-ssh-server.com:path/to/secure.git

This early exposure to these beautiful, precision instruments has always inspired me to work with them - I just never knew how, or what I would use them for.

About a year ago I saw the GitHub stoplight pass along my feed reader, and I was hooked. I began researching stoplights and how to incorporate them, but it was too easy; no challenge after my Price is Right, Cliff Hanger clone.

I wanted a similar system to inform me of my mistake on my software, but I also wanted a challenge, and I wanted it to be unique. Ultimately, I decided on using an Arduino with a motor control shield to control the chiming of a Ship’s bell: One strike meant success, a second strike meant a failure.

Controlling the Strike

As it turns out, a clock’s chime isn’t rocket science. Something mechanical triggers the chime, and it runs autonomously. In this case, every half hour a trip rotating pin pushes the strike trigger out, allowing the process to begin.

Brilliantly enough, in order for the strike to do one chime (instead of two), the second strike lever falls and prevents the hammer from hitting the bell a second time.

By mechanically pulling the strike trigger, and lifting (or dropping) the second strike lever, I could reliably and fairly easily trigger the clock to strike once or twice.

Controlling the Strike… Electronically

Setting up a couple of motors to tug on the strike trigger and the second strike lever was fairly easy. I used a SparkFun Dual Motor Driver along with a couple of geared motors and the hub from a wheel. Using a bent paper clip and some dental floss, I was able to trigger both actions without much motor movement.

I knew from the get-go I needed a way to know if I had moved the levers far enough, but I wasn’t sure how to go about it.

Sensing the Levers

I discovered by mistake that I could ground both levers from the same point, and that placing a wire at the right point would allow me to know for certain when I had pulled a lever far enough. By waiting for these circuits to be completed, I could stop pulling the lever at the precise moment.

I used copper desoldering braid wrapped around an insulated peg to sense when the second strike lever was all the way up.

All Together Now

Possibly the hardest part of this project was figuring out a way to know when to stop pulling on the levers. This took by far the longest, but looking back, was the most pivotal in having it be successful. In fact, additional value would be gained by adding feedback systems to know exactly when they are in their “off” position too, as these are still done via a timer.

All in all, I’m fairly pleased with the results - all that is functionally left now is having my testing server push notifications to the chime. Eventually, I would like to clean it up; put the electronics away, mount it on wood and possibly put a quartz movement in there as if it were a real clock.

Now this was pretty close to enough for me to enjoy it, but since NationalField is a data and analytics driven company, we needed to step it up.

Thursday Night

Three days before the big day we decided that in order to really know for certain how many wings we’ve eaten, we needed an accurate count. We needed a great, fun, real-time dashboard of how many wings we’ve snarfed down, and an easy, BBQ-sauce friendly way to report that data. A solution that means no sticky fingers or beer on my laptop.

I had a lot to do in just 48 hours:

• build an easy reporting system that didn’t involve a sticky mess.
• get anything that was necessary to build and implement this system.
• build a website to display this information in real time.

The Dashboard

Note, this was taken after KeepWinging ended; otherwise the WPH would show a real number.

We wanted a very simple dashboard which would display only the most important stats in our venture to eat 250 chickens worth of buffalo wings. We also wanted it to be live-updating, and give some level of competition between users.

This was done with a simple jQuery $.get() call plus a setTimeout at the end. The polling was simple and quick, and for a guy who doesn’t know Javascript very well - it was just perfect. I looked at Node.JS, however with only three days to go I didn’t have the time to learn it.

What We Came Up With

• Leaderboard: The top nine users, showing their total wing consumption and how they compared to the people behind them. This seemed to encourage people who were five below a spot to eat more, and had people fighting to stay on the board.
• Total Wings Consumed: We actually had this in two places, in graph form (which for some reason didn’t display accurately, but the trend was correct) and as a big number along the top. It was so exciting seeing it roll over to 900, we knew it had to be possible to finish. Below this we also displayed the total number of wings remaining.
• Wings Per Hour (WPH): We thought it was very neat to know just how many wings we were eating per hour, and we also displayed the number of wings per hour which were necessary to complete the 1,000 wings by the end of the game. At one point we were up to 300 WPH.

Each of these items would automatically refresh every second or two, so feedback was almost instant. We knew exactly when consumption was falling off, as well as when fresh “meat” joined. This was displayed on a gigantic television screen next to the wings.

The Feed

The feed would update every second.

Let’s face it, the internet loves a good live-updating feed of what is going on. We love that instant feedback of change in our lives, so we couldn’t resist building one.

The feed was extremely simple, and only updated when a user registered or ate more wings. This view however, was exciting to have and be able to see how quickly people were allocating themselves more wings.

How It All Worked

The solution to mess-free reporting was pretty obvious: RFID (Radio Frequency IDentification, like what you might scan to open a keyless door.) Each participant would register an RFID tag with the system, and every time they threw out 5 wings they would be allowed to scan their tag. Since I already had a reader, this was feasible.

KeepWinging.com

KeepWinging was written entirely in Symfony using Propel; it had a very simple JSON RPC API for providing the data we needed for the content pages - and a single reporting endpoint.

I started writing the code on Friday evening and finally pulled it all together at 2:50 PM, Sunday afternoon, with the party starting at 3:00PM. Perfect timing.

The RFID Reader

Thursday night was spent building the RFID reader: an arduino with a Parallax RFID reader which I got from RadioShack, a couple of LEDS, all mushed into a used Korean Bibimbop take-out dish. It was perfect. All we needed were the RFID tags.

I also incorporated some simple human feedback: users are used to visual and audio feedback when reading a tag like this. I incorporated a visual cue by having a red and blue LED. When the tag was scanned, the blue light would switch to red. Unfortunately I didn’t have a buzzer loud enough to stand a chance at a party, so I removed this part of the feedback. (Yes, I did get complaints that it didn’t beep or buzz.)

One note about RFID readers which I learned during this project:

Since RFID readers use radio frequencies, the number of devices and signals we have going through the airways can cause false-positive matches. These codes would look like F0000F0000 and were almost always obviously fake.

In order to reduce the number of false positives received, it is recommended to try reading the tag twice within a few seconds (I used 2) before it considers it a successful read. This will prevent almost all false positive reads.

The RFID Tags

The tags were a little bit tricky. I only had the two which came with the kit, and we were expecting around 30 people. Now that wouldn’t be fun, we’d only have two teams; we had to order more.

It was 10:00 PM on Thursday, the party was on Sunday, and I had to order 50 RFID tags. This meant overnight shipping. This meant Saturday delivery. This meant an AM delivery so I would have some time to make sure they worked. This meant that the cost of shipping was 2x the cost of the actual tags themselves. Boy, when I ordered these tags I committed myself to a lot; namely that they would work.

The Glue (Arduino -> Laptop -> KeepWinging.com)

In order to get the tag data from the Arduino to the laptop, I had my brother and a friend help hack out a Perl based script to read from the serial port and execute a command when it received one. This would just execute a local Symfony task: ./symfony tag:reportRemote --env=prod $tag

The code was simple enough: read 10 bytes from the reader, send it along through the USB serial port to my laptop using the Arduino’s handy reader.

By having the registered check, setting up a new RFID tag was a snap and nearly problem-free.

That task would perform the registration or the reporting through a very simple RPC API I built. This was no attempt at being RESTful; I just needed to get the thing done.

Nearing the End

At the end of the night, we were about 20 away from finishing off a thousand - but the simple fact was that nobody could stand to eat five more. Solution: Reduce the report number down to 1, and everyone only has to eat one more wing to push us over. After a little bit of cheering and motivational speaking, we polished off an entire thousand buffalo wings right as the Super Bowl’s clock ran out of time.

Success.

This was also posted on NationalField’s blog, at http://nationalfield.org/2011/02/keepwinging/

The resulting Makefile is fairly basic, but performs essential operations:

• clean: Delete the locally built files
• build: Generate the website files locally (automatically runs clean)
• push: Push the generated files to the remote server. This uses scp by default.
• new: Generate a new Markdown document in _posts. Pass TOPIC="foo" to change the name of the generated post (new article by default).

To use this Makefile with Jekyll, just reconfigure the REMOTEHOST and REMOTEDIR parameters at the bottom of the file with your own settings.

deploy: build push

help:
	@echo "You may provide several parameters, like:"
	@echo "make [target] KEY=\"value\""
	@echo ""
	@echo "The following parameters are available (with the defaults): "
	@echo "REMOTEHOST=$(REMOTEHOST)"
	@echo "REMOTEDIR=$(REMOTEDIR)"
	@echo ""
	@echo "You may provide the TOPIC variable to the 'new' target."
	@echo ""

.PHONY: clean
clean:
	rm -rf _site/*

push:
	scp -r _site/* $(REMOTE)
	growlnotify -m "BLOG: Uploaded."

build: clean
	jekyll
	chmod -R 755 _site/*
	growlnotify -m "BLOG: Built."

serve: clean
	jekyll --server

new:
	echo "---" >> $(FILE)
	echo "title: $(TOPIC)" >> $(FILE)
	echo "layout: post" >> $(FILE)
	echo "published: false" >> $(FILE)
	echo "---" >> $(FILE)
	open $(FILE)

# Change these settings for your own use, for example:
# REMOTEHOST ?= yourwebsite.com
# REMOTEDIR ?= /path/to/your/webroot
# Note the lack of a / on webroot
REMOTEHOST ?= yakko
REMOTEDIR ?= ~/grahamc.com/main/public
REMOTE = $(REMOTEHOST):$(REMOTEDIR)

TOPIC ?= new article
FILE = $(shell date "+./_posts/%Y-%m-%d-$(TOPIC).markdown" | sed -e y/\ /-/)

This Makefile also utilizes the Mac program Growl. If you don’t use a Mac (or don’t have Growl, remove the growlnotify lines.)

When you get into contexts, configurations, applications, database integration, or nearly anything else that touches even the basic symfony bits, you drag in the whole symfony stack. Something in the stack caused every test to throw an unintelligible RuntimeException, with thousands of literal question marks (read: not an encoding error.)

Even through modifying symfony and PHPUnit’s sourcecode, running XDebug, not running XDebug, etc. nothing could convert those question marks into real-life, intelligible errors. Imagine how silly I felt searching Google for ”phpunit symfony question marks” - nothing, as you could imagine, came up.

I had worked through both of the existing Symfony plugins, sfPHPUnitPlugin and sfPHPUnit2Plugin. One of the is very over-arching and depends on an old version of PHPUnit (however it did work) and the newer one gave me the same issue.

After a long time of organized debugging, I devolved (as one usually does) to a series of shotgun debugging.

How I Solved It

Firstly, you’re going to need to use a bootstrap file to initialize the context. Basic, cut and dry:

<?php
$path = realpath(dirname(__FILE__) . '/../config/');
require_once $path . '/ProjectConfiguration.class.php';

// Initialize the application
$m = ProjectConfiguration::getApplicationConfiguration('frontend',
                                                       'testing', true);

// Now initialize the database bits
sfContext::createInstance($m);
new sfDatabaseManager($m);

error_reporting(E_ALL | E_STRICT);
ini_set('display_errors', true);
?>

Drop this into test/bootstrap.php. Every time you run phpunit from here, you pass the --bootstrap test/bootstrap.php

<phpunit
    colors="true"
    verbose="true"
    convertErrorsToExceptions="true"
    convertNoticesToExceptions="true"
    convertWarningsToExceptions="true"
    stopOnFailure="true"
    processIsolation="false"
    syntaxCheck="true"
    bootstrap="test/bootstrap.php">

    <testsuites>
        <testsuite name="Unit Tests">
            <directory>test/unit/</directory>
        </testsuite>
    </testsuites>

    <filter>
        <blacklist>
            <directory suffix=".php">lib/vendor</directory>
            <directory suffix=".php">cache</directory>
            <exclude>
                <directory suffix=".php">lib/vendor</directory>
            </exclude>
        </blacklist>
    </filter>
</phpunit>

Typing in phpunit --bootstrap test/bootstrap.php is pretty ridiculous, so drop this XML into phpunit.xml.

That phpunit.xml will make it pretty and ensure to exclude the extraneous files which you don’t want included in any coverage data. It will also specify the bootstrap.php which you just created. At this stage you just execute phpunit and all should be well.

For the record, I never did discover the cause of the question marks. I did, discover, however, that the trick is to enable process isolation.

While working on a new project, I was exhausted by all the link_to writing I had been doing. It was repetitive, and even though I was using proper object-based routes - it was still ugly. All of them were formatted the exact same way, it was just silly. Definitely not DRY.

I started refactoring this by making a helper to link to an object. It was specific - link_to_group(NFGroup $group), link_to_user(sfGuardUser $user) and about the third time I wrote a nearly identical helper, I wrote another helper - link_to_object(BaseObject $object, $route). This wasn’t pretty, and I wanted a simpler solution.

What I ended up with was an interface for objects that I wanted to be linkable. The interface provided a way to get the route, parameters for the route, and then used the __toString() method on the object for the link text.

<?php
/**
 * Makes an object easy to link to by passing it to the
 * linkableLink helper
 *
 * @author Graham Christensen <graham@grahamc.com>
 */
interface NF_Linkable {
	/**
	 * Get the route
	 * @return string The route
	 */
	public function getRoute();

	/**
	 * Get route parameters
	 * @return array of route parameters
	 */
	public function getParameters();

	/**
	 * Get the text to use as link-text
	 * @return string
	 */
	public function __toString();
}
?>

The NF_Linkable interface, allowing it to be easily linked to in a symfony view. To use this, place it in lib/NF_Linkable.php

My new helper is named linkableLink(NF_Linkable $object) and it will output a link, nice and simply. The helper is fairly straightforward too:

<?php
/**
 * Link to any object extending NF_Linkable
 * @param NF_Linkable $object
 * @return string
 * @author Graham Christensen <graham@grahamc.com>
 */
function linkableLink(NF_Linkable $object) {
	$url = $object->getRoute() . '?';
	$url .= http_build_query($object->getParameters());

	return link_to($object, $url);
}
?>

This is pretty straightforward. It takes the object’s route, builds the query string from the parameters, and passes it along to symfony’s link_to helper. For those following along, place this in lib/helpers/LinkableHelper.php

To use this is simple enough, using the example of a group’s Propel object - we simply implement the interface’s methods:

<?php

/**
 * A group object which is linkable using linkableLink
 *
 * @author Graham Christensen <graham@grahamc.com>
 */
class FieldGroup extends BaseFieldGroup
						implements NF_Linkable {
	/**
	 * Get the route to show a group
	 * @return string
	 */
	public function getRoute() {
		return '@group_show';
	}

	/**
	 * Get the route's parameters for building the final URL
	 * @return array
	 */
	public function getParameters() {
		return array('id' => $this->getId());
	}

	/**
	 * Get the text to appear between in <a>text</a> tags
	 * @return string
	 */
	public function __toString() {
		return $this->getName();
	}
}
?>

This is an example of how I use the NF_Linkable interface. This would create a link to this particular group using the name as the link text. The URL generated would be something like @group_show?id=1, which

symfony translates into group/view/1

To then use this in your codebase, write a simple view:

<?php
// Get a FieldGroup just for the example
$group = FieldGroupPeer::doSelectOne(new Criteria());

use_helper('Linkable');
echo linkableLink($group);
?>

Turns out it was pretty obvious:

To create the branch:

git svn branch foo

To switch to the branch:

git branch foo remotes/foo

To delete a remote branch you have to do it with SVN’s command line tool:

svn rm http://svn.foobar.org/branches/foo

Add the following code to your app/app_name/config/factories.yml file:

all:
    storage:
    class: sfPDOSessionStorage
    param:
      db_table:    session
      database:    propel
      # Optional parameters
      db_id_col:   sess_id
      db_data_col: sess_data
      db_time_col: sess_time

Make sure your remove all unnecessary references to setting a different storage mechanism

Now add the following YAML to your config/schema.yml file. This creates the table structure.

# Session
  session:
    _attributes: { phpName: Session }
    sess_id: { type: varchar, size: 64,
               required: true, primaryKey: true }
    sess_data: { type: longvarchar }
    sess_time: { type: INTEGER, size: '11'}
    _indexes: { SESSIONTIME: [sess_time] }

I try to keep the generated SQL as up to date as possible, so do that now.

./symfony cc
./symfony propel:build-sql

If you need to make this change to an existing dataaset, here is the raw SQL to create the table:

CREATE TABLE `sessions`
(
        `sess_id` VARCHAR(64)  NOT NULL,
        `sess_data` TEXT,
        `sess_time` INTEGER(11),
        PRIMARY KEY (`sess_id`),
        KEY `SESSIONTIME`(`sess_time`)
)Type=InnoDB;

Note: If the session table is MyISAM, you’re going to hurt yourself with table-level locking. Making it InnoDB means row-level locking, and much better performance. Also, this only scales so far - eventually memcache is the solution.

Fatal error: Class 'Swift_Message' not found in apps/lib/email/DomainReportMessage.class.php on line 3

And then the task:

<?php
class emailTestTask extends sfBaseTask {

    /* ... configure() - nothing special */

    protected function execute($arguments = array(),
                                   $options = array()) {

        // initialize the database connection
        $databaseManager = new sfDatabaseManager($this->configuration);
        $db = $databaseManager->getDatabase($options['connection']);
        $connection = $db->getConnection();


        $c = new DomainReportMessage();
        $this->getMailer()->send($c);
    }
}
?>

The simple fix for this is to add a call to $this->getMailer() before you use the Swift_Message class, which seems like a fairly basic issue with symfony’s autoloader.

Here is the working code:

<?php
class emailTestTask extends sfBaseTask {

    /* ... configure() - nothing special */

    protected function execute($arguments = array(),
                                   $options = array()) {

        // initialize the database connection
        $databaseManager = new sfDatabaseManager($this->configuration);
        $db = $databaseManager->getDatabase($options['connection']);
        $connection = $db->getConnection();

        $this->getMailer(); // These are the magic words
        $c = new DomainReportMessage();
        $this->getMailer()->send($c);
    }
}
?>

Writing a Custom sfValidator

Writing a custom validator really isn’t as difficult as you would imagine. Open up one of the existing ones, and you’ll see that. Primarily, you only need to worry about two methods.

configure()

The configure method is where you can specify customization options, making some required or optional. This method is used to build flexibility into the validator.

doClean()

The doClean method may seem confusing at first, however it’s very simple. Upon failure, throw an exception.

How to Signal a Failure

<?php
throw new sfValidatorError($this, 'invalid', array('value' => $value));
?>

invalid is the name of the message you want to error with, and the options array at in the third parameter allow you to replace variables in the message.

How to Handle a Success

If the validator is successful - you have two options:

• Your validator can act as a filter, and return something different from the input value
• Your validator can return exactly what it was passed in.

Implementing the new sfValidator into an sfForm

This validator is used just like any other:

<?php
$form->setValidator('domain', new sfValidatorDomain());
?>

<?php
/**
 * @author Graham Christensen <graham@grahamc.com>
*/
class sfValidatorDomain extends sfValidatorBase
{
  /**
   * Configures the current validator.
   *
   *
   * @param array $options    An array of options
   * @param array $messages   An array of error messages
   *
   * @see sfValidatorBase
   */
  protected function configure($options = array(),
                                $messages = array()) {
      // The type of method to use to validate it.
      $this->addOption('clean_type', 'url');

      // List of valid protocol schemes to allow in URLs
      $this->addOption('schemes', array('http', 'https'));

      // The default scheme
      $this->addOption('default_scheme', 'http');

	  // Setup some basic error messages
	  $msg = 'The provided domain does not appear to be valid.';
      $this->addMessage('badform', $msg);
      $this->addMessage('badscheme', $msg);
      $this->addMessage('nohost', $msg);
      $this->addMessage('invalid', $msg);
  }

  /**
   * @see sfValidatorBase
   */
  protected function doClean($value) {
      if ($this->getOption('clean_type') == 'domain') {
          // If it's a domain, then it's simple to check it.
          $domain = $value;
      } else {
          // It's probably a complete URL, so check it in
          // more depth.

          // Verify that it can be parsed as a URL.
          // Note: @'s are bad practice, however if a method is
          // being checked and we can't stop the error, then
          // we want to hide it.

          $parts = @parse_url($value); // May throw a warning

          // If there is no scheme (http, https, etc.) then it's
          // likely that parse_url parsed it incorrectly, so
          // prepend a scheme and try again. if we don't do this,
          // we may get "example.com/foobar" as our path.
          if (!isset($parts['scheme'])) {
              $value = $this->getOption('default_scheme')
                       . '://' . $value;
              $parts = @parse_url($value);
          }

          // If it wasn't parsed, then something was wrong.
          if (!$parts) {
              throw new sfValidatorError($this, 'badform',
                            array('value' => $value));
          }

          // Validate that the scheme provided is valid
          if (!in_array($parts['scheme'],
                        $this->getOption('schemes'))) {
              throw new sfValidatorError($this, 'badscheme',
                            array('value' => $value));
          }

          // Ensure that the host was found
          if (!isset($parts['host'])) {
              throw new sfValidatorError($this, 'nohost',
                            array('value' => $value));
          } else {
              // Finally set the domain for the final, unified
              // verification.
              $domain = $parts['host'];
          }
      }

      // Convert the domain to an IP address
      $ip_address = gethostbyname($domain);

      // Unfortunately, gethostbyname's only response if it
      // fails, is returns the input $domain. Try to convert it
      // to a packed IP address. If that fails, then it isn't a
      // valid domain name.
      if (@inet_pton($ip_address)) {
          return $value;
      }

      // Didn't validate...
    throw new sfValidatorError($this, 'invalid',
                    array('value' => $value));
  }
}
?>

Install this into lib/validator/sfValidatorDomain.class.php

<?php
$form = new SomeFancyForm();
$new_choices = array('Selection 1', 'Selection 2', 'Selection 3');
$widget = $form->getWidget('select_widget_name');
$widget->setOption('choices', $new_choices);
?>

Of course, that can easily be adapted to modify any option on any form - just find the appropriate option you want to adjust. Additionally, you may need to adjust the validator as well.

Relevant Documentation:

• sfForm API Documentation
• sfWidgetFormSelect API Documentation

Unfortunately, Symfony’s documentation is notoriously sketchy, however this is verifiably functional.

Add the following to your app.yml:

all:
  sf_guard_plugin:
    success_signin_url:      '@my_route?param=value'
    success_signout_url:     module/action

The Dataset

The dataset that I’m ranking is much more complex, however the concept is still the same.

CREATE TABLE `response` (
	id INT(11) AUTO_INCREMENT NOT NULL,
	time FLOAT NOT NULL,
	meta VARCHAR(255),
	PRIMARY KEY(id)
);

Temporary Tables…Ish.

My first idea was to have a table that would be regenerated every $increment (probably an hour, maybe a minute when we were small.) These tables would be pretty simple, just:

CREATE TABLE `response_ranking` (
	id INT(11) AUTO_INCREMENT NOT NULL,
	response_id INT(11) NOT NULL,
	PRIMARY KEY(id),
	FOREIGN KEY (response_id) REFERENCES `response` (id)
);

And then in order to populate the database with the appropriate ranks, we would run:

-- Truncate the table
TRUNCATE TABLE response_rank;

-- Put the data into the table in order by time.
INSERT INTO response_rank (response_id, time)
SELECT id AS response_id, time
FROM response
ORDER BY time;

The SQL code would take the data from the response table in order of time, fastest to slowest, and then insert it into the response_rank table. The response_rank table’s ID would then directly be their rank amongst that dataset. The first record would be the fastest, the 257^th record is the 257^th fastest, etc. In order to find the rank for a particular record, the query is quite simple too:

SELECT id AS rank
FROM response_rank
WHERE response_id = ?;

Why This is a Bad Idea (tm)

For the number of records I’ll have and the size of the site, caching the data isn’t necessary. Adding that level of complexity to a smaller site isn’t worth the potential scalability it may provide.

How I Did It

Now this may not be the most efficient way to do it, however I imagine it’s much closer to a Good Idea (tm).

Instead of regenerating indexes or pushing massive amounts of data, let’s use a simple database operation which the database can be easily optimized to do, if it isn’t already by adding an index to the time column.

I present to you, The Better Way:

SELECT count(1) AS rank
FROM response
WHERE time < ?;

When you run the query, rank is exactly the rank within the current dataset, with nothing waiting to be aggregated into the database, or an index needing to be regenerated.

Now I know, this is stupidly simple, but it wasn’t extremely obvious to me or my coworker - so maybe someone else is stuck on such a simple problem too. However, this also may not be the most efficient way to do it either - if there’s a better way, I’d love to know.

The code I wrote It is based off of Say No To Flash’s filter that they wrote, however I was generally displeased with their method. t loops over the list twice, and is generally a garble of logic that is a little bit hard to understand, and could probably be done better.

I was planning on making this system more complicated by specifying if SSL was enabled or disabled by default, however that became too distorted in the code to make good, logical sense that was easy to read and maintain.

Edit your filters.yml File

Open up your apps/app_name/config/filters.yml file, and add in after the security filter:

sslFilter:
  class: sslFilter

This code tells Symfony to load and execute the filter you’ll be creating.

It should look something like this:

rendering: ~

security:  ~

sslFilter:
  class: sslFilter

cache:     ~

common:    ~

execution: ~

Edit your app.yml File

Open the apps/app_name/config/app.yml file, and add the following at the end:

all:
  ssl:
    strict: true
    modules:
      - { module: some_module }
      - { module: another_module }
      - { module: insecure_module, action: secure_action }
      - { module: insecure_module, action: another_secure_action }

In the previous code, the strict: true code forces strict checking. Strict

checking means that if someone accesses a URL that isn’t specified as secure via HTTPS, it will redirect them to the HTTP. If strict is set to false, it will allow them to access any module through HTTPS.

After that, the - { module: insecure_module, action: another_secure_action } code is how you set each module and action. If you want an entire module to be secure, don’t include the action section of that, however you need a line like that for every module and action you want to secure.

Create a Filter

Create a file in apps/app_name/lib/ named sfSslFilter.class.php, and put in it:

<?php
/**
 * @author Graham Christensen
 *          graham@grahamc.com
 */
class sslFilter extends sfFilter {
    public function execute ($filterChain) {
        $context = $this->getContext();
        $request = $context->getRequest();

        // Perform strict checking of security
        // IE: If it's HTTPS and shouldn't be, make it HTTP
        if (sfConfig::has('app_ssl_strict')) {
            $only_explicit = (bool)
                             sfConfig::get('app_ssl_strict');
        } else {
            $only_explicit = false;
        }

        // Get a list of all the modules to check for
        $modules = sfConfig::get('app_ssl_modules');

        // Set the modules variable to an array, this is
        // if it's not configured for this particular environment.
        if (!is_array($modules)) {
            $modules = array();
        }

        // Store the module name and action name into variables
        // to simplify the code, and reduce function calls.
        $module_name = $context->getModuleName();
        $action_name = $context->getActionName();

        // Check if the current request matches a security module
        // If the module or module & action is specified, then
        // ensure it's correctly set.
        $listed = false;
        foreach ($modules as $action) {
            // If the module name is listed
            if ($action['module'] == $module_name) {
                // If the whole module is listed, or the action
                // specifically
                if (!isset($action['action'])
                    || $action_name == $action['action']) {
                    $listed = true;
                    break;
                }
            }
        }

        $is_secure = $request->isSecure();

        // If modules have to be explicitly listed, it is
        // secure, and it's not listed - then redirect
        if ($only_explicit && $is_secure && !$listed) {
            return self::doRedirect($context);
        }

        // If it's not secure, and it's listed as having to be
        if (!$is_secure && $listed) {
            return self::doRedirect($context);
        }

        // Continue on with the chain, but it will only do that if
        // we didn't need to redirect.
        $filterChain->execute();
    }

    public static function doRedirect($context) {
        $request = $context->getRequest();
        $controller = $context->getController();

        // Determine which direction we want to go
        if ($request->isSecure()) {
            // Switch to insecure
            $from = 'https://';
            $to   = 'http://';
        } else {
            // Switch to secure
            $from = 'http://';
            $to   = 'https://';
        }

        $redirect_to = str_replace($from, $to, $request->getUri());
        return $controller->redirect($redirect_to, 0, 301);
    }
}
?>

The beginning of this is pretty simple, it just gets the configuration settings outlined in the app.yml file, and then goes to town doing it’s job.

<?
// Check if the current request matches a security module
// If the module or module & action is specified, then
// ensure it's correctly set.
$listed = false;
foreach ($modules as $action) {
    // If the module name is listed
    if ($action['module'] == $module_name) {
        // If the whole module is listed, or the action
        // specifically
        if (!isset($action['action'])
            || $action_name == $action['action']) {
            $listed = true;
            break;
        }
    }
}
?>

That code goes over every module in the app.yml file, and sees if the current requests’ module matches, or if the module and action matches. If it does, it sets $listed to true and exits the loop. This tells the code that it is needs to ensure security on the current request.

<?php>
// If modules have to be explicitly listed, it is
// secure, and it's not listed - then redirect
if ($only_explicit && $is_secure && !$listed) {
    return self::doRedirect($context);
}
?>

That code ensures that if explicit checking is on and that the request is currently using HTTPS and it shouldn’t be, it is redirected to become HTTP.

<?php
// If it's not secure, and it's listed as having to be
if (!$is_secure && $listed) {
    return self::doRedirect($context);
}
?>

If the module/action is listed as needing to be secure and it isn’t, then perform the redirect.

<?php
public static function doRedirect($context) {
    $request = $context->getRequest();
    $controller = $context->getController();

    // Determine which direction we want to go
    if ($request->isSecure()) {
        // Switch to insecure
        $from = 'https://';
        $to   = 'http://';
    } else {
        // Switch to secure
        $from = 'http://';
        $to   = 'https://';
    }

    $redirect_to = str_replace($from, $to, $request->getUri());
    return $controller->redirect($redirect_to, 0, 301);
}
?>

The previous method is really pretty simple. If doRequest is called, it means that the current protocol is not valid. So, if it is HTTP, make it tHTTPS. If it’s HTTPS, make it HTTP.

Finishing Up

When you’re finishing up, make sure you clear your cache via ./symfony cc and then test it on your site. Make sure that your strict option is working, and you should check each individual module and action to make sure you didn’t forget anything.

Today I was working alongside Vid Luther and running into peculiar problems. Lo and behold, it was a minor edit that I had forgotten to test. Dammit. Company policy is pushups for fuckups, and so well.. you know.

I began thinking about switching back to Subversion for the pre-commit hooks, as I stupidly forgot that Git must have them too. (The lack of a centralized server threw me off.) I did a bit of research, found that, in fact, Git does have them - and in a way that I much prefer. Thus the development began of, essentially, a locally-run continuous-integration script to verify that none of the developers (including myself) screw anything up.

Behold, The Symfony Pre-Commit Hook For Git:

#!/usr/bin/env php
<?php
/**
 * @author Graham Christensen <graham@grahamc.com>
 * @license Here, just take it.
 */

// A list of commands you want to run on the entire codebase.
// You could also add in tests
$test_commands = array('./symfony propel:build-sql',
                       './symfony propel:build-model',
                       './symfony propel:build-forms',
                       './symfony propel:build-filters',
                       './symfony propel:insert-sql --no-confirmation',
                       './symfony propel:data-load');

// This is pretty static, since the git repository doesn't move
$repository_parent_directory = realpath(__DIR__ . '/../../');

// Where to put the testing directory. It cleans up after itself, I
// promise. Note: if the testing directory is inside the
// repository's parent directory, it might get stuck in an infinite
// copy loop.
$test_parent_directory = '/tmp/';

// Negotiate a temporary directory that doesn't exist yet, so
// it doesn't get in the way of anything already there.
$i = 0;
do {
    $test_directory = $test_parent_directory
                    . '/git_pre_commit_hook_' . $i++;
} while (file_exists($test_directory));

// Create a testing environment
debug('Copying the working copy from '
      . $repository_parent_directory . ' to ' . $test_directory);
mkdir($test_directory);

// the run_command function is used for easy debugging of what's
// actually being executed.
$pdir = escapeshellarg($repository_parent_directory . '/');
$tdir = escapeshellarg($test_directory);
run_command('cp -r ' . $pdir . ' ' . $tdir;

// Get into it, start running the test commands
chdir($test_directory);

// Iterate over all the commands. This has to happen one by one in
// order to catch the errors as they happen. Also, the debugging
// code is the same.
foreach ($test_commands as $command) {
    // Create error files within the testing directory so they're
    // cleaned up nicely
    $error_file = $test_directory . '/errfile_'
                  . md5($command . mt_rand()) . '.err_log';

    // Pipe ALL of the command's output to the file for convenient
    // error messages.
    $cmd = $command . ' > ' . $error_file . ' 2>&1';
    exec($cmd, $r, $return_code);

    // Symfony doesn't always return something other than 0 when
    // errors occur. Because of that you have to test for both
    // conditions, note that because errorsInLog is second - it is
    // only executed if the first one doesn't pass which saves time
    // if Symfony does what it should be.
    if ($return_code != 0 || errorsInLog($error_file)) {
        // because $error_code isn't always 1 when it fails, set it
        $return_code = 1;
        debug($command . ': Failue.');
        debug(file($error_file));
        break;
    } else {
        debug($command . ': Success.');
    }
}

// Delete the temporary testing location
debug('Removing ' . $test_directory);
chdir($repository_parent_directory);
shell_exec('rm -rf ' . $test_directory);

// Git reads the returning error code. At this point, $return_code
// is either set to 0 by succesfull executions, or 1 by a failure;
// so exit with the correct status.
exit($return_code);

// Check for errors on the provided log file. They don't
// necessarily have a standard format, so check a few things that
// are generally common.
function errorsInLog($logfile) {
    $emsgs = array();
    $emsgs[] = 'If the exception message is not clear enough, '
             . 'read the output of the task for more information';
    $emsgs[] = 'Some problems occurred when executing the task:';
    $emsgs[] = 'Aborting';

    // This could be optimized to fgets the file line by line
    foreach (file($logfile) as $line) {
        foreach ($emsgs as $error) {
            if (strstr($line, $error)) {
                return true;
            }
        }
    }
    return false;
}

// Handle debug messages, recursively if necessary.
function debug($message) {
    if (is_array($message)) {
        foreach ($message as $line) {
            debug($line);
        }
    return;
    }
    echo trim($message) . "\n";
}

// This function could easily be switched out to dump the exact
// command being executed, which can be quite handy.
function run_command($command) {
    shell_exec($command);
}
?>

Even If You’re Starving

• Write bad checks, even if you can promise that the money will be there.
• Under-value your services. If you are good, they will pay. If they won’t pay, chance are they aren’t worth it.
• Over-promise on what you can deliver. Do what they want, and later on see what you can do to improve their project.
• Under-deliver and let them be disappointed.
• Give unrealistic deadlines. A salesman might say its good for sales, but your development team will hate you, and you will lose them.
• Take flat-rate projects. These are screaming feature creep, and they have probably have no budget for it.
• Allow feature-creep for free. Feature creep is a natural part of a project. Clients don’t know everything that they want, and will refine and expand the project as it goes. This is ok, but not if they don’t pay for it.
• Let the client strangle you. If they can’t be pleased and won’t pay, make sure your company can survive to finish the project. Be willing to drop a client that is strangling your company, its not worth it.
• Let a good idea keep you from getting anything done. Just because something might be better long-term, doesn’t mean now is a good time to do it.

What You Should Do

Especially If You’re Starving

• Keep open conversations with your clients to make sure they’re happy. An unhappy client makes everything harder, try to keep communication open to understand their needs and desires, and to make sure that minor things aren’t becoming major.

Of course, the “Do” list is short because “boss” didn’t make very many good decisions in my opinion.

Well that’s a good question. For my purposes, this is what I need to know:

1. Create a Private Key. These usually end in the file extension “key” If you already have one, don’t worry - it’s cool, we’ll be using that one.
2. Create a Certificate Signing Request. These usually end in the extension “csr”, and are sent to the certificate authority to generate a certificate.
3. If you’re not going to be using an existing service (usually for pay) as a certificate authority, you can create your own Certificate Authority, or self-sign your certificate.
4. Submit your CSR to the CA and get the results. If you’re doing it yourself, I’ll tell you how. The CA creates a Certificate file, which ends in “.crt”.
5. Take the whole collection of files, keep them somewhere safe, and mash them together to create your PEM file (this is usually just used for email.)

So. Let’s get started, eh?

Step Zero: Basic Assumptions

• I’ll assume your domain name is domain.tld.
• I’ll assume you have OpenSSL installed.
• I’ll assume that you are running some form of Linux. I use Debian.

Step One: Create your Private Key

Ok, here you’re going to create your key - and treat is as such. This should be kept private, and not shared with anyone.

Now, you have a couple of options here - the first is to create your private key with a password, the other is to make it without one. If you create it with a password, you have to type it in every time your start any server that uses it.

Important: If you create your private key with a password, you can remove it later. I recommend creating your private key with a password, and then removing it temporarily every time you need to use it. When you’re done with the key without a password, delete it so it isn’t a security risk.

Create your Private key with a password

openssl genrsa -des3 -out domain.tld.encrypted.key 1024

Create your Private key without a password

openssl genrsa -out domain.tld.key 1024

If you created your private key with a password, you’ll want to complete the rest of the steps using a decrypted private key - else you’ll have to type in your password every time you use the certificate (ie: every time you start a daemon using that certificate.)

Remove the password and encryption from your private key

openssl rsa -in domain.tld.encrypted.key -out domain.tld.key

Step Two: Create a CSR

On this step you’re going to create what you actually send to your Certificate Authority. If you set a password with your Private Key, you’ll be required to enter it to create the CSR. After you finish all these steps, you can delete your CSR.

Create your Certificate Signing Request

openssl req -new -key domain.tld.key -out domain.tld.csr

Step Three: Create your Certificate

You have three options here:

1. Self-signing
2. Creating a certificate authority (CA)
3. Paying a CA to create your certificate for you.

Here’s what’s up: Self-signing is easy, free, and quick. Creating a CA isn’t terribly difficult, but probably more than you want to handle for something small. Paying for a CA can be cheap ($20), easy, quick, and comes with browser-recognition, which is generally important for public websites; especially commercial ones.

My advise: Self-sign your certificates for personal things, and pay for a certificate if its public and important.

If you’d like to pay for someone to sign your certificates, do some research and find which one you want to use. Next, find their instructions for submitting your CSR file.

Self-Sign your Certificate

openssl x509 -req -days 365 -in domain.tld.csr -signkey domain.tld.key -out
domain.tld.crt

If you do happen to want to setup your own certificate authority, check these resources out:

• http://www.g-loaded.eu/2005/11/10/be-your-own-ca/
• http://codeghar.wordpress.com/2008/03/17/create-a-certificate-authority-and-certificates-with-openssl/

Step Four: Creating a PEM file

A PEM file is used by many different daemons, however how to generate such a PEM file can be hard to come by. There are some complicated ways to build one, however I have had pretty good success with simply combining the .key and the .crt file together:

cat domain.tld.key domain.tld.crt > domain.tld.pem

Disclaimer

I am not an expert with SSL, which is exactly why I created this. This may not be accurate, YMMV, etc. Be careful. Also: Your .key is private. Keep that safe, with appropriate permissions. Make sure nobody else can access it, and do not give it away to anyone. If you have any insight, feel free to comment - I would appreciate them.

Sources

Just a thank-you to everyone that was kind enough to document this process.

• http://www.rapidssl.com/ssl-certificate-support/generate-csr/apache_mod_ssl.htm
• http://www.akadia.com/services/ssh_test_certificate.html

SELECT User FROM mysql.db WHERE Db = 'databasename';

That will retrieve all of the users that have access to databasename. Thanks to Elazar (on Freenode) for this solution.

Only issue with this, is it doesn’t list the users who have global permissions (ie: *.*). Any ideas on solutions that would include that? Or is there not a “simpler” way.