Arduino Traffic Lights #1: Introduction

Introduction

January will be my last month working at Ibuildings. During my time at Ibuildings I have worked on a lot of interesting challenging projects, made a lot of new friends, learned a lot from my co-workers and hopefully they learned something from me too. But now it is time for me to master a new set of programming languages, move to Amsterdam en meet some new people. Next month I start working at booking.com, which is mainly based on the Perl programming language. But before leaving Ibuildings I am going to build a cool gadget for the new bigger office location in Utrecht, where I have been working. The gadget will be a traffic light system, which displays the status of the latest build of the Continuous Integration server (Atlassion Bamboo).

About the project

I will create four blog posts about the development of this traffic light project. This first post contains the project introduction, scope and architecture. The second post will be about programming the Arduino micro controller board. The Arduino will contain a small web server for interfacing with the lights. Using HTTP GET parameters the lights can be switched to on/off individually or all at once. The third post describes the electronic circuit of the system. Two circuits have been built for the system. We start with a very simple set-up, without the relays and lights in order to program the controller. Next the electronic circuit will be extended with relays in order to switch lights with the required voltage. In the final set-up the system will be placed in a nicely fitting box. The last post will be about integrating the Arduino traffic with a Continuous Integration server. It contains a how to setup the integration for Jenkins (personal usage) as well as Atlassian Bamboo (used at Ibuildings).

Arduino Duemilanove

Arduino Duemilanove

The final system consist of several components. The traffic lights will be controlled by an Arduino connected to an Ethernet module. It will contain a small web server which listens for some HTTP request arguments (example: green=on). It should display a small page containing some buttons for switch the lights easily using a browser.

With my colleagues we agreed that the micro controller should only be a very simple device for switching on/off the lights. This way the traffic light device can also be used for other purposes in the future. The interaction with the CI server will happen on a regular computer with PHP5 installed. It possibly can show extra information about the last build such as the name of the project, time, duration, etc. In the case of Ibuildings we cannot set-up a trigger on the CI server, because it is not located within the internal network. Therefore we need a polling mechanism for checking the latest build. The diagram below illustrates how the different components are connected with each other.

System structure

System structure

The internal network consist of a switch/router, the CI server poller and the network connected traffic light. The CI server poller regularly polls the CI server located on an external network. In case the traffic light and the CI server are on the same network, it is possible to control the traffic light directly from the CI server by installing an extra trigger.

Source code

The software used for this project will be released open source. The project source code will exist out of the arduino controller code, CI server poller code and some documentation for setting up the electronic circuits. The code source can be found on GitHub: https://github.com/DirkEngels/ArduinoTrafficLights. In the next few days I will publish the rest of this serie of blog posts explaining how the following elements are created.

  1. Introduction
  2. Programming the Arduino Controller
  3. Putting together the electronics
  4. Creating the poller for the CI server
Share/Bookmark
Posted in Arduino, Open Source, PHP, Projects | Tagged , , , , | Leave a comment

PhpRestService: Simple PHP5 REST framework

Once again I started another open source project. This project focuses on setting up RESTful webservices with PHP. The REpresentational State Transfer (REST) protocol is a set of architectural principles for designing RESTful webservices. The term representational state transfer was introduced and defined in 2000 by Roy Fielding in his doctoral dissertation.

According this Packtpub arcticle called Resource Oriented Clients with REST principles, the key REST principles include:

  • The concept of resource (for example, a document is a resource)
  • Every resource given a unique ID (for example, document URL)
  • Resources can be related (for example, One document linking to another)
  • Use of standard (HTTP, HTML, XML, JSON)
  • Resources can have multiple forms (for example, status of a document, updated, validated, deleted)
  • Communicate in a stateless fashion using HTTP

Another key aspect of RESTful webservices is using HTTP status codes to communicate the result of a certain operation and negotiate about the representation. Following these principles correctly makes it easy to implement cache and/or a layered architecture.

Before starting a new project I did a bit of research of current REST frameworks available for PHP. First of all there is the Zend Framework with a Zend_Rest_Service, which does not follow all of the REST principles. This class wraps around an existing class aiming to make it RESTful. Another solution from the Zend Framework is the Zend_Rest_Route component. It exists of a skeleton controller with the http methods as actions. My personal problem with this solution is that it does not make a difference between a collection and an item. The Recess framework also contains a solution for creating restful webservices, but it is packaged with a complete framework for setting up web pages.

PhpRestService will be a library for setting up rest services quickly. It will make use of HTTP status codes and headers for communicating the result of an action and selecting the preferred representation of a resource. Many things will be configurable by editing configuration files without touching any code. The project structure is almost identical to the one of PhpTaskDaemon, containing a library and application directory. When using the library, the user creates a set of resources in the application directory by defining, the one of the HTTP methods: get, post, put, delete, head and/or options. A distinction is made between a collection (/blog) and an item resource (/blog/<id>). The development progress of this project can be found on GitHub:  https://github.com/DirkEngels/PhpRestService/

The end result will contain two sample applications; the first is a simple blog application, where as the second one displays the status of a PhpTaskDaemon instance. I will look forward to your comments and remarks regarding this project.

Posted in Open Source, PHP, PhpRestService, Projects | Tagged , , , , , , | Leave a comment

Dutch PHP Conference 2011

Last weekend I went to the Dutch PHP Conference 2011 (DPC11) in Amsterdam.  The first day, Thursday, was the tutorial day, where you could choose between four workshops. On Friday and Saturday talks were given by speakers from all around the world speaking about the latest trends and technologies about or related to PHP. In this blog post I will mention a few talks, which I found very interesting and/or enjoyable.

Friday started with the keynote of Aral Balkan ‘The Art of User Experience: Making beautiful, delightful, fun things’. In this session, Aral Balkan outlined the important role that user experience design plays in the making of virtual products. One of the first talks I visited after that was the talk of Paul Matthews about Solr called: ‘Searching with Solr – Why, When and How‘, in which he showed some very nice features of Solr. David Zulke presented a talk about ‘Large-scale data processing with Hadoop and PHP‘. Also my colleague Evert Pot presented a talk about REST called ‘Do you think you know REST?‘. Finally I will mention the talk by Alistair Stead called ‘Cooking up your development enviroment‘.

There was also a DPC unconference, which is a fourth additional track without a predefined schedule. The speakers in this track are the conference attendees and are scheduled on the day itself. (You have to be very early in order to reserve a slot, because it is full within no time). I was able to get a slot of 15 minutes the second day to present a my talk about PhpTaskDaemon and PlugWeb called: ‘Tasks, Daemons and Lightbulbs‘. I would like to thank all people who visited my talk for being there and also for the feedback I received to improve my presentation skills. Also I had some nice discussion about new ideas for the PhpTaskDaemon project .

Posted in Personal, PHP, PhpTaskDaemon, PlugWeb | Tagged , , , | Leave a comment

PlugWeb: Creating the driver

In the previous post I introduced the concepts for the PlugWeb application. In this blogpost I will cover developing the driver for the plugwise network.

The plugwise network
The plugwise network consist of a set wireless devices communicating with each other using the zigbee protocol. The devices are called Circle’s and each network contains one special device, called a Circle+. Communication between the Plugwise network and a computer can be done using the Plugwise USB Stick, which is essentially a serial device.

Device commands
Commands can be sent to the network to switch on/off devices or gather device information. Below are the actions that will be implemented in the PlugWeb system.

  • Device Info: This command retrieves information about the device.
  • Power Info: This command retrieves information the current power consumption.
  • Switch On/Off: This command switches on/off devices.
  • Calibrate Device: This command gets the calibration information needed for calculating historical power consumptions.
  • Historic Usage: This command retrieves information about the historic power consumptions.
  • Init Stick: This command initialises the connection between the usb stick and the Plugwise network.
  • Scan network: This command discovers scans the network for devices.
  • Register device: This command attachs a device to a network.

Plugwise protocol

Driver classes
The driver consists of the following classes:
* Network: This class is the public api for a plugwise network.
* Device: This class is the public api for a plugwise device.
* Serial: This class sends/reads data to/from the plugwise usb stick (serial device like /dev/ttyUSB0).
* Request: This class builds the request string for sending to the serial device.
* Response: This class interprets the received information from the serial class and returns nicely formatted data.
* Utils: This class is used for static methods used in other parts of the driver.

Using the driver
Below is an example how to use the driver for a single device.

{code type=php}
$mac = ‘XXXXXX’;
$device = new \Plugwise\Driver\Device($mac);
$powerInfo = $device->getPowerInfo();
if (isset($powerInfo)) && ($powerInfo[‘state’]==1)) {
$powerInfo->switchOff();
} else {
$powerInfo->switchOn();
}
{/code}

The example below illustrates how to use the driver for querying the plugwise network.

{code type=php}
$network = new \Plugwise\Driver\Network();
$devices = $network->getDevices();
foreach($devices as $device) {
$device->switchOff();
}
{/code}

Example action: Device info
A single action will be shown in more detail; retrieving device information. The action is accessible through deviceInfo method of the device class:

{code type=php}
public function deviceInfo() {
$input = $this->getRequest()->actionDeviceInfo($this->_mac);
$this->getSerial()->sendData($input);
$data = $this->getSerial()->readData();
$out = $this->_formatData($data);
return $out;
}
{/code}

The first step is to create the input string, which is handled by the request object. After that the input string is sent to the serial port and reads data from the serial port. A serial class handles communications with a serial port. The last step interprets the data from the serial port and converts the information into a usable array.

Building the input string
Below is the method actionDeviceInfo of the request class. The _renderActionString method used here finishes the input string by adding a checksum.

{code type=php}
public function actionDeviceInfo($mac) {
// 0012 000D6F0000 729310
$str = self::ACTION_DEVICE_INFO . self::MAC_PREFIX . $mac;
return $this->_renderActionString($str);
}
{/code}

Reading the response
The readString method of the request class reads the first four characters of the string which represents the plugwise device action. Based on the device action the appropiate method for the input string will be invoked. Below is the method for reading the input string of a device info response.

{code type=php}
protected function _processDeviceInfo($string) {
if (strlen($string)!=66) {
return array(‘error’ => ‘Wrong length of Response::DeviceInfo string’);
}
// Prepare output data
$data = array(
‘sequenceNumber’ => hexdec(substr($string, 0, 4)),
‘macAddress’ => substr($string, 14, 6),
‘clockYear’ => hexdec(substr($string, 20, 2)),
‘clockMonth’ => hexdec(substr($string, 22, 2)),
‘clockMinutes’ => hexdec(substr($string, 24, 4)),
‘logAddress’ => hexdec(substr($string, 28, 8)),
‘currentState’ => substr($string, 36, 2),
‘currentFrequency’ => substr($string, 38, 2),
‘currentVersion’ => substr($string, 40, 12),
‘currentFirmware’ => substr($string, 52, 8),
‘unknownParam’ => substr($string, 48, 2),
‘otherParam1’ => hexdec(substr($string, 50, 12)),
‘crcString’ => substr($string, 60, 4)
);
return $data;
}
{/code}

Getting the source
The source is available on GitHub as a seperate project: PlugWeb-Driver. The driver library will be included within the PlugWeb application.

Future blog posts in this series
Next time I will blog about the following aspects of the PlugWeb system.
* Plugweb introduction
* Creating the driver
* Unit testing the driver
* Designing the sqlite database
* Creating background daemon
* Creating a service layer with zend framework
* Designing the website interface with javascript/css
* Creating a PlugWeb Android application

protected function _processDeviceInfo($string) {
if (strlen($string)!=66) {
return array(‘error’ => ‘Wrong length of Response::DeviceInfo string’);
}

// Prepare output data
$data = array(
‘sequenceNumber’ => hexdec(substr($string, 0, 4)),
‘macAddress’ => substr($string, 14, 6),
‘clockYear’ => hexdec(substr($string, 20, 2)),
‘clockMonth’ => hexdec(substr($string, 22, 2)),
‘clockMinutes’ => hexdec(substr($string, 24, 4)),
‘logAddress’ => hexdec(substr($string, 28, 8)),
‘currentState’ => substr($string, 36, 2),
‘currentFrequency’ => substr($string, 38, 2),
‘currentVersion’ => substr($string, 40, 12),
‘currentFirmware’ => substr($string, 52, 8),
‘unknownParam’ => substr($string, 48, 2),
‘otherParam1’ => hexdec(substr($string, 50, 12)),
‘crcString’ => substr($string, 60, 4)
);
return $data;
}

Posted in Open Source, PHP, PlugWeb, Projects | Tagged , , , , , | 6 Comments

Visited the monthly PHPBenelux monthly meeting

Last Tuesday I went to the monthly PHPBenelux meeting after work. This time the meeting was held at Enrise located in Amersfoort. Look on the PHPBenelux agenda for more information about when and where the meetings are.

The first talk by Joshua Thijssen consisted of 15 pro-tips for MySQL users. He started with some basic tips every respected mysql user must/shoudl know, but are forgotten many tips, such as analyzing your table after importing a lot of data. Near the end where some more complex examples of performances consequences of using certain methods. A simple example would be using non-deterministic functions (as NOW()), which can not be cached.

The second talk was about WebDav and in particular SabreDav by Evert Pot. He started with a short introduction of what WebDAV is and what is consist of. After that he introduced one of his own projects called SabreDAV. SabreDAV is a PHP library for implementing WebDAV in your PHP application. The code is object oriented and easily to integrate in existing software in order to expose your application data as a generic network filesystem.

Posted in Algemeen, Personal, PHP | Tagged , , | Leave a comment

PhpTaskDaemon: Configuring, starting and monitoring the daemon

Last post I talked about the defining Tasks using the abstract Task class of the PhpTaskDaemon project. I also provided a simple example task implementation. This time I will go into how the daemon can be started, stopped and monitored. Although most of the options are already implemented and available in the develop repository of the project, some are still on the todo list. (specifying command line options such as a different configuration and log file.

The command line utility
The command line utility provided by PhpTaskDaemon is used to start, stop and request the status of the daemon. The following command shows the help message of the PhpTaskDaemon daemon.

{code}
./bin/phptaskdaemon –help
Usage: daemontaskdaemon [ options ]
–config|-c [ <string> ]  Configuration file (defaults: /etc/{name}.conf, {cwd}/application/config/{name}.ini)
–logfile|-l [ <string> ] Log file (defaults /var/log/{name}.log, {tmpdir}/{name}.log)
–tmpdir [ <string> ] Temp directory (defaults {cwd}/tmp/ or /tmp/)
–taskdir [ <string> ] Task directory (defaults {cwd}/application/daemon/)
–daemonize|-d            Run in Daemon mode (default) (fork to background)
–action|-a <string>      Action (default: start) (options: start, stop, restart, status, monitor)
–verbose|-v              Verbose
–help|-h                 Show help message (this message)
{/code}

Starting and stopping the daemon
The command below starts the daemon in verbose mode. Stopping and restarting the daemon can be done by replacing the action with stop or restart.

{code}
./bin/phptaskdaemon –verbose –action start
{/code}

A default init script is provided to start the daemon each time the host computer boots. Edit the path of the daemon executable and save it as /etc/init.id/plugweb.

Monitoring the daemon
The PhpTaskDaemon library also provides capabilities to monitor the activities of running task managers. The current activities of a running instance can be viewed using the status or monitor action. The status action displays the current managers and their activities once, while the monitor action displays and refreshes the status continuously. Use the following line to start the monitor function:

{code}
./bin/phptaskdaemon –action monitor
{/code}

Configuring the daemon
The daemon reads a configuration file with global options and manager specific options. By default the daemon reads the following configuration file: ‘application/config/phptaskdaemon.ini. It contains the locations of the logfile and temp directory. It can override manager specific properties such as the amount of forked processes to run for a particular task. See below for an example configuration file (docs/files/phptaskdaemon.ini-dist).

Overriding configuration options with the command line
Command line arguments override the options in the configuration file. The locations of the log and configuration file, the log verbosity and the locations of the temp and task directories can be overridden on the command line. An example is shown below using the properties listed above and also limits the gearman and forked managers to spawn 3 processes at maximum.

{code}
./bin/phptaskdaemon \
–log custom.log \
–config custom.ini \
–verbose 9 \
–task-dir tasks/ \
–tmp-dir tmp/ \
–max-workers 3 \
–max-forks 3
{/code}

Running a single action once
It is also possible to run a single action once, so it can be easy integrated within other systems (such as triggering a single task by the linux cron itself).

{code}
./phptaskdaemon –once –task Example
{/code}

Installing the init script
The PhpTaskDaemon project provides an init script for starting the daemon at every boot. The daemon also responses on posix signals, which means it shuts down nicely. Installing the init script is easy by copying the default init script to your /etc/init.d directory. You need to specify the path of your application in the init script.

{code type=bash}
cp docs/files/initscript-dist /etc/init.d/<projectname>
edit /etc/init.d/<projectname>
{/code}

Displaying the daemon status on a website
The daemon class can be used to request status information of the daemon activities. A sample implementation of displaying the daemon status through web is provided in a seperate sub project: PhpTaskDaemon-WebStatus. The web application consisting of a single controller and a view script for a zend framework application. To install it you will need to edit and copy the docs/files/apache.conf to your apache directory.

Posted in Algemeen, PhpTaskDaemon, Projects | Tagged , , , , , , , | Leave a comment

Just back from the PHPBenelux conference

Last weekend I went to the PHPBenelux conference in Edegem near Antwerpen, Belgium. The conference begun at Friday and started with four workshops to choose from. First day of conference is divided in 2 parts. The 1st part is the tutorial morning followed by a lunch. The 2nd part is the conference in the afternoon. In this blog post I will list some talks I found very interesting.

The first talk I mention is called “PHP in the mobile ecosystem” by Ivo Jansch. He talks about the mobile revolution and looks at the role of PHP in this environment. (more info).

Next the talk “Harness the power of sed and awk” by Joshua Thijssen was not PHP related, but however very interesting and useful. He give some examples of problems developers faces from time to time, and are probably solved in the language the developer knows best. Next he gave an introduction of the unix tools sed and awk and some nice examples for solving the problems mentioned earlier in an elegant and efficient way using the two tools (more info).

Martin de Keijzer give a talk about continuous integration called “Improving code quality with continuous integration”. “Continuous Integration is the combination of any number of the available tools used to improve quality of both code and process.” In his talk he looked at the available tools and how these can be used to increase the quality of your software project (more info).

Another nice talk was called “Advanced CouchDB” by Sander de Graaf. He explained what CouchDB is and where it is made for. After a short introduction of CouchDB he showed up some cool things you can do with CouchDB (more info).

The last talk I will mention is the talk by Stephan Hochdoerfer called “How to build customizable multitenant web applications”. Multi-tenant web applications are applications sharing the same code base for a lot of client with different requirements. He shows some challenges and solutions for building muli-tenant web applications (more info).

The organisation of the conference was very good for which I would like to thank all crew members of the PHP Benelux conference for the great weekend. See you all next year!

Posted in Algemeen, Personal, PHP | Tagged , , , | Leave a comment

PhpTaskDaemon: Defining Tasks

Last blog posts I introduced the PhpTaskDaemon project and provided a current state of alternatives for running php in the background. This time I will provide the basic concept for creating PhpTaskDaemon. First of all I start with some (useful) examples for running php in the background. Before implementing the example using the PhpTaskDaemon project I will list one or more alternative ways to implement the task.

PhpTaskDaemon provides a more generic solution for defining and running php background. A single command is provided to start and monitor multiple tasks with different behaviour. The task abstract class of the project only defines how tasks are loaded from a queue and how a single task will be execute. The class also provides methods to inform the daemon about its current activities. The way task queues and loaded and executed will be handled by a manager. The project contains different default managers like a interval, cron, gearman and forked manager.

The daemon
The PhpTaskDaemon project includes a single script to start a daemon for running multiple tasks. By default it uses a single configuration and log file. This script is also capable of viewing the current activities of the daemon.

Defining the task class
The PhpTaskDaemon project includes an abstract Task class and interface for used implementing tasks. The interface only defines two methods: loadQueue and executeTask. The first method is used by the manager for loading a queue. The second method executes a single task once. It is used by the manager to execute all task in the queue. The manager class defines if these tasks are executed in sequentially, in parallel or as gearman worker. The abstract class provides a set of methods to inform the daemon about the current status during the execution of a task.

{code type=php}
class Dew_Daemon_Task_Example
extends Dew_Daemon_Task_Abstract
implements Dew_Daemon_Task_Interface {

static protected $_managerType =
Dew_Daemon_Manager_Abstract::PROCESS_TYPE_INTERVAL;

public function loadTasks() {
}
public function executeTasks() {
}
}
{/code}

The managers
The project is distributed with a set of default managers. The interval manager is the default. It loads all tasks, executes them sequentially and then waits for a number of seconds before restarting. The cron manager works the same, except it waits untill the cron time definition matches the current time. The forked manager can be used to run tasks in parallel. The cron manager loads all tasks and forks the process x times to execute tasks. When all tasks are finished it waits for a predefined amount of seconds before reloading the queue. The fourth manager can be used to start gearman workers. This manager does not use the loadTasks method, because the queue is handled by the gearman job server.  To change a task to run with a certain manager the following line can be used:

{code type=php}
static protected $_managerType =
Dew_Daemon_Manager_Abstract::PROCESS_TYPE_CRON;
{/code}

Simple task example
The most basic example, also included in the package, loads a set of tasks randomly and executes a single task by sleeping for a certain amount of time.

Loading a task queue
{code type=php}
public function loadTasks() {
$queue = array();
if (count($queue)==0) {
for ($i=0; $i<rand(0,10); $i++) {
array_push($queue, array(‘taskId’ => $i, ‘sleepTime’ => rand(1000, 500000)));
}
}
return $queue;
}
{/code}

Executing a single task
{code type=php}
public function executeTask() {
$inputData = $this->getTaskInput();
$randomString = substr(md5(uniqid()), 0,10);

for ($i=1; $i<10; $i++) {
usleep($inputData[‘sleepTime’]);
$this->updateMemoryTask(($i*10), ‘Task data: ‘ . $randomString);
}
return true;
}
{/code}

Blog posts in this series
This blog post is an item in a series of blog posts about the development of the PhpTaskDaemon project. Follow the real progress of the project on the GitHub project page. The following blog posts of this series have been published in the past:
* An introduction of the PhpTaskDaemon project
* The current state of running PHP scripts using the command line
* Defining Tasks (this post)

Next time I will blog about the following aspects of the PlugWeb system.
* Running and monitoring the daemon
* Building managers: shared memory, semaphores and sockets
* Creating a small web front end for monitoring the daemon

Posted in Algemeen, PHP, PhpTaskDaemon, Projects | Tagged , , , , , | Leave a comment

PlugWeb: System architecture overview

PlugWeb goals
The first goal of this project is create a component which is able to communicate with the plugwise devices in a linux environment. After this goal has been achieved it is time to use the component to create various scripts and interface. The features/wishlist of the project is listed below:

  • Historic usage information stored in the plugwise devices needs to be saved in a database to create nice graphs.
  • The user must be able to switch devices on/off using the web interface.u

System architecture overview
The PlugWeb system consist of several components. Below is an overview of the components and their relations.

Component: Driver
This is the most essential component of the system. The driver component handles all the communication with the Plugwise usb stick.

Component: Daemon
The daemon components is process running in the background for switching on/off devices on-demand and also for polling the devices periodically and saving the results in the database.

Component: Database
A sqlite database is used for storing historical device information.

Component: Models
The models form an abstraction layer between the software objects and the physical storage of these objects.  The component provides a way to define models for retrieving and storing domain specific information in a structured manner.

Component: Service layer
The service layer handles the database operations and the communication with the daemon. The website will make use of the service layer for retrieving information about the Plugwise network.

Component: Website
The website consist of three pages which all need authorisation.

  • Dashboard: The dashboard consist of a list of (all|favourite) devices showing the current power consumption which can easily be switched on/off.
  • Statistics: The statistics page consist of a chart, a table containing the data in the chart and a set of options to customise the chart and table data.
  • Management: The management page is used to edit device information and manage the area’s and types.

Optional Component: Mobile Application
Possibly in the future the project could be extended with an mobile (Android, iPhone) application to monitor and control your devices always and everywhere. Such mobile application should also use the service layer. The service layer minimises code duplication and enables easy future extensions.

The database
Below is the proposed database structure for the plugweb project. The main entity is a device, containing a mac address, name and descriptions. Devices can be grouped into types (lamp, tv, etc) and area’s (living room, bed room, etc). The daemon collects historical power usage information which is saved in the history table. The daemon also checks the states of devices regularly. State changes are saved in the log table. Users can create planning rules for switching devices on/off automatically. The planning is executed by the daemon, which stores information about executed actions in the action table.
In this post I provided the system architecture and database structure of the PlugWeb system. Next time I will blog about the concept and implementation of the plugwise driver.

Blog posts in this series
This blog post is an item in a series of blog posts about the development of the PlugWeb system. The following blog posts of this series have been published in the past:
* An introduction to the PlugWeb project
* Software architecture of PlugWeb system (this post)

Next time I will blog about the following aspects of the PlugWeb system.
* Creating the driver
* Designing the sqlite database
* Implementing the daemon using PhpTaskDAemon
* Creating the service layer with zend framework
* Designing the website interface with javascript/css
* Creating a PlugWeb Android application

Posted in Algemeen, PlugWeb, Projects | Tagged , , , , , , | Leave a comment

PhpTaskDaemon: Current state of running PHP scripts using the command line

The goal of PhpTaskDaemon is to provide an easy and usable API to define task loading and execution run. A command line utility will be provided to execute and monitor one or more tasks. This blog posts I will provide an overview of currect technologies for starting PHP scripts. Minimalist examples will be provided to show the usage of the different technologies. All examples will use the following two functions. The implementation of the two functions below aren’t any useful, but are needed for the examples.

The first method defines the execution of a single task.

{code type=php}
function executeTask($input) {
echo “Task started!\n”;
sleep (1);
echo “Task done\n”;
}
{/code}

The second function loads the tasks in the queue. For simple programs, normally run as cronjobs, this function will return a single array with task input.

{code type=php}
function loadQueue() {
$tasks = array(array());
echo “Queue loaded (” . count($tasks) . ” tasks)!\n”;
return $tasks;
}
{/code}

Running PHP From the command line
In order to run PHP scripts from a terminal the command line has to be enabled. PHP scripts can be executed through the command line by putting the PHP script filename as an argument of the PHP executable.

{code type=php}
$queue = loadQueue();
foreach($queue as $task) {
executeTask($task);
}
{/code}

{code}
php tasks.php
{/code}

To run the php script directly, the script has to be made executable and a SheBang needs to be added.

{code}
chmod +x tasks.php
./tasks.php
{/code}

Command line arguments for the script are stored in the $argv variable. The Zend Framework library contains a nice component, called Zend_Console_Getopt, for setting and handling command line arguments. For more information about running PHP scripts from the command line see the post PHP on the Command Line.

Linux Cron
The Linux Cron allows tasks to be automatically run in the background at regular intervals. Each cron job definition exists of six fields separated with a space. The five arguments define the time (minute, hour, month, year and day of the week) when to run the script. The sixth argument is the script itself. The following crontab line runs a script every workday (mon-fri) at 06:40.

{code}
6 40 * * 1-5 php /path/to/file/tasks.php
{/code}

For more information about the crontab see an article of Kevin van Zonneveld about scheduling tasks on Linux using crontab. For more examples see the article from the geek stuff: 15 practical crontab examples.

Daemon
WikiPedia defines a daemon as a programs that runs in the background, rather than under the direct control of a user; they are usually initiated as background processes. A big difference with using PHP for generating web pages is the time a script runs. Mostly a daemon is supposed to be running all the time. The script needs to be adjusted so it keeps running. A simple while(true) will do the trick. The script sleeps a single second each time after loading and executing tasks.

{code type=php}
while (true) {
$queue = loadQueue();
foreach($queue as $task) {
executeTask($task);
}
sleep(1);
}
{/code}

There are several alternatives to run a program in the background, such as adding an ampersand (&) to the end of the command. The program stops when the terminal is closed. This can be solved by starting the program within a screen. Another way is to solve the problem within the code by forking the process and exit the parent, while the child process will continue running.

{code type=php}
$pid = pcntl_fork();
if ($pid == -1) {
die(‘could not fork’);
} else if ($pid) {
exit;
} else {
while (true) {
$queue = loadQueue();
foreach($queue as $task) {
executeTask($task);
}
sleep(1);
}
}
{/code}

Another way for running php script as daemons can be done with the linux daemon commando. It provides several functionalities commonly used by daemon programs, such as logging and configuration.

Gearman
The gearman website describes gearman provides a generic application framework to farm out work to other machines or processes that are better suited to do the work. It allows you to do work in parallel, to load balance processing, and to call functions between languages. Below is an example of a gearman worker registering the executeTask method.

{code}
$gmworker= new GearmanWorker();
$gmworker->addServer();
$gmworker->addFunction(“executeTask”);
echo “Waiting for job…\n”;
while($gmworker->work()) {
if ($gmworker->returnCode() != GEARMAN_SUCCESS) {
echo “return_code: ” . $gmworker->returnCode() . “\n”;
break;
}
}
{/code}

The gearman worker object only needs to register the executeTask method, because gearman provides its own queuing mechanism. Adding a task to the queue can be done through a gearman client object.

{code}
$gmclient= new GearmanClient();
$gmclient->addServer();
echo “Sending job\n”;
do{
$gmclient->do(“executeTask”, “3”);
}
{/code}

For a more usefull example see Matthew Weier O’Phinney blog post about writing gearman workers in PHP.

Parallel execution
Some scripts can benefit from running in parallel. This is especially the case when the script has to wait for other resources (server). In a multi-core systems performance benefits can be achieved by running scripts in parallel. By splitting up the tasks and running multiple instances of a PHP script for each core such script can benefit from a multi-core environment. The process control function library provides the ability to fork a running process into a child and parent. Most common is to use the parent as a queue manager, which loads the queue and dispatches single tasks to forked processes. Processes are forked only when there are tasks in the queue. Another method is to control the queue from each child process. The child processes will run autonomously and will check for new tasks itself. The drawback of such approach is keeping the queues synchronised. Below is simple example of a script that forks itself for each loaded tasks.

{code}
$pids = array();
for($i = 0; $i < 3; $i++) {
$pids[$i] = pcntl_fork();
if(!$pids[$i]) {
// child process
executeTask();
exit();
}
}
for($i = 0; $i < 3; $i++) {
pcntl_waitpid($pids[$i], $status, WUNTRACED);
}
{/code}

For more information about forking PHP scripts, see an article about Process Forking with PHP on Electric Toolbox or view a presentation called PHP CLI: A cinderella story on SlideShare.

Conclusion
In this blog post I provided an overview of ways to run PHP scripts from the command line and/or as a background script. Each proposed method has its own advantages and disadvantages. A cronjob runs only once per minute and does not check if a process is already running. A daemon can be created to ensure a program is running continuously, the developer has to implement a queuing system himself. Gearman contains a queuing system and uses gearman workers to process the tasks. The gearman client needs to be used to add tasks to the queue. Forking processes can increase performance by running scripts in parallel. Managing and monitoring tasks is a difficult task when using process forks.

The PhpTaskDaemon projects tries to combine a set of the approaches above into a easy to use library. It provides a single start-up script for all background tasks and contains a lot of commonly used functionalities for creating daemons, such as logging. Next article I will blog about creating the managers and task definitions for the PhpTaskDaemon project.

Blog posts in this series
This blog post is an item in a series of blog posts about the development of the PhpTaskDaemon project. Follow the real progress of the project on the GitHub project page. The following blog posts of this series have been published in the past:
* An introduction of the PhpTaskDaemon project
* The current state of running PHP scripts using the command line (this post)

Next time I will blog about the following aspects of the PhpTaskDaemon system.
* Defining Tasks (updated 16 jan 2011)
* Running and monitoring the daemon
* Building managers: shared memory, semaphores and sockets
* Creating a small web front end for monitoring the daemon

Posted in Algemeen, Open Source, PHP, PhpTaskDaemon, Projects | Tagged , , , , , , | 1 Comment