Parse Amazon S3 Logs with PHP

Amazon let’s you turn on logging for S3 buckets to capture (lots) of information on what’s happening there.

I wrote a PHP script to download those log files and parse the HTTP GET requests into a database for easier analysis. The data could also be sent to another source if you preferred.

S3logs-php on Github

Example Usage:


<?php
require 'vendor/Autoload.php';

$cfg = require 'config/config.php';

$logs = new \S3LP\Logs( $cfg['aws'], 'logs/', 'storage/logs/' );
try {
  $files = $logs->download();
} catch(Exception $e) {
  exit( $e->getMessage() );
}

//Parse logs to array
try {
  $data = $logs->parseLogs($files, 'DELETE');
  if(!$data) {
    exit('No GET requests found in logs');
  }
} catch(Exception $e) {
  exit( $e->getMessage() );
}

//Save to DB
try {
  $db = new \S3LP\Db( $cfg['pdo'] );
  $inserts = $db->insert($data);

  echo "Inserted $inserts rows";
} catch(Exception $e) {
  exit( $e->getMessage() );
}

Get the code on Github.

WordPress Backups to AWS S3

Amazon Web Services S3 (Simple Storage Service) is a cheap and reliable way of storing data and is ideal for backups. Scheduling regular automatic backups of your WordPress website to S3 is pretty easy with a plugin, but it can be worth tweaking your AWS Credentials for better security.

This post will show you how to create a new user on your AWS account that has limited S3 permissions. It means if your site is ever compromised and the credentials stolen you’ll be in a far better position than having used your root AWS details! It’s also especially useful if you are managing backups of multiple client sites and do not want cross-access.

Step 1 – Create a new user with IAM in the AWS Console

  • Log into the AWS Console. Go to Services > Security & Identity > IAM
  • Create a new user (e.g. backup_myexample)
  • Copy and paste the Access Key and Secret somewhere; we’ll use those within WordPress shortly.

After creating your new user, go to their Policies and create a new inline policy. We’ll use inline, rather than group permissions so that each user you create (for backing up different websites) is isolated to their own S3 path.

Give the policy a name and paste and modify this Policy Document. Change my_awesome_bucket and my_directory to the bucket and path you’re using for these backups.

{
  "Version": "2012-10-17",
  "Statement": [
  {
  "Sid": "Stmt1441240868000",
  "Effect": "Allow",
  "Action": "s3:*",
  "Resource": [
  "arn:aws:s3:::my_awesome_bucket",
  "arn:aws:s3:::my_awesome_bucket/website_backups/my_directory/*"
  ]
  }
  ]
}

Your screen should look a bit like this
AWS IAM Policy

Step 2 – Install & Configure BackWPUp

  • Log into your WP Dashboard, go to Plugins > Add New Plugin and search for BackWPUp
  • Install it and create a new job. For testing you may want to do Database backup only, or list of plugins. This is much faster that a full site (Files) backup. Once you know it’s working setup a full site backup.
  • Set the backup to S3 Service.
  • On the S3 Service page select your Region and paste in the Access Key and Secret key from before.
  • Type in your bucket name and path to store the backups. It should match the IAM Policy Document

s3-plugin-cfg

Save your settings and run the job.

The plugin logs will let you know if it worked.

A few notes

  • The IAM Policy allows all S3 actions on the given S3 path. I was not able to get this plugin to work with more restrictive permissions.
  • The new S3 Standard-IA class is good for these backups. The storage cost is cheaper than the Standard class without sacrificing redundancy as with Reduced Redundancy Storage. The downside is that downloads of these objects are more expensive.
  • Remember to check your backups periodically

wkhtmltopdf 0.12 on Ubuntu 14.04

The Problem

wkhtmltopdf is a library for rendering PDFs from HTML or live web pages.

  • apt-get install wkhtmltopdf  will install the older 0.9 version, which isn’t compatible with the Snappy PHP wrapper
  • wkhtmltopdf 0.9 is also unable to render on ‘headless’ operating systems and needs another package to simulate a display. The newer 0.12 version doesn’t have this problem.

The Solution

Manually Install the ‘Static’ Binary from wkhtmltopdf.org

(NB: If you’re not a root or super user most commands will need to be sudo…)

Download the Ubuntu “Trusty” build.
At the time of writing this is wkhtmltox-0.12.2.1_linux-trusty-amd64.deb

wget http://downloads.sourceforge.net/project/wkhtmltopdf/0.12.2.1/wkhtmltox-0.12.2.1_linux-trusty-amd64.deb

Rename it something easy to type like wkhtml.deb

Extract it with this command

ar x wkhtml.deb
# if ar not installed run this:
# apt-get install binutils

Extract the binary

tar xf data.tar.xz

Move the binary file to /usr/bin

mv wkhtmltopdf /usr/bin/wkhtmltopdf

Now PDF Rendering Should Work

wkhtmltopdf http://example.com myTest.pdf

Laravel 5 Environment Config

Laravel 4 used a function that checked the servers hostname to determine the environment. Laravel 5 simplifies environment detection by having a .env file present in your project root.

.env.example is ignored by Laravel’s detection. You can fill it with the keys your application expects to act an an example for your own file.

.env is for the current environment

Add .env to your .gitignore file. It’s got to stay out of your repo so you don’t overwrite it each time you pull.
Copy/Rename .env.example to .env for each environment.

set the APP_ENV value within your .env file to tell Laravel where it’s running.

APP_ENV=local
APP_DEBUG=true
APP_KEY=123abc...
$app->environment(); //get the current environment
if($app->environment('local', 'staging')) { }  //test if local or staging env

You can retrieve your .env config values with the env() function.


$cfgValue = env('MY_SWEETAS_VARIABLE');

Web Hosting: VPS vs Shared

Everyone prefers a fast website. Even a few extra seconds spent loading each page can be a drag on your visitors attention. While there are a lot of factors that determine a website’s loading speed I recently ran a test of different web hosting environments on my own site.

The defacto hosting standard for many WordPress sites is a cPanel account on shared web hosting. This setup is ubiquitous and easy to manage. One alternative is a Virtual Private Server (VPS) with SSD storage. The trade off is that a VPS requires more knowledge to setup and manage, but the performance can be a big step up.

I ran a test, running this WordPress 4.0 website on reasonable (i.e. not bottom dollar) shared hosting account and a Vultr VPS with 768mb RAM and SSD storage and monitored response times.

Website Details

basic WordPress 4.0 site, without page caching. The VPS was running Apache 2.4 and the shared hosting on Litespeed HTTP.

Results

(Lower is better. Both servers are in Sydney and response time measured from the US)

VPS vs Shared hosting response

  • Shared Hosting: averaged 1583 ms with the slowest measured peak at 4602 ms
  • VPS SSD Hosting: averaged 448 ms with a slow peak of only 656 ms

The result was noticeably snappier pages on the VPS. The other take away is the consistency. On a shared server your response times can vary depending on what other sites on the server are doing. The VPS which has isolated resources gave far more consistent performance.

 

Detecting Environment in Laravel 4

Update: How to Get Environment in Laravel 5

Your PHP application should have different settings for its development and production environments. Here’s how you can set your Laravel 4 app to detect the environment, load the appropriate settings and keep installation-specific things like database credentials out of your Git repo.

Detecting The Environment

Laravel lets you provide arrays of hostnames it will check the server against to determine the environment.

Open bootstrap/start.php

$env = $app->detectEnvironment([
   'local' => ['MyMachineName', 'AnotherDevMachine'],
   'staging' => ['StagingEnvName']
]);

If unsure, use PHP’s gethostname() function to find your machine names.
This is the value to use inside the local/staging arrays

Laravel will default to ‘production’ if no other environment is matched.

 

Setting Environment Config Values

Laravel looks for .env.*.php files in the root directory (beside your composer.json and .gitignore files).

You will want to exclude these files from your Git repo. Add the following lines to your .gitignore

.env.*.php
.env.php

Then create an env file for each of your environments. Use .env.php for the production environment.
They should look something like this:

<?php

/**
 * Local Development
 */
 return array(
	'app.env'		=> 'dev',
	'app.debug'		=> true,
	'app.url'		=> 'http://example.dev',

	'mysql.host'	 => '127.0.0.1',
	'mysql.database' => 'bestDbEva',
	'mysql.username' => 'me',
	'mysql.password' => 'myTremendousPassword'

 );

The actual values are up to you. You can store whatever values you need within the application.

Using The Environment Values

Laravel config can now call getenv() to get the dynamic environment variable. For your DB config open app/config/database.php

'mysql' => array(
	'driver'    => 'mysql',
	'host'      => getenv('mysql.host'),
	'database'  => getenv('mysql.database'),
	'username'  => getenv('mysql.username'),
	'password'  => getenv('mysql.password'),
	'charset'   => 'utf8',
	'collation' => 'utf8_unicode_ci',
	'prefix'    => '',
),