As part of this blog, and my plan to continually update the template and theme code, there'll be a number of occasions when I'll want to publish and share code examples and the modifications I'm making.

Frustratingly, this is hard to do in a user-friendly and helpful way without the addition of a plugin to render and present the code in an aesthetically and easily shared copyable manner.

As such, I've installed SyntaxHighlighter Evolved, one of many 'Syntax Highlighter' plugins available in the repository (which tend to be based upon Alex Gorbatchev's JavaScript package). The plugin allows me to easily add wrapping tags to code which I enter in the post editor, which results in nicely formatted, functionally enriched and shareable code snippets - producing something like this:

session_start();
if (!isset($_SESSION['count'])) {
  $_SESSION['count'] = 0;
} else {
  $_SESSION['count']++;
}

I'll be using this plugin periodically to reference the code for changes or additions I'm making to the site. At some point, I'd like to review other solutions and to potentially develop my own, as I'm not keen on it outputting two script files, a css file and a significant amount of inline code into my blog's header!

Running a blog – or indeed any website which allows people to submit content and comments – pretty much guarantees that you’re going to receive comment spam. You’ll likely find that comments are submitted on popular or prominent posts, but that these comments often don’t make sense, are off topic, or are simply illegible.

Since launching this website, I've received well over 100 obviously spam comments - and whilst this isn't the end of the world, I'd rather spend my time writing than digging through adverts for questionably effective pharmaceuticals.

The Challenge

Whilst it can be feasible to manage and manually delete small amounts of this kind of spam, this becomes impractical when dealing with large volumes of comments. It’s also not always as easy as you might think to identify spam content – the quality of spam ranges from illegible characters and broken text through to clever, targeted messaging; intelligent spam is often remarkably easy to miss. The time required to determine whether or not a seemingly legitimate comment may in fact be spam by, e.g., ensuring that the author's website appears to match their poster profile and contains legitimate content, becomes a significant time sink when trying to keep on top of large numbers of comments.

This is loosely on-topic, has a name, a face, and isn't obviously linking out to a spam domain - it could easily have slipped through if I was dealing with large volumes of comments and wasn't reading them closely.

As such, an automated solution which can identify and stop spam is a must-have. Beforejumping in to the deep end and picking plugins, systems and solutions, it’s valuable to understand what comment spam is, why it exists, and how to avoid some pitfalls.

Why does comment spam exist, and who are the spammers?

The most important thing to understand about spam is that it’s rarely random, and almost always has commercial intent. More often than not, this kind of spam is a SEO technique. The spammers are attempting to gain a large volume and variety of links pointing back to their websites in order to increase their authority, and in turn to sell links to other websites (including those of legitimate businesses) in order to increase their authority - the more websites upon which they can place links pointing back to their sites, or intermediary sites pointing back to their clients and contacts, the more performance and profit they can achieve.

Thankfully, the major search engines got savvy to this kind of manipulation very early on and introduced the much misunderstood 'nofollow' link attribute, which instructs search engines that the destination of the link isn't explicitly endorsed (or is explicitly not endorsed, depending on your perspective), and that its shouldn't be treated in the same way as a link from within, e.g., an authored blog post*. Contrary to popular belief, nofollow links actually tend to be 'followed' (or crawled) in the same way as other links on a page, but no value, context or authority is thought to passed through them in the same way that it might be through a normal link; though it should be noted that theory shows that the presence of nofollow links within an overall profile has a correlation with higher rankings - this may be a direct relationship, or simply a correlation with a more organic and naturally segmented link profile.

*This distinction was used historically to manipulate the flow of value through internal links within a website - a process known as 'PageRank sculpting'. This worked on the premise that, in line with a very basic model of PageRank, value flowed proportionately between all links on a page; and adding a nofollow attribute to links would result in proportionately more value flowing through the remaining links. Google announced in 2009 that this 'redistribution' of value had actually been removed from as long since 2008, and that the proportional value simply 'evaporates'. This again is over-simplistic, and based on a very simple PageRank model - the reality is likely to be vastly more complex.

Frustratingly, spam often leads to more spam - once you've received just a few comments, you'll likely find that significantly more will follow. The presence of spam comments can subtly alter the inbound and outbound link profiles of a site sufficiently for it to be easily detectable by the processes used by comment spammers to find likely prospects. The increased (albeit off-topic) content created by the spam comments is also liable to make the page look more on-topic in terms of what the spammers are looking for - if your authoritative post has content discussing illicit pharmaceuticals, you'll find spammers queuing at your door to add their links.

The example spam comment I've highlighted at the top of this post is particularly interesting, as the motivation isn't entirely clear. They're linking out to a Facebook profile (which upon investigation is either fake, or in no way associated with the pseudo-identity which presents the comment - in this case, it looks like the profile of a teenage Japanese girl), which would suggest that either they're working to promote that account itself (possibly as part of a multi-layered spam network where that profile in turn links to and promotes a subsequent URL), or they're looking to add a semi-legitimate comment in the hopes of increasing the relevance of the affected post page for terms they're looking to target - and they'll follow up with a second round of comments once the page appears to be more on topic for 'SEO Pack for WordPress' in the example above (and yes, I've probably just done their job for them).

I'm still not sure if this one is actually spam, or just poorly written - except that the domain is heavily promoting some kind of muscle building voodoo... That is some impressively targeted spam.

It should be noted that, historically, spam comments are rarely submitted by humans – the effort required to manually write, complete and submit comment forms across pages spanning thousands of websites (with no guarantee that your comments would get through any spam filters, or be accepted by website owners) is a huge manual overhead. Spammers are often software programmers who create bots, crawlers and scrapers in an attempt to find valuable websites and pages with comment forms, and to intelligently submit content through these forms.

More rarely, comment spam that isn't designed as an SEO tactic may simply be targeting your readers, attempting to incentivise them to click through to a targeted link, and to ultimately spend their money (all the while attempting to look like a legitimate comment).

Disclaimer: As a technically-minded SEO practitioner, I'm absolutely fascinated by this kind of tactic, the nuances of the approach, and the mechanics required to execute it at a scale which makes the production effort and maintenance overhead worthwhile... However, as an SEO practitioner with an aspirational outlook and business-consultancy focused approach, I do not endorse or encourage this kind of tactic.

Comment spam is not the kind of SEO I'd ever recommend to a client or to any legitimate business - moral and ethical considerations aside, it's my firm opinion that no matter how technically clever and scalable the approach, the return and value generated (both direct and secondary) can never be as high as the returns generated through undertaking strategic improvements to a business, website, content strategy, product/service offering and online marketing mix; even the best comment spam infrastructure will never rival a genuinely synergistic SEO and Social strategy which connects real consumers with a business, regardless of how many links those comments might acquire.

Creating a system capable of making this work is a pretty cool, geeky pipe dream, and a criminal waste of time; even if it makes you rich overnight (after months of solid development), there's an increasing chance that it will become utterly redundant in any number of ways just quickly, as Google focus increasingly on authorship, link quality and user signals.

Why does my WordPress blog get lots of spam?

As a blogger using WordPress as a CMS I've some considerable advantages and disadvantages when it comes to spam. WordPress' size and popularity means that there are all manner of tools, plugins and processes for managing spam, when makes resolving or burying the problem reasonably achievable; and more are developed, released and improved each day to tackle the latest challenges (much like an anti-virus software solution on your computer). However, consider that the primary mechanic of most comment spam systems is to find ways to inject comments through web forms - the proliferation of WordPress websites (which all share a universal back-end system, and where a vast majority of sites use off-the-shelf comment systems) means that generally speaking, if the spammers can work out a way to submit comment spam through a single plugin or website, they can roll it out in principle to the 15% of the Internet which is powered by WordPress (according to W3Tech's recent [Feb 2012] report on the utilisation of content management systems across the web). Comment spammers, especially in the case of WordPress sites, are working at incredible economies of scale.

W3Techs reporting WordPress as the underlying platform for nearly 16% of platforms - Feb 2012

This weakness or vulnerability of scale isn't just skin-deep, and take WordPress' greatest strength and use it against the platform.

From a development perspective, one of WordPress' key features is that every element of functionality and behaviour can be tapped into, controlled, modified, extended, changed or removed entirelyu through the use of 'hooks' - bits you can grab onto and take control. For example, there hooks for when posts are saved, when pages are viewed, when settings change, and when comments are submitted - and these hooks are universal 'under the hood'.

If I develop a plugin which changes the presentation, functionality and behaviour of the way in which my comments system works, chances are that I'll produce that plugin by building on WordPress' existing hooks - regardless of the look, feel, bells and whistles, nothing's changed at the very core of the system. As such, the vast majority of WordPress websites, regardless of their design, functionality or comments system utilise a single back-end system, which is designed to play nicely with whatever their plugins and addons want it to achieve.

If I know and can anticipate that WordPress is expecting to process a piece of POST data for a 'comment' field (i.e., somebody has filled in a textarea with an ID attribute of 'comment' and submitted a form), then it's almost irrelevant what the form looks and feels like, and how it functions - I can bypass the front end entirely. In fact, there are some scenarios where a page doesn't even need to be published and/or accessible in order to target it and submit spam comments to that post.

Essentially, all a comment spam system needs to be able to do is to anticipate what the WordPress comment system hooks are anticipating (in terms of POST data), and to submit that appropriately. Clever captchas, varying field layouts and ID attributes and other differences between individual websites, plugins and approaches are generally minor hurdles.

WordPress' flexibility makes it a wide, open target for this kind of spam.

How do I stop spam?

Intelligent spam is incredibly difficult to stop completely - given their economies of scale, commercial drive and myriad of attack vectors, the best you can hope is to attempt to stay ahead of (or, at least, in line with) the latest trends.

Machines might struggle to read this, but cheap labour sure can - and at scale.

Even solutions such as CAPTCHAs aren't fool-proof - though I've alluded to the fact that much spam is generated by software rather than humans, human labour can certainly form part of their automated spam attacks. CAPTCHAs are great at blocking automated comment spam, but the rise of cheap, digitally managed labour (such as that available from Mechanical Turk or oDesk), human-powered CAPTCHA-breaking can be easily integrated into an automated system at an incredibly low cost, where mass labour can complete CAPTCHAs at a rate of hundreds - if not thousands - per hour. Aside from the poor usability implications of this approach, it's by no means a robust enough solution to significantly reduce or solve your spam problem.

The increasing availability of this kind of labour is making human-generated and managed spam much more commercially viable; where machine-based spam is failing, simply hiring low cost labour to write spam by hand, to brief, and on topic is often more cost effective. This has worrying implications, and will become increasingly challenging to tackle and avoid.

So, what are the options?

Plugins to the rescue - Akismet and Bad Behavior

Akismet is shipped with all WordPress installations as standard, and it represents one-half of the most comprehensive but hands-free solution to spam that's easily available off-the-shelf.

It works against spam in much the same way that spam itself works - they maintain and evolve a central, scalable system which attempts to identify and intercept spam in increasingly intelligent ways, by monitoring the nature and behaviour of spam and attempting to continually keep up with and overtake the spammers. Every time a comment is submitted to your website, Akismet uses WordPress' comment submission hook to intercept the comment, check it against their spam detection algorithm and software (hosted on their servers), and determine whether it's spam or not - and process it appropriately in your admin area.

Akismet is massively underutilised, primarily due to its requirement for entering an API key. Whilst Akismet is free for bloggers, it isn't obvious on their website (despite them mentioning it frequently) how to get a free key. Registering an account with WordPress.com entitles you to a key as part of your account, but this journey isn't well signposted - and the perceived 'technicalness' of this process means that whilst all WP sites come with Askismet installed, it's rarely activated, and rarer still configured and active.

It should be noted that Akismet isn't limited to WordPress (and that there are a myriad of ways to tap into their servers, API and systems), but that usage outside of free, not-for-profit blogging beyond requires (and deserves) a licensed subscription.

Bad Behavior (note the American spelling) represents the other half of our magic solution - it operates in a completely different way from Akismet, and tackles spam by stopping it before it even arrives on your website.

Every time a user (or piece of software) requests a URL on a website, it must perform a handshake with the server - the requesting device presents it's credentials, exchanges details, and then proceeds on to load the webpage. Bad Behavior assesses these credentials and looks for anomalies, missing elements, or signals consistent with spam behaviour. As Bad Behavior's website points out, the development quality and adherence to web standards on spam systems is generally low, and generally tend to leave an obvious fingerprint in their server requests.

Aside from stopping the spam systems ever getting near your website, the fact that these systems and users aren't requesting resources, downloading HTML and images, and using your website means that you're saving a nice bit of bandwidth. Win/win.

(If you're installing Bad Behavior, don't forget to un-tick the 'show statistics in blog footer' option in the settings page, unless you're really keen on adding that information to all of your pages)

The combination of both Akismet and Bad Behavior means that you stop a significant chunk of spam at the door, and anything that gets as far as actually submitting a comment will be subject to rigorous testing and checking - anything that gets through probably deserves to get posted just for sheer ingenuity and perseverance.

Moral of the story

I'll admit that I've cheated on this one, and actually installed both of these plugins several days ago, as the half-dozen painfully low quality spam submissions that I was receiving each day was driving me mad - I've not had a single spam comment since.

I've been slow to publish a second update in this series, as many of the very basic additions and alterations which come next on the to-do list come with a lot of baggage; my decisions around which tracking and analysis packages to use, how best to integrate social functionality, spam prevention and general security improvements represent just a few areas where the actions I'd usually take on auto-pilot as part of a build are the result of years of experimentation, plugin vetting, experience and requirements matching to find the best approaches - as such, blogging about them necessitates delving quite deep into those areas and becomes something of a daunting task.

An extract from the WP Mail SMTP settings page

Thankfully, an easy escape has cropped up! I noticed that a few comments have been submitted to the blog, but that I hadn't received any email notification. This is because my hosting provider (eUKhost, for reference) disabled some of the core PHP mail functions (which WordPress relies on to send emails) some months ago in order to reduce the scope and quantities of spam activities originating from their shared hosting packages. There's a nifty plugin called WP-Mail-SMTP, which leaps to the rescue in this scenario by providing a highly configurable approach to using SMTP mail in place of PHP's mail() function.

WordPress is now capable of sending emails, so I'll receive a notification when new comments are posted; hopefully this will allow me to respond in a more timely manner!

One of the first decisions I make when installing a new WordPress site (and even prior to this step, if proper consideration has been given to the information hierarchy and content structure) is to configure the permalink structure. Configuring permalinks defines how WordPress produces, structures and manages the URLs of pages, posts and content.

An illustration of WordPress' template hierarchy

From a technical perspective, it's important to understand that regardless of the URL you type in, WordPress 'routes' any request through a central index.php file (which is handled via the default WordPress .htaccess configuration) which attempts to match the requested URL to one of the WordPress template files.

This means that as long as URLs have a predictable pattern, it doesn't matter what that pattern is - URLs can utilise any logical structure or pattern, including date-based systems (where, e.g., URLs might reflect the year, month and day of a post), author based approaches, or entirely custom structures.

Whilst WordPress provides a handful of reasonable defaults, without fail, I almost always elect to change from the default of '?p=n' (where n is a numeric identifier for the post) and to set a custom structure of /%category%/%postname%/.

Using /%category%/%postname%/ as a custom permalink structure

This creates URLs which output the name of the page, and if present/applicable, intelligently prefix this with any categorisation (or levels of categorisation) that page has. It changes the URL of this very post from ?p=16 to /blog-updates/permalinks-category-pagination/. You'll notice that I've also removed some redundant words from the string - 'change-1-configuring-permalink-types-category-pagination-fix-plugin' is a little unwieldy, and I don't strictly necessarily need all of those 'stop words' in place to accurately convey the core concepts of the post; I've also avoided stripping out so much information so as to leave the URL generic, and to risk having large numbers of very similar URLs addressing very different topics.

The /%category%/%postname%/ configuration is more adaptive than it might seem, and automatically aligns itself to different content types, requests and structures - date-based content (e.g., an archive page listing all posts from 2012) follow a date-based hierarchy even though there are no category or post-name components in the URL, and tag, taxonomy and custom structures all follow suit. '/%category%/%postname%/' essentially outputs a structured, hierarchical URL regardless of content type, request or page.

This approach has a number of advantages, not least of which is that is passes the 'gran' rule of thumb test, where an elderly relative should be able to follow instructions over the phone URL to enter a URL. At a very basic level, this is a much more comprehensible structure from a human perspective, and that should always be a key priority.

Secondarily, this approach allows for ease of identification of present location - regardless of where you are in the site, the URL acts as a clear breadcrumb and signpost; any URL with multiple levels represents a page which which has ancestors which reside at a 'level above' the current page, whereas ?p=n, or a date, author or other solution may not provide this clarity of location.

From an SEO perspective, this approach can be benefitial, as it contains relevant keywords in the URL and clearly demonstrates the location of the page in relation to other pages to search engines and crawlers. Whilst Google aren't fussed about the hierarchy of your content (and rather look much more closely at the internal linking and relationships between pages), it still helps to have a 'structured' hierarchy for housekeeping and to clarify the relationship. Interestingly, Bing (and Yahoo) do allude to having a 'maximum crawl depth', and suggest that they'll tend not to crawl deeper than 5 folders down. Whether this explicitly relates to the number of components in the URL, or the depth of the content when navigated down through (or a hybrid - or neither) they're not clear. Food for thought!

Analytics tools also benefit here, as many (including Google Analytics) provide facilities to analyse content 'silos' through 'drilldown' reports - these are all based on the premise of categorised, hierarchical content, and whilst these features can be configured to work with non-hierarchical or categorised URLs, it's a heft piece of work to get that kind of integration off the ground and continually maintained.

Conversely, this hierarchical approach does some times cause challenges by introducing a significant level of depth to URLs. Attempting to navigate deeply through site sections might result in complex or long URLs involving both categorisation and sub-categorisation (as well as pagination, date refinement and so forth), leading to results such as /blog-updates/subcategory-name/long-post-name-here/. This is arguably borderline on our 'gran' test, and in an ideal world would be much 'flatter' (i.e., it wouldn't contain the sub-category string, and/or the category, or perhaps even both); this is fine in principle, but the more we try to artificially manipulate the hierarchy the less maintainable the whole becomes, where each tweak and change results in an increasingly fragmented experience and less coherent hierarchy.

There is a wider argument here around whether a straight hierarchy is the best way to structure content. There is a strong school of though around date-based structures (e.g., /yyyy/mm/dd/post-name/), and a strong school of thought for completely 'flat' structures - there are definitely cases where these kinds of approaches makes sense (e.g., where content has a significant temporal or transient nature), but I'd argue that for a generic 'blog' approach that this is the best all-round approach.

Unfortunately, this type of permalink structure is somewhat inefficient from a technical perspective. Because all URL requests are routed through a central index file, the system (server and database) must work to process the URL and to 'map' it to the correct content. With the default URL structure this is easy, as, '?p=n' or indeed '?category=c' tells the file precisely which page, template and content it should return - however, if we're not explicitly identifying the post or category ID, the system must search for, find and retrieve the correct content with each request. On small sites this is may represent only a minimal performance hit, but on larger sites (which may have similarly named pages, category names which are similar to page names, and large databases) it can take some serious processing power to retrieve and serve the correct content, and it might not even always get it right.

Fortunately, WordPress are aware of these issues, and are continuing to release performance improvements which specifically address efficiencies in this area. This argument has historically been one of the very few compelling reasons not to make this permalink structure a standardised approach, as owners of large sites would find that as content built up, not only would it slow down as the system attempted to process the URLs, but that their sites might increasingly tend to return entirely incorrect content for some fringe case URLs (e.g., where posts have similar character strings, etc.).

One final drawback is that this approach breaks WordPress pagination within categories. Because the permalink structure is /%category%/%postname%/, any component after 'category' which isn't a subcategory is assumed to be a post name - however, in the case of paginated results (such as /category/page/2/), the word 'page' shouldn't be considered as part of the structure from a functional perspective - it's a human conceit to maintain the hierarchical, structured approach. Thankfully, there's an easy fix for this available in the form of the Category Pagination Fix plugin which simply rewrites requested URLs to get them to ignore the 'page' component.

Bingo, clean, optimal, (mostly efficiently) URLs.
I'm reasonably convinced that, with the exclusion of some fringe cases involving non-standard content, this is the best approach for URL structures on blog-like sites. What do you think?

Resources:

1. "Using Permalinks" from the WordPress Codex

As a WordPress and SEO fanatic, I spend a huge amount of time digging around with WordPress themes. I usually build from scratch - in fact, I'm fanatical about it. However, in order to get something moving here, I took a risk and found a free, minimal theme I liked the look of, and which did all the basics.

Whilst it's fine for now, I'm already finding things I want to change - and rather than just getting on with it, I thought that it might be an interesting exercise to blog about the changes as I make them.  I'll document each step I take, the rationale behind it (whether it's SEO, aesthetic, or otherwise), and provide some insight into the mechanics of how and why I've made the changes. Many of the tweaks, changes and functions I include in my WordPress work is subconcious, and I feel that it'd be valuable for to share my 'best practice' list, as well as to force myself challenge every aspect of it, including every habit and each assumption, on a continual and evolving basis.

The original Skeleton theme visuals

To kick off, and for some background, the theme I'm working from is a responsive layout called 'Skeleton', by simplethemes.com. It's minimal, functional, and I like the idea of making sure that the responsive approach is as top notch as it can be from an SEO, usability and accessibility perspective.

At this stage, all I've done is set up some categories, 'Blog Updates' for this series and 'Conjecture' as a catch-all for anything else for now. There'll be lots more exciting changes, so stay tuned.