Help! My developer won’t implement / wants to remove / objects to my redirects.

A client recently asked me to chip in on a debate they were having with an internal tech team about retiring old 301 redirects. Concerns had been raised about ageing and legacy CMS functionality which needed to be replaced, and as part of this process the IT folks pushed to remove (and not replace) the thousands of redirects which the system was powering. The only hint as to why they’d think this would be a good/necessary idea was a quote from the proponents stating that as Google had seen, visited and clearly indexed the destination URLs for some time, the redirects were no longer necessary, and their removal shouldn’t cause any harm. At best, they’d concede that the SEO guys needed to maintain a smaller list of redirects which were specifically set up ‘for SEO purposes’ (such as forcing http/s and lowercase, and a handful of specific page-level redirects), but that otherwise they’d need to ‘trim the fat’.

Of course, this instantly set alarms bells ringing – the business dominates their market, largely because of the performance of the destination URLs for the redirects in question, and so any negative impact whatsoever on the equity flowing to those pages could have huge real-world, commercial implications.

This is a type of scenario which I see frequently. There’s always huge reluctance from the IT/developer community to implement, support or maintain ‘bulky’ redirect lists either on a day-to-day basis or as part of website launches or significant overhauls, and it’s invariably a painful experience fighting to justify why “let’s just redirect the top 100 most visited URLs on the website” isn’t a viable option. Because it’s hard to quantify the tangible opportunity cost of failing to implement redirects, it’s an incredibly challenging argument to have, and often resorts to pleading for people to take leaps of faith.

Given how many times I’ve fought this corner in my own experience, I thought I’d go all out, and attempt to constructive a comprehensive argument which weighed indisputably in favour of maintaining legacy redirects. Here’s the (slightly edited to preserve anonymity) transcript.

Don’t ever remove redirects. There’s no reason to, and no benefit in doing so. The performance gloom and doom your developers preach is nonsense.

Don’t ever remove redirects. URLs still gets hit/crawled/found/requested years after it’s gone. This impacts equity flow and crawl quotas.

Don’t ever remove redirects. It gives a bad user experience, which impacts the bottom line.

There are a few core considerations from my perspective (from my agency experience, technical SEO knowledge, and Linkdex experience):

  • Tech teams are always highly skeptical of redirects, as I’m sure you’re well-aware. IThey’re often hesitant to implement even foundational numbers of global redirects (such as case-forcing, http/s, etc) due to either:
    • A lack of understanding/consideration of the user experience and/or the way in which Google discovers, crawls and indexes (and the way in which equity flows in this process)
    • Concern over server performance resulting from high levels of redirect lookups on each request
    • A desire to ‘trim the fat’ as a part of general housekeeping because it’s good practice to do that kind of thing in other scenarios and dev processes
  • In each of these cases, I’ve found that focusing on the user experience component is a good solution, where setting an expectation that any scenario in which a user hits a 404 error page has a negative commercial impact on the business removes a lot of the ‘SEO politics’ from the equation. You can even go so far as to quantify this through surveying (e.g., “would seeing an error page on our site affect how much you trust the brand?”) and ascribe a £loss value to each resultant 404 per hit, based on estimated impact on conversion rates, average order values, etc. This gives you lots of ammunition to justify the continued existence of the redirects in a context which they can understand and buy into.
  • This also goes some way to resolving the performance concerns; any small investment made into optimising the impact of large volumes of redirects (which is always marginal at most, and vastly less than they anticipate). I should note that I’ve personally handled and argued for implementing/retaining large redirect sets on a number of huge site launches/relaunches, and in each case experienced these kinds of reactions; I’ve never once seen any actual performance impact following the successful deployment of redirects numbering in the low to mid thousands (formed of mixes of static and pattern matching / regex rules). London Stock Exchange were particularly hesitant to implement ~1k redirects as part of a large site relaunch, but were surprised to see no measurable performance impact following their implementation.
  • If performance concerns are still a barrier, I typically resort to identifying equivalent performance opportunities, load speed improvements and technical optimisations which can offset the perceived cost; and there’s never any shortage of opportunities for simple things like improved resource caching, client-side rendering efficiencies, etc. If it’s helpful, I can assist/support on reeling off some suggestions in this arena, from hardcore back-end stuff to softer front-end stuff. If they can quantify the ‘damage’, it’s easy enough to offset.
  • It’s also worth considering that almost all of the major CMS platforms operate by looking up the requested URL against a database to find a pattern matched result; as such, depending on the approach to implementation, having a large list of redirects to check against needn’t necessarily represent an extra lookup, process, or performance hit. If your team are particularly sophisticated, emulating the load-balancer style approach with layers of high-level caching can further mitigate performance concerns.

In an ideal scenario, you’d have a solution in place which monitors how many times a redirect has been hit, and the last time/date at which this occurred; I use a system like this on a number of my own sites, and periodically review for and retire redirects which have been stale for over 6 months, and have a low overall hit count [I use the Redirection plugin for WordPress, it’s one of my favourite tools].
What I’ve found interesting from running this system for a number of years over multiple sites is that:

  • Any URLs which used to have any real levels of equity, links and/or performance keep getting hit for years.
  • Redirects which have been removed and return a 404 keep getting hit by search engines, indefinitely
  • I run ~4k redirects on one of my own sites at present, with no measurable performance hit for turning them all on/off
  • Removing redirects in these scenarios has a definite impact on SEO, even if this is only indirect, due to the impact being on the reduced discovery, crawlability, and crawl equity distribution resulting from a bot entering the site at a dead URL
  • People still link to dead content; much of the web is stale. We also know that urls which used to be linked to still maintain some degree of equity, so removal of any of these is liable to negatively impact the destination page’s equity.
  • Any marginal performance overhead for running this kind of system (the performance fears resurface here in the context of maintaining these kinds of logs) is vastly offset by the value retention and improved user experience

I think that crawl quota is a particularly relevant point for you guys; with a site of your size and scale, any activity which is likely to result in Google allocating lower crawl resource is liable to have enormous consequences to indexation levels and speed, which is obviously to be avoided at all costs.

I’d expect a site of your scale to want to maintain as many of these legacy redirects as possible, for as long as possible, until the impact of them is measurably and demonstrably detrimental and this cannot be offset elsewhere. Upwards of 6,000 sounds a lot, but I think that it’s a reasonable and realistic volume given your site, legacy, etc. Having said that, I’d definitely want to be putting processes in place in the future to minimise the likelihood of URL structures expiring, and planning for more future-proof’d patterning (obviously easier said than done!). Good practice suggests that no URL should ever change or die, which is a card you might be able to play, but I suspect that may lead to blame games around who-chose/designed-which-legacy-URL-structure, which may cause more harm than good at this stage!

If there’s continued pressure to ‘trim the fat’, I’d personally want to investigate, query and quantify every single rule which is proposed for deletion to understand whether there’s still anything linking to it, whether the URL has/had any social shares, and whether server-logs indicate that it’s been hit recently. This is something Linkdex could definitely help with, however, it’s likely to be a resource-intensive process and may not provide any value – even if the vast majority of URLs turn out to be ‘duds’, all of the above rationale around user experience and equity management still apply.

I wonder – as part of the migration process, is there any opportunity to combine any of the redirects into pattern matching rules? E.g., if there are 100 URLs with a shared folder root, it may be prudent to craft a single rule based on matching that pattern. This should reduce quantities significantly.

From a completely alternate perspective, if deletion is to proceed, I’d potentially consider returning a 410 rather than 404 status on the URLs in question. This may help in sending a stronger signal that, rather than a large section of your website having broken/vanished (and the negative connotations associated with this), that a deliberate decision has been made to remove those pages. I’m not convinced that this would make any realistic difference, but it feels like a low risk/effort which may help out.

In summary…
Hopefully some of this can help bolster your arguments.
As a predominantly technical person, I’ve often argued in favour of redirects from an equity-preservation perspective; it’s only in writing the email that I realised that the two biggest points here are the tangible impact on user experience, and the expected (and somewhat measurable) impact on crawl quotas. I think that next time I run into this scenario, I might not talk about ‘link juice’ or the difference between different HTTP status codes at all, and just focus on these elements. Who’d have thought?

SearchLove 2014 – “Turbo-charging your WordPress website”

Some of the key resources, tasks, tools and code referenced in my SearchLove talk. Let me know if there’s anything missing!

Skilling up at up hosting & server config

  • Get some ‘proper hosting’ (at minimum a Linux VPS, or something more advanced, or an Amazon EC2 T2 small instance).
  • Buy a domain (often easier to configure if/when it’s with the company you host with)
  • Point your domain at your hosting (configure nameservers & DNS)
  • Familiarise yourself with cPanel (demo) for managing your site(s), & phpMyAdmin (demo) for managing your database(s).
  • Install WordPress from scratch
  • Familiarise yourself with WHM (demo) for managing your server. Build your own Apache config via ‘EasyApache’, and tweak some key server settings via ‘Tweak Settings’

WordPress core

Must-have Plugins

Other recommended plugins

Resources for monitoring & measuring performance

  • Google PageSpeed Insights – don’t forget that a high score doesn’t mean a fast site. It’s just looking at whether you’ve jumped through hoops.
  • ySlow – Useful as a second opinion on score-based speed analysis.
  • Google Analytics – don’t forget to change your sampling settings to avoid slow mobile devices skewing the average!
  • Pingdom – on demand site speed testing based on request monitoring, and subscription uptime & speed monitoring packages
  • WebPageTest – more hardcore, technical site speed testing
  • iPerceptions – user experience feedback via sophisticated on-page surveying
  • NewRelic – server-level performance monitoring, error tracking, user satisfaction indexing (a bit tricky to install but worth the learning curve)

General front-end

  • Configure a CDN (either a third party, or your own subdomains)
  • Use subdomains to parallelise connections
  • Minimise the number of requests each page makes
  • Use CSS sprites
  • Minimise reflow and repaint

Expert stuff

  • Reduce reliance on ad-hoc calls to WP functions like the_title(); store commonly used variables instead
  • Manage erroneous requests which show up in GWMT and Redirection’s logs
  • Serve large, static resources from a CDN
  • Clean up plugin overheads
  • Optimise your media; file types, sizes, behaviours
  • Kill unnecessary templates
  • Tailor your site’s behaviour using custom post types and custom fields
  • Fix errors and notices by exploring your error_log file

Upgrading to SSL

Visualising the performance stack

  1. Hosting Type
  2. Server spec & config
  3. Theme / CMS
  4. Domain(s) config
  5. Media / Assets
  6. HTML & DOM (and CSS + JS)
  7. Reflow, repaint and FPS

Common 404/error hits you can catch with Redirection

  1. apple-touch-icon.png (and variants)
  2. favicon.ico (and variants)
  3. browserconfig.xml
  4. feeds which don’t / shouldn’t exist
  5. invalid page/date ranges
  6. broken internal links (and missing http links)
  7. alternate sitemap and meta data urls
  8. pages & images with weird, breaking parameters
  9. security probes

Digital Marketing by Numbers revisited

I presented a revitalised version of my approach on how to map organisational objectives to tangible and effective KPIs at last week’s Linkdex Think Tank, exploring how you can use the resultant framework to become truly data driven.

The biggest challenge in becoming truly data driven is one of language and process. My framework for aligning strategic business objectives to tactical activity and tailored KPIs ensures that senior stakeholders have confidence, that recommendations align to organisational goals, that efforts are measurable and produce tangible outputs, and that the whole process is simple and transparent. Tried and tested by some of the world’s largest organisations, the framework can transform companies and put data at the heart of the decision-making process.

Getting Around Finance – Keyword Research & Tagging

In order to win in competitive markets like finance, you need to understand the market, your audience, and the opportunities. You can only do this from an orbital view of keywords, language and behaviour – and to then zoom in to specific niches and areas of focus.

Stop thinking about ‘keywords’, and start thinking about keyword ‘families’ – groups of contextually related phrases which indicate similar need groups; from synonym phrases through to simple typos, there’s a world of insight you’re missing out on.

Measurefest – Data Layers 101

A whistle-stop tour through the concept of the data layer, why it’s not just techy-stuff, and some of the real-world applications and implications of adopting your own.

…featuring such exciting topics as ‘Hands-on tips and tricks for Google Tag Manager’, ‘Reducing your dependency on frustrating development challenges when al you want to do is get a tag live’, and ‘Doing really clever stuff with variables, classifying user types, and scoring behaviours’.