We use and recommend the Aweber email autoresponder, but for some time we’ve been facing a recurring bug.
When you include an external link in your Aweber message, Aweber runs an automatic link checker that is meant to highlight invalid links.
Invalid inks would be reported like this:
“Invalid Links: The following link(s) in your message were unable to be loaded. They may contain a misspelling or the server on which the pages reside may have taken longer than 20 seconds to respond to the request. [ ] Ignore invalid links and save message”
This error would regularly pop up on links that were valid AND fast. If you are confident the links are valid then you learn to ignore them. But if you are learning how to use Aweber (maybe because you are just starting out with email marketing), error messages that are flat out wrong are the last thing you need.
Initial attempts to sort out what was happening via the help desk didn’t resolve the issue. Fortunately Justin Premick was able to get a bit more information for us, and using this we were able to figure out what was going on.
Aweber’s link validator is requesting header information via curl, sending a request with a blank referer:
HEAD / HTTP/1.1 User-Agent: AWeber Communications Inc. Link Validator - For info, visit http://www.aweber.com/faq/questions/526/ Host: om4.com.au Accept: */* Referer:
Our own sites (and a lot of our client sites) run the popular Bad Behaviour plugin to help reduce comment spam. Bad Behavior was picking up the blank referer and returning a 403 Forbidden status. And yes, referer is a misspelling and yes, that is the way it is in the RFC.
So, if you have found this post because you’ve been getting this error, there are a few options:
- Aweber might update their link validator to send a referer string [update: maybe the referer shouldn’t be there at all as the validation checking is originating from a password protected page] (for a company that doesn’t like email spam, it seems strange they might accidentally leave the referer blank, but I’ll leave that to experts in this area to figure out)
- Bad Behaviour might decide to whitelist Aweber
- You can ignore the invalid link reports
- You can modify your local version of Bad Behaviour to whitelist the Aweber calls
Update 15 Aug 2009: AWeber has listened to our suggestions and has now fixed their link checker so it no longer sends a Referer field at all. Well done AWeber!