Like many of you, for a long time I really couldn’t see the use of those URL shortener service thingies. Sure, when I was designing sites I tried to avoid long, ugly URLs, but I never saw slapping some random characters after a common base URL as being any more useful. I considered my awareness of the existence of these obscure services as an aberration induced by my geek genes, rather than validation of their existence or popularity.
Then came Twitter, and the world of URLs was never the same.
Twitter firmly swapped URL shorteners out of the occasionally useful into the pretty darn essential column. That magical 140 character limit, combined with the propensity of major sites to use URLs nearly as long as their software user agreements, thrust shorteners in front of millions of new eyeballs.
One issue, pointed out by more than a few security pundits and rickrolling victims, is that these shorteners completely obscure the underlying URL. It’s trivial for a malicious attacker to hide a link and redirect a user to any sort of malicious site. It didn’t take long for phishers and drive-by malware attacks to take advantage of the growing popularity of these obfuscation services.
Some of the more popular Twitter clients, like Tweetie, added optional URL previews to show users the full link before clicking through to the site. In part, this was enabled by shorteners like bit.ly enabling previews through their APIs. A nice feature, but it’s not one that most users enable, and it isn’t available in most web interfaces or even all standalone Twitter clients.
Bit.ly announced today that they are taking things one major step further and will soon be scanning all links, in real time, using multiple security services. Bit.ly will be using a collection of databases and scanning services to check both new and existing links as users access them. Websense’s cloud-based scanner is one of the services (the one that pre-briefed me), and bit.ly will use at least one other commercial service as well as some free/open databases.
Update: according to the bit.ly blog, VeriSign and Sophos are the other scanning/database engines.
In the case of Websense, bit.ly will tie directly into their content scanning service to check links in real time as they are added to the bit.ly database. Websense uses a mix of real time scans (for things like malware and certain phishing techniques) and their database of known bad sites. The system won’t rely only on the database of previously-detected bad sites, but will also check them at access time.
If a link is suspected of being malicious, Websense marks it and bit.ly will redirect users to a warning page instead of directly to the site. Users can still click through, and I’m sure plenty will, but at least those of us with a little common sense are less likely to be exploited.
Bit.ly won’t only be scanning new links added to the database, but will be checking existing links in case they’ve become compromised. This also reduces the chances of the bad guys gaming the system by adding a clean version of their site for an initial scan, then sneaking in malware for future visits.
I like bit.ly’s approach of checking existing links in case they get compromised, rather than only scanning new links as they are added. This will make it harder for bad guys to game the system. This solution is a lot better than the anti-phishing built into browsers and some search engines, since those rely only on databases of previously-discovered known bad sites.
It’s also a two-way system, and although Websense is being paid for the scanning, they gain the additional benefit of now leveraging the results once millions of new (and old) links start flowing through their service. Every bad website Wensense finds when a user submits a link to bit.ly is added to the database used by all their other products.
Finally, there’s nothing that says we’re only allowed to use bit.ly for Twitter. The entire Internet now gains a real-time security scanning service… for free. Have a questionable link? Shorten it through bit.ly and it’s scanned by Websense and at least one other commercial service, as well as all the free/open/cheap databases bit.ly uses (sorry, I don’t know what they are).
This isn’t to say that any of the individual scans, or all of them together, can identify every malicious link they encounter, but this is a significant advance in web services security. It’s a perfect example of cloud computing enhancing security, rather than creating new risks. Links sent through bit.ly will now be safer than the original links viewed directly.
This isn’t live yet, but should be by the end of the year.