Like many of you, for a long time I really couldn’t see the use of those URL shortener service thingies. Sure, when I was designing sites I tried to avoid long, ugly URLs, but I never saw slapping some random characters after a common base URL as being any more useful. I considered my awareness of the existence of these obscure services as an aberration induced by my geek genes, rather than validation of their existence or popularity.
Then came Twitter, and the world of URLs was never the same.
Twitter firmly swapped URL shorteners out of the occasionally useful into the pretty darn essential column. That magical 140 character limit, combined with the propensity of major sites to use URLs nearly as long as their software user agreements, thrust shorteners in front of millions of new eyeballs.
One issue, pointed out by more than a few security pundits and rickrolling victims, is that these shorteners completely obscure the underlying URL. It’s trivial for a malicious attacker to hide a link and redirect a user to any sort of malicious site. It didn’t take long for phishers and drive-by malware attacks to take advantage of the growing popularity of these obfuscation services.
Some of the more popular Twitter clients, like Tweetie, added optional URL previews to show users the full link before clicking through to the site. In part, this was enabled by shorteners like bit.ly enabling previews through their APIs. A nice feature, but it’s not one that most users enable, and it isn’t available in most web interfaces or even all standalone Twitter clients.
Bit.ly announced today that they are taking things one major step further and will soon be scanning all links, in real time, using multiple security services. Bit.ly will be using a collection of databases and scanning services to check both new and existing links as users access them. Websense’s cloud-based scanner is one of the services (the one that pre-briefed me), and bit.ly will use at least one other commercial service as well as some free/open databases.
Update: according to the bit.ly blog, VeriSign and Sophos are the other scanning/database engines.
In the case of Websense, bit.ly will tie directly into their content scanning service to check links in real time as they are added to the bit.ly database. Websense uses a mix of real time scans (for things like malware and certain phishing techniques) and their database of known bad sites. The system won’t rely only on the database of previously-detected bad sites, but will also check them at access time.
If a link is suspected of being malicious, Websense marks it and bit.ly will redirect users to a warning page instead of directly to the site. Users can still click through, and I’m sure plenty will, but at least those of us with a little common sense are less likely to be exploited.
Bit.ly won’t only be scanning new links added to the database, but will be checking existing links in case they’ve become compromised. This also reduces the chances of the bad guys gaming the system by adding a clean version of their site for an initial scan, then sneaking in malware for future visits.
I like bit.ly’s approach of checking existing links in case they get compromised, rather than only scanning new links as they are added. This will make it harder for bad guys to game the system. This solution is a lot better than the anti-phishing built into browsers and some search engines, since those rely only on databases of previously-discovered known bad sites.
It’s also a two-way system, and although Websense is being paid for the scanning, they gain the additional benefit of now leveraging the results once millions of new (and old) links start flowing through their service. Every bad website Wensense finds when a user submits a link to bit.ly is added to the database used by all their other products.
Finally, there’s nothing that says we’re only allowed to use bit.ly for Twitter. The entire Internet now gains a real-time security scanning service… for free. Have a questionable link? Shorten it through bit.ly and it’s scanned by Websense and at least one other commercial service, as well as all the free/open/cheap databases bit.ly uses (sorry, I don’t know what they are).
This isn’t to say that any of the individual scans, or all of them together, can identify every malicious link they encounter, but this is a significant advance in web services security. It’s a perfect example of cloud computing enhancing security, rather than creating new risks. Links sent through bit.ly will now be safer than the original links viewed directly.
This isn’t live yet, but should be by the end of the year.
Reader interactions
4 Replies to “Coming Soon: Bit.ly Adding Real Time Security Scanning for All Links”
@lonervamp
It’s a real time scan for malware, not just a database score check (I specifically confirmed that). They are using Websense and Sophos in that mode, and Verisign in blacklist/database check mode.
Supposedly the scans occur on an ongoing basis to pick up a newly infected, previously clean, site. Does that assuage your concerns? If it were just a blacklist check I’d consider it mostly worthless (like the anti-phishing in most browsers). The real time bit is what I think is interesting.
They’ll also hit you with a splash screen if it’s potentially bad, so you can always click right through.
Note that http://j.mp/ appears to be the new bit.ly.
Count me as still healthy-skeptical, but it is a step up from the current situations.
The initial question is whether Websense scans the sites for security vulns or “scans” a URL by comparing it against its internal database based on various reputation scores. I believe they do mostly the latter where a particular URL has a certain score, and they just obey that score. Somewhat beneficial and better than nothing, but still not really helpful against trusted sites with holes or new sites not properly scored. I don’t think they actually scan sites for vulns enough to be relevant.
…Although, if they do, remind me to submit your website multiple times and confuse your IDS/logging. 🙂
I’m a minority, but as a security-conscious person, I really only want to see the darn URL I’m being sent to. That really is the crux of the problem: not seeing the link.
I don’t need Websense making decisions for me. I’ll certainly expect plenty of false positives from security Twitterers who link to “hacking” sites that Websense flags. I know, I can click through. But more important I guess is the idea that I *am* a minority. I’d certainly rather my friends and family get that warning. So…touche; good move Bit.ly.
PS. I groaned reading the Bit.ly/Websense press release as it mentioned “cloud…”
Thanks, Rich! I did think twice hoping you had confirmed those things before writing them. 🙂