This is a genre, though not everyone agrees with it. @AshelynnS on twitter mentioned this genre because it is one her novels fall within and I agree with her. But here is more of an explanation and a few web sites to go with the topic.
Anti-romance has a few different definitions. One of these is one that the podcast Writing Excuses used for the anti-hero (which will be discussed further down as well). I don't trust the source that much but it's worth a little use at least. Wikipedia explains anti-romance as a story that has a self-doubting or apathetic anti-hero as the protagonist who fails in the object of the plot/journey. Another way it is put is the antithesis of romance, or opposite. http://en.wikipedia.org/wiki/Anti-romance
I like AshelynnS' definition much better. She explains that an anti-romance is a book that could be romance but it's one that doesn't end well. It doesn't have the typical desired happy ending. And yes, while some romance don't have happy endings according to Leigh Michaels in her book On Writing Romance, happy ending is preferred by far. So, it makes sense that anti-romance would mean the romance part of the book, which is key, doesn't end well. No happy wedding endings here.
Some examples of Anti-Romance books: Amazon Books, Anti-Romance
Not sure if I agree with the books chosen for it, but it's a start and more proof of the genres existence.
Anti-Hero / Anti-Heroine
This isn't a new idea, but it's one that not everyone understands. People don't always agree on what qualifies as an anti-hero. The podcast Writing Excuses had a recent episode discussing this topic.
The anti-hero might be a hero that is hard to like at first, unsympathetic. This is one look that some don't believe qualifies as an anti-hero but it's one that I've seen in a non-published draft from a critique group. It is hard to have that desired interest in the protagonist if at first they seem too unlikeable. This can look like an anti-hero but the biggest issue is keeping the reader interested. Some have succeeded but it's not easy. Wikipedia defines this as a character that doesn't follow the archetype for a hero.
Another look at the anti-hero is one from the Writing Excuses. http://www.writingexcuses.com/
Their discussion included: a hero who has the qualities of a typical hero for most of the tale/quest but at the end they do not complete their goal. They fail.
This is a hard sell for some because most people don't like seeing the good guy fail, though sometimes it's a relief to get a good dose of reality. But not that many want reality in their fiction, lol.
More explanations: What is an anti hero?
Examples of Anti-heroes or Anti-heroines:
Alex from A Clockwork Orange
Holden Caulfield from Catcher in the Rye
Rincewind from Terry Pratchett discworld series
Iago from Othello
This is sometimes viewed as an opposite to the anti-hero. The definition explains it as an antagonist that isn't solely evil or unsympathetic. Includes villains that are defined as such because they go against the protagonist but could be the hero if the story was told in a different way.
Another definitions is similar, it is that the hero has good reasons for the bad behavior. Kind of like, Robin Hood, stealing from the rich because he wants to give to the poor. Not exact but has the idea within it.
Another look: Anti-Villain Trope
It is an attempt to humanize the villain.
This is a trait that most writer's go for these days at least to a small degree. The villain that is just evil and does it for that reason, doesn't have as much interest to the public. Most writers now have a humanistic element to their villain but I'd say the anti-villain is extended further than that.
These are the main ones. There is some discussion on anti-mystery v. anti-fantasy but it's not of our interest at this point.