Not the cleanest but since the filter was always the last one in my sidebar I have hidden it in CSS like this. You may have to adapt the code depending on the theme you are using.
#search_filters_wrapper aside:last-child ul li:first-child {
display: none;
}