Councillors worried the holiday lets would contribute towards "erosion on the character” of the Old Town conservation area.