标题：DDSN: Duplicate Detection to Reduce Both Storage and Bandwidth Consumption
作者：Zhang, Jiaran; Yu, Xiaohui; Liu, Yang; Lin, Liwei
作者机构：[Zhang, Jiaran; Yu, Xiaohui; Liu, Yang; Lin, Liwei] Shandong Univ, Sch Comp Sci & Technol, Jinan 250100, Peoples R China.
会议名称：IEEE International Conference on Big Data (Big Data)
会议日期：OCT 06-09, 2013
来源：2013 IEEE INTERNATIONAL CONFERENCE ON BIG DATA
关键词：Duplicate Detection; Network File System
摘要：As highly centralized storage facilities are gaining popularity, duplicate detection becomes a critical problem. Traditional methods focus on reducing the storage space consumption; however, for network storage system with remote clients, the network overhead cannot be ignored, especially when the system is accessed over WAN. We propose a new duplicate detection method and implement a network file system prototype called DDSN based on this new method. It can reach the same performance in terms of storage space consumption as the state-of-the-art sliding blocking method. Meanwhile, our method overcomes its drawback that the whole file needs to be transmitted over the network, and therefore saves massive bandwidth for duplicate data. Experiments confirm the effectiveness of the proposed method.