标题：Pollux: Towards scalable distributed real-time search on microblogs
作者：Lin, Liwei ;Yu, Xiaohui ;Koudas, Nick
作者机构：[Lin, Liwei ;Yu, Xiaohui ] School of Computer Science and Technology, Shandong University, Jinan, China;[Koudas, Nick ] Department of Computer Science 更多
会议名称：16th International Conference on Extending Database Technology, EDBT 2013
会议日期：18 March 2013 through 22 March 2013
来源：ACM International Conference Proceeding Series
关键词：data stream; distributed processing; fault tolerance; microblog; search
摘要：The last few years have witnessed a meteoric rise of microblogging platforms, such as Twitter and Tumblr. The sheer volume of the microblog data and its highly dynamic nature present unique technical challenges for the platforms that provide search services. In particular, the search service must provide real-time response to queries, and continuously update the results as new microblogs are posted. Conventional approaches either cannot keep up with the high update rate, or cannot scale well to handle the large volume of data. We propose Pollux, a system that provides distributed real-time indexing and search service on microblogs. It adopts the distributed stream processing paradigm advocated by the recently developed platforms that are designed for real-time processing of large volume of data, such as Apache S4 and Twitter Storm. Although those open-source platforms have found successful applications in production environments, they lack some critical features required for real-time search. In particular: (1) they only implement partial fault tolerance, and do not provide lossless recovery in the event of a node failure, and (2) they do not have a facility for storing global data, which is necessary in efficiently ranking search results. Addressing those problems, Pollux extends current platforms in two important ways. First, we propose a failover strategy that can ensure high system availability and no data/state loss in the event of a node failure. Second, Pollux adds a global storage facility that supports convenient, efficient, and reliable data storage for shared data. We describe how to apply Pollux to the task of real-time search. We implement Pollux based on Apache S4, and show through extensive experiments on a Twitter dataset that the proposed solutions are effective, and Pollux can achieve excellent scalability. © 2013 ACM.