For efficiently searching through large amounts of data, Teezir uses proprietary indexing structures. Since Teezir’s search functionality is required to operate in dynamic environments, the data is changing continuously. To provide efficient access to up-to-date content, it is important to update the indexes regularly. At the same time we want to provide users with transparent and uninterrupted search functionality on the latest data.
Since the search functionality and the index updating work on the same data structures, this requires a careful design of the updating mechanisms. We developed our system in such a way that index and search operations do not lock each other out. This way, searching and index updating are truly concurrent. This also means that many active users can search the index simultaneously.
To make the indexing process more efficient while balancing this with the real-time requirements of our customers, we batch together ‘add’ and ‘remove’ operations, which are executed and processed after a few minutes. In the same time, smart caching is used to keep the most frequently read structures in memory, thereby increasing throughput during both indexing and searching.