In a deduplication storage system, the collision-resistant fingerprint of each data segment must be calculated using a hash algorithm. The g-Dedu project proposes a GPU based accelerator for processing the hash computation of the deduplication system. The g-Dedu accelerator algorithm is especially designed for handling the variable and small size of the data used in a deduplication system, which cannot be processed efficiently by a GPU in a straightforward way. Our data organization approach uses a hierarchical data structure to organize the processing data. A scheduler manages these data for optimal GPU processing. Our patterned data segment approach overcomes some noticeable performance drops resulting from the GPU memory model. Furthermore, different from some previous GPU hash accelerator work, our approach strictly follows the hash processing standard.

Currently, the initial g-Dedu achieves noticeable speedup on the SHA-1/SHA-256 computation when compared with a CPU-based implementation. We are working on integrate the g-Dedu with a practical deduplication system. We are also focus on mapping other processes of the deduplication system, e.g., hash searching and data compression, onto the g-Dedu platform.