arrow
Volume 9, Issue 2
Adaptive Tradeoff in Metadata-Based Small File Optimizations for a Cluster File System

X. Li, B. Dong, L. Xiao & L. Ruan

Int. J. Numer. Anal. Mod., 9 (2012), pp. 289-303.

Published online: 2012-09

Export citation
  • Abstract

Metadata-based optimizations are the common methods to improve small files performance in local file systems. However, several problems will be introduced when applying the similar optimizations for small files in cluster file systems. In this paper, we study the tradeoffs between the performance of metadata and small files in metadata-based optimizations for a cluster file system. Our method aims to guarantee the metadata performance by adaptively migrating small files among file system nodes. We establish a theory model to analyze the small files load need to be migrated. To compute the migrated load in advance, a novel forecasting method is devised to accurately predict the one-step-ahead load of metadata and small files on a MDS. Then we propose a adaptive small file threshold model to decide the small files to be migrated. In the model, we consider the long-term and short-term tradeoffs respectively. To reduce the migration overhead, we discuss the migration tradeoffs for small files and present methods and schemes to eliminate unnecessary overheads. Finally, experiments are performed on a cluster file system and the results show the efficiency of our method in terms of promoting the load forecasting accuracy, trading off the performance of metadata and small files, and reducing migration overhead.

  • AMS Subject Headings

62M10

  • Copyright

COPYRIGHT: © Global Science Press

  • Email address
  • BibTex
  • RIS
  • TXT
@Article{IJNAM-9-289, author = {}, title = {Adaptive Tradeoff in Metadata-Based Small File Optimizations for a Cluster File System}, journal = {International Journal of Numerical Analysis and Modeling}, year = {2012}, volume = {9}, number = {2}, pages = {289--303}, abstract = {

Metadata-based optimizations are the common methods to improve small files performance in local file systems. However, several problems will be introduced when applying the similar optimizations for small files in cluster file systems. In this paper, we study the tradeoffs between the performance of metadata and small files in metadata-based optimizations for a cluster file system. Our method aims to guarantee the metadata performance by adaptively migrating small files among file system nodes. We establish a theory model to analyze the small files load need to be migrated. To compute the migrated load in advance, a novel forecasting method is devised to accurately predict the one-step-ahead load of metadata and small files on a MDS. Then we propose a adaptive small file threshold model to decide the small files to be migrated. In the model, we consider the long-term and short-term tradeoffs respectively. To reduce the migration overhead, we discuss the migration tradeoffs for small files and present methods and schemes to eliminate unnecessary overheads. Finally, experiments are performed on a cluster file system and the results show the efficiency of our method in terms of promoting the load forecasting accuracy, trading off the performance of metadata and small files, and reducing migration overhead.

}, issn = {2617-8710}, doi = {https://doi.org/}, url = {http://global-sci.org/intro/article_detail/ijnam/628.html} }
TY - JOUR T1 - Adaptive Tradeoff in Metadata-Based Small File Optimizations for a Cluster File System JO - International Journal of Numerical Analysis and Modeling VL - 2 SP - 289 EP - 303 PY - 2012 DA - 2012/09 SN - 9 DO - http://doi.org/ UR - https://global-sci.org/intro/article_detail/ijnam/628.html KW - Metadata-based small files optimization, adaptive tradeoff, load forecasting, cluster file systems. AB -

Metadata-based optimizations are the common methods to improve small files performance in local file systems. However, several problems will be introduced when applying the similar optimizations for small files in cluster file systems. In this paper, we study the tradeoffs between the performance of metadata and small files in metadata-based optimizations for a cluster file system. Our method aims to guarantee the metadata performance by adaptively migrating small files among file system nodes. We establish a theory model to analyze the small files load need to be migrated. To compute the migrated load in advance, a novel forecasting method is devised to accurately predict the one-step-ahead load of metadata and small files on a MDS. Then we propose a adaptive small file threshold model to decide the small files to be migrated. In the model, we consider the long-term and short-term tradeoffs respectively. To reduce the migration overhead, we discuss the migration tradeoffs for small files and present methods and schemes to eliminate unnecessary overheads. Finally, experiments are performed on a cluster file system and the results show the efficiency of our method in terms of promoting the load forecasting accuracy, trading off the performance of metadata and small files, and reducing migration overhead.

X. Li, B. Dong, L. Xiao & L. Ruan. (1970). Adaptive Tradeoff in Metadata-Based Small File Optimizations for a Cluster File System. International Journal of Numerical Analysis and Modeling. 9 (2). 289-303. doi:
Copy to clipboard
The citation has been copied to your clipboard