The Small-Files Problem in Modern Data Lakes: From HDFS NameNode Constraints to Metadata and Manifest Bottlenecks in Iceberg and Delta Lake

The researchers analyze the staying power of the small-files problem in the three popular open-source data lakes: HDFS, Apache Iceberg, and Delta Lake. It benchmarks the effectiveness of compaction in terms of the number of fragments, the number of metadata overhead, the delay of query planning, execution performance and other factors. The results establish that HDFS can result in the most significant degradation over time. Iceberg can have lower overhead over time, and Delta Lake can have lower overhead compared to growth in the transaction log. The effectiveness of compaction is over 98.8% on platforms. The findings indicate that the new table formats don't overcome the challenge of small files, and strategies based on metadata and file optimisation can need to be constantly implemented.