开发者

Speed of writing to a text file or via key/value database

开发者 https://www.devze.com 2023-04-12 23:14 出处:网络
I tested writing to a text file simply via shell command or php. Using a while loop for 1 million cycles, appending cycle number into the file (as $i\\n). I thought this must be fastest way to store d

I tested writing to a text file simply via shell command or php. Using a while loop for 1 million cycles, appending cycle number into the file (as $i\n). I thought this must be fastest way to store data on the disk. Where in shell or php, it took about 40s for 1 million records.

Alternatively, I used fast key/value databases such as QDBM, GDBM, etc. The overall 开发者_StackOverflow中文版time was even lesser (15-30s). What is the theoretical explanation for faster storage of data via simple database systems? I thought the rate-limiting should be hardware (e.g the speed of hard disk or CPU).

Another Question: When increasing the cycle from 100k to 1M, the overall time was increased 10 times; but when increasing from 1M to 10M, it took 100 times longer. What threshold makes the system slow this way?


The bottleneck for writing data to disk is usually the disk I/O. The fastest continuous output can be achieved when writing contiguous blocks to the disk(s).

I assume that the file was opened and closed 1 million times in your shell script. Each close issues a synchronization of the file to disk, which probably causes a wait (depending on file system and storage solution).

0

精彩评论

暂无评论...
验证码 换一张
取 消