• mvirts@lemmy.world
    link
    fedilink
    English
    arrow-up
    1
    ·
    3 months ago

    It’s more likely you’ll eat up storage when you read a 600mb parquet and try to write it as CSV.

      • mvirts@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        3 months ago

        Ah I was trying to point out that CSV is the inefficient format. Reading a large amount of data from a more efficient format like parquet is more likely to cause trouble because the memory required can be more than the file size. CSV is the opposite where it will almost always use more disk space than is required to represent the data in memory.