TransWikia.com

Table Glitch / Error with binary data

Mathematica Asked by atlasgeo on June 23, 2021

I have a large group of binary files that I am loading with

binaryList = Table[ BinaryReadList[ paths[[i]], "Byte"], {i, Length@paths}];

specifically 1019 files that are 4.5 mb each. I then use the following to parse through the first binary list in order to extract specific bytes and make a list.

headerBytes = Table[ int32bytesToInt@binaryList[[ 1, 12240*(i-1)+185;;12240*(i-1)+188]], {i, 364}]

where int32bytesToInt is my function, and each entry of binaryList is actually 364 data streams, 12000 bytes long (with a 240 byte header), all run together (I’m extracting bytes 185 through 188 from the header of each data stream). The code worked perfectly on smaller data sets (~400 files, 2 mb each) but with this amount of data, the second Table freezes once the ‘i’ iterator value gets to around 275. Simply changing from Table to Do and the second step completes like normal. Additionally, I can do this

tempList = binaryList[[1]];
headerBytes = Table[ int32bytestoint[ tempList[[ 12240*(i-1)+185;;12240*(i-1)+188]], {i, 364}]

and it also completes like normal (and in about 0.07 seconds). Why can’t I pass the full list to Table without making this tempList? Is this a problem with Table or am I missing something?

Note that it only freezes once I’ve loaded over 2 gb of binary data, so I can’t easily share this, I’m more interested if I’m using bad practice, or if someone has encountered a similar issue with Table.

Add your own answers!

Ask a Question

Get help from others!

© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP