You are not logged in.

#1 2016-03-04 23:02:33

ericjanota
Member
Registered: 2016-03-04
Posts: 2

55,000 files to process. Batch only run first 45

I have 55,000 that I need to process but InfixPro only ran through the first 45. It worked great but what about the remaining 54,955 files? When I told it to "Add Folder..." it thought about the task for awhile but it didn't list out 55,000 file names. I thought it was just trying to save memory space.

I checked the source folder and InfixPro was able to actually process those 45 correctly. But this can't possibly be the limit, 45?

When I add the folder I want to process it reports "Unprocessed 45". So it didn't actually count all 55,000 PDFs, it seems to be stopping at 45.

I am running on the Mac in OS10.10 with InfixPro 6.48 for Mac.

Please help. I convinced my boss that we needed this tool and he wasn't really convinced. So now I can't go back to him and tell him that it only works on 45 at a time so I have to run it 1000 times.

Offline

#2 2016-03-07 12:42:34

guybushnell
Moderator
From: Norwich, England
Registered: 2013-01-28
Posts: 88
Website

Re: 55,000 files to process. Batch only run first 45

Hello.
There is no in-built limit although we've never tried it on anything that large.
I suspect that there's something about file 46 that is causing Infix to stop. Perhaps and unusual name or content.
Are you able to determine which file is file number 46? If so, please try removing it from the list or sending it to us so we can take a look (send it to sales@iceni.com).
If we can't get very far with that approach, we'll send you a special build of Infix that will provide us with more debug information.

Offline

#3 2016-03-07 21:48:27

ericjanota
Member
Registered: 2016-03-04
Posts: 2

Re: 55,000 files to process. Batch only run first 45

I tried removing the 46th file, it then stopped with the next file in the list as the last one. I finally gave up on using the folders (they all stop between 42 & 46). I went into the folder and selected 16,000 files. These ran over the weekend and got about 2,000 done. They were taking 20 minutes to an hour each to process.

So today I broke the files into smaller batches. If I run 100 of them they run in about 15 seconds each. When I tried 500 at a time, they take about 30 seconds each. I assume there is some memory issue that is causing the slowdown as the app tries to track the long list?

I am now planning to run the app 550 times, each folder will have 100 files in it. When I run the app I have to navigate to the folder, open it up and "select all" to get more than just the first 40 or so files to run.

I am using the "Find & Replace in Files". The problem is that I have to run this twice to change two different things in the files. This means I have to navigate and run this app 1,100 times to get through all of these files.

Does it work better on the PC? I haven't been able to find another way to make this change so if I have to do it this way I will but wow, what a pain.

Each file is about 4megs, 5 page pdfs. There are probably 10 text items per page while the rest of the file is imagery.

Thanks,
Eric

Offline

#4 2016-03-08 11:35:39

guybushnell
Moderator
From: Norwich, England
Registered: 2013-01-28
Posts: 88
Website

Re: 55,000 files to process. Batch only run first 45

For volumes of documents as high as yours, we'd normally recommend users to try Infix Server rather than Infix. It can spool through the jobs and complete the task more quickly that Infix. It has no GUI, is command-line driven and is intended for exactly this kind of large volume processing.
Although costly to buy, there is a pay-as-you-go option which works out well for one-off tasks like yours. Please let me know if you'd like to try it out. We'd be happy to prepare a command-file for you if you let me know what replacements you are trying to do in your PDFs.

Offline

Board footer

Powered by FluxBB