Using a pool size of > 1 for inbound file service
Is there any issue with using a pool size setting greater than 1 for an inbound file service? The reason I ask is we can easily dump 100k+ files to a folder in minutes, but using a pool size of 1 can take hours for that backlog to clear.
Comments
If FIFO is not required, a pool size >1 allows for processes to run in parallel.
Is that for all inbound adapters, especially a recordmap based one?
The reason I ask, is while trying this out for a recordmap service I am seeing the following messages logged:
- Skipping locked file key.
- ERROR #5001: No records mapped from Submission
I can't speak to your specific configuration and scenario but for more complete Pool Size information, the documentation is at http://docs.intersystems.com/latest/csp/docbook/DocBook.UI.Page.cls?KEY=ECONFIG_PoolSize
Is it possible to append the files outside of Ensemble and then use batch record map? The current version of Ensemble handles batch record maps much more efficiently than it can handle individual record maps. If you are on an older version, let me know and I'll find out if the efficiency improvements are in it.
I think that increasing pool size will give you some improvement, but batch record map can avoid a lot of overhead handling each individual record.
Part of the efficiency improvement depends on what you're doing with the record. If you're just loading data, it's very efficient. If you're performing a transformation on the incoming data then that could still take a significant amount of time.
The big efficiency improvement is in 2016.1. Before that you'll only get a minor improvement going to record map batch. This is an incentive to upgrade to a new version.