A word of warning to those who use AWS Elastic Load Balancing. The default configuration is that any request that hasn't had any data sent from the web server for 60 records, results in the balancer resending the request. The reason for this, presumably, is so that if a server has dies mid-way through a request, the user still would get their request processed by, potentially, another server.
However, at intelligentgolf, we do some processing that can result in requests that quite naturally last longer than 60 seconds: sending mass email, or interfacing with external systems, which require 100s or 1000s of external connections to interface with 3rd party legacy systems. In some of these cases, we found that duplications were happening, even though we were sure that the form wasn't being submitted twice.
So, what we do in a lot of cases now:
- Do the batch processing in the background, using a cron job, or other similar method.