inquiry race condition fix; default max concurrency, db max delay

[guzzijones]

1. Fix a race condition with inquiries.
   • lock coordinator in workflow engine and action execution engine to prevent race with inquiry resuming.
2. Add a default concurrency for workflows.
   • if set to 0 then no concurrency is the default .
   • default config value is 0 to keep current behavior
   • if workflow sets a concurrency it overrides this
3. Added stop max delay
   • stop_max_delay=cfg.CONF.workflow_engine.retry_stop_max_msec
     (60 s default) added to every retry_on_transient_db_errors
     decorator in st2common.services.workflows — 10 call sites
     including request, request_pause, request_resume,
     request_cancellation, update_task_state,
     update_task_execution, resume_task_execution,
     update_workflow_execution, resume_workflow_execution,
     fail_workflow_execution. When the ceiling is hit, the last
     StackStormDBObjectWriteConflictError propagates out to
     WorkflowExecutionHandler.process, which routes it to
     fail_workflow_execution and releases the coordination lock —
     freeing every other message waiting behind that workflow.

Back story

We had an unbounded with items that filled up the work queues. The actions were continuously rescheduled due to the process that looks back 1 minute for zombie actions. Fix 3 is more of a fail safe for mongo writes. It came about when reviewing this. Fix 1 was noticed when reviewing logs that every inquiry causes a db write conflict. Fix 2 actually bounds unbounded with items as an option.