Avoid stack overflow in Datafusion #1444

Rewrite recursive procedures into iterative procedures, suggested by @houqp @alamb

This is an acceptable short term workaround, but I think it would be more efficient and more elegant if we rewrite these recursive procedures into iterative procedures.

Do you mean to use push-based model？

@xudong963 , I think what @houqp is suggesting is to rewrite code that is recursive to not be recursive.

The pattern for Datafusion probably looks like taking code like

fn visit(expr: &expr)  {
  for child in expr.children() {
    visit(child)
  }
  // do actual expr logic
}

And changing it so the state is tracked with a structure on the heap rather than a stack. I think VecDeque is a good one for rust:

fn visit(expr: &expr)  {
  let mut worklist = VecDequeue::new();
  worklist.push_back(expr);
  while !worklist.is_empty() {
    let parent = worklist.pop_front();
    for child in parent.children() {
      worklist.push_back(child)
    }
    // do actual expr logic on parent
  }
}

(aka avoid the call back to visit)

Originally posted by @alamb in #1437 (comment)