For computer models of complex physical and engineering systems, surrogates are often necessary to accelerate analyses that require large numbers of model evaluations. When a computer model is stochastic, its surrogate should also be stochastic. However, most models for stochastic surrogates are conditionally Gaussian, which can be inappropriate especially when summaries of functional outputs are used. Here we propose a non-parametric model for stochastic surrogates of stochastic computer models, where the conditional probability distributions can be asymmetric and multi-modal. Our method exploits the geometric structure of modal manifolds, which are learned from non-parametric modal regression, and estimates the conditional distributions around modal manifolds non-parametrically. We apply our method to an ocean pollutant transport model, and use the surrogate for uncertainty quantification problems relevant to oil-spill emergency planning.