Missing data imputation is crucial in finance to ensure accurate financial analysis, risk management, investment strategies, and other financial applications. Recently, tensor factorization and completion have gained momentum in many finance data imputation applications, primarily due to recent breakthroughs in applying deep neural networks for nonlinear tensor analysis. However, one limitation of these approaches is that they are prone to overfitting sparse tensors that contain only a small number of observations. This paper focuses on learning highly reliable embedding for the tensor imputation problem and applies orthogonal regularizations for tensor factorization, reconstruction, and completion. The proposed neural network architecture for sparse tensors, called "RegTensor", includes multiple components: an embedding learning module for each tensor order, MLP (multilayer perception) to model nonlinear interactions among embeddings, and a regularization module to minimize overfitting problems due to the large tensor rank. Our algorithm is efficient in factorizing both single and multiple tensors (coupled tensor factorization) without incurring high training and optimization costs. We have applied this algorithm in a variety of practical scenarios, including the imputation of bond characteristics and financial analyst EPS forecast data. Experimental results demonstrate its superiority with significant performance improvements: 40%-74% better than linear tensor completion models and 2%-52% better than the state-of-the-art nonlinear models.