Abstract: The current research methods of the end-to-end automatic driving system mainly take an image or an image sequence as input, and directly predict the steering wheel angle with convolutional neural network. It has achieved good results, but it is not sufficient for the self-driving vehicle control by taking the advantage of the steering wheel alone. In order to better realize the horizontal and vertical control of the self-driving vehicle, the CNN-LSTM multi-modal multi-task neural network model based on end-to-end learning was constructed, the image, speed sequence and steering wheel angle sequence were taken as inputs, and the steering angle and speed of the vehicle were predicted. The model was trained and tested on the GTAV simulation platform dataset and the real-world scene dataset. The experimental results show that the model can better complete the driving behavior of lane keeping and basically realize the obstacle avoidance test of automatic driving.