When you begin the screen recording process, make sure that the recording dimensions are set to 1280 pixels x 720 pixels.
From the Camtasia Recorder, go to Tools -> Options, change video configuration to manual and set the screen capture rate to 30 frames per second. Under audio format, use PCM with 44.100 kHz 16 bit Stereo. Save these settings and record your screencast video.
Once you are done with the recording and editing process, it's time to produce your screencast video in HD format for YouTube. You may either use the MP4 or MOV formats for the output video as they both use the same MPEG-4 / H.264 codec.