Three comments:
1) Being able to take a class pass/fail is an existing way to achieve this to some degree
2) Eliminating, or guaranteeing, a final letter grade of a B in a graduate class means many would still aim for an A in which case the grade incentive isn't gone. In addition, even if the final course grade has a guaranteed minimum, that does not stop the professor from giving interim grades and feedback that show true performance level. In your case, it didn't seem to interfere with your ability to accurately gauge your level.
3) Effectiveness may depend on level - for graduate level courses or specialty courses in a major, student interest and curiosity would still provide motivation for hard work and effort. In lower level and non-major classes, it might not work well for all students.

One of the best courses I had in ecology was an Organization for Tropical Studies (OTS) tropical ecology field course in Costa Rica. While being in this beautiful country didn't hurt, it was the structure of the course that really helped me become a better ecologist. We explored an area as researchers, first working in a new area with a team of students and a faculty leader then later with student partners and later, on our own. In each case, our goal was to 1) observe nature, 2) formulate an interesting question, and then devise and carry out an experiment on some aspect of that question. We then analyzed our data and presented our findings to the larger group. We had the opportunity to be creative, to discuss, argue, experiment, fail, and learn - it was wonderful and inspired exactly the kind of learning we say we want from our students and it involved lots of practice and feedback on what scientists actually do. The environment was collaborative and supportive so it also fostered an excellent climate for learning.