Go into SMA -> Advanced -> Etier Runtime Environment
For Java class name (e.g.: com.microsoft.sqlserver.jdbc.SQLServerDriver) click on the Lookup Java Class button and confirm that the Java class is being found in the lib directory where you have put the jar file.
If the class is not found there please confirm that you restarted the app server after adding the jar file to the lib directory.
If the class is found in the lib directory like it should be please go back to the Database rule and try the test connection again and this time it should work.