This is not answering your question directly, but I feel I put it as an answer (plus it is too big to post as a comment).
My advice: please stop playing with htaccess for this kind of task (force few URLs to use HTTPS and force the rest to use HTTP).
The best way is to generate FULL URLs for all links (pages, not resources), where URL includes domain name and protocol. In this case all URLs will have proper protocol (HTTP/HTTPS) straight away. Of course: you can still fix (301 or 302 redirect) requests to supposed-to-be-https if they (for some strange reason) are requested via HTTP. That's where .htaccess can be safely and easily used.
If user will request normal page (should be served over HTTP) via HTTPS -- then let him do it -- there is nothing wrong with that. Yes -- HTTPS requires a bit more resources on server side, but if you generate all links in such way, there will be virtually no such situations, unless user specifically changes protocol. Even if such one page will be served over HTTPS, the next "normal" link he click will be HTTP -- 1 extra HTTPS-based page view will not kill your server.
I'm using this approach all the time when site is having secure area .. and based on the logs, we have less than 0.01% of ALL page views that were viewed/attempted to be viewed via "wrong" protocol -- vast majority of them were bots or attempts to hack/vulnerability search.
Based on such stats I would say -- it is working perfectly. yes -- you need to alter you code/templates a bit to implement this .. but it is much better than messing with .htaccess and mod_rewrite.
In any case, here are the rules that would do the job for you:
# force https for all URLs in /checkout
RewriteCond %{HTTPS} =off
RewriteRule ^checkout https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
# don't do anything for images/css/js
RewriteRule .(gif|jpe?g|png|css|js)$ - [NC,L]
# force http for all other URLs that are not in /checkout
RewriteCond %{HTTPS} =on
RewriteCond %{REQUEST_URI} !^/(checkout|index.php/checkout)
RewriteRule .* http://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
# your other rules here, e.g.:
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php/$1 [L]
OR
# force https for all URLs in /checkout
RewriteCond %{HTTPS} =off
RewriteRule ^checkout https://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
# do not do anything for already existing files
RewriteCond %{REQUEST_FILENAME} -f [OR]
RewriteCond %{REQUEST_FILENAME} -d
RewriteRule .+ - [L]
# force http for all other URLs that are not in /checkout
RewriteCond %{HTTPS} =on
RewriteCond %{REQUEST_URI} !^/checkout
RewriteRule .* http://%{HTTP_HOST}%{REQUEST_URI} [R=301,L]
# your other rules here, e.g.:
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*)$ /index.php/$1 [L]
与恶龙缠斗过久,自身亦成为恶龙;凝视深渊过久,深渊将回以凝视…