command line - How to use Ruby's readlines.grep for UTF-16 files?

Question

Given the following two files created by the following commands:

$ printf "foo
bar
baz
" | iconv -t UTF-8 > utf-8.txt
$ printf "foo
bar
baz
" | iconv -t UTF-16 > utf-16.txt
$ file utf-8.txt utf-16.txt
utf-8.txt:  ASCII text
utf-16.txt: Little-endian UTF-16 Unicode text

I'd like to find the matching pattern in UTF-16 formatted file, the same way as in UTF-8 using Ruby.

Here is the working example for UTF-8 file:

$ ruby -e 'puts File.open("utf-8.txt").readlines.grep(/foo/)'
foo

However, it doesn't work for UTF-16LE formatted file:

$ ruby -e 'puts File.open("utf-16.txt").readlines.grep(/foo/)'
Traceback (most recent call last):
    3: from -e:1:in `<main>'
    2: from -e:1:in `grep'
    1: from -e:1:in `each'
-e:1:in `===': invalid byte sequence in US-ASCII (ArgumentError)

---

I've tried to convert the file based on this post by:

$ ruby -e 'puts File.open("utf-16.txt", "r").read.force_encoding("ISO-8859-1").encode("utf-8", replace: nil)' 
?tfoo
bar
baz

but it prints some invalid characters (?t) before foo, secondly I don't know how to use grep method after conversion (it reports as undefined method).

How I can use readlines.grep() method for UTF-16 file? Or some other simple way, where my goal is to print the lines with the specific regex pattern.

---

Ideally in one line, so the command can be used for CI tests.

Here is some real world scenario:

ruby -e 'if File.readlines("utf-16.log").grep(/[1-9] error/) {exit 1}; end'

but the command doesn't work due to UTF-16 formatting of the log file.

Answer 1

While the answer by Viktor is technically correct, recoding of the whole file from UTF-16LE into UTF-8 is unnecessary and might hit the performance. All you actually need is to build the regexp in the same encoding:

puts File.open(
  "utf-16.txt", mode: "rb:BOM|UTF-16LE"
).readlines.grep(
  Regexp.new "foo".encode(Encoding::UTF_16LE)
)
#??foo