delphi - Modifying or deleting a line from a text file the low-level way?

Question

I'm working with a Text File in Delphi, and I don't wish to use the method of loading/saving with a string list. I intend to maintain an open filestream where I read and write my data there, keeping massive amounts of data on the hard disk instead of in the memory. I have the simple concept of writing new lines to a text file and reading them, but when it comes to modifying and deleting them, I cannot find any good resources.

Each line in this file contains a name, and equals sign, and the rest is data. For example, SOMEUNIQUENAME=SomeStringValue. I intend to keep a file open for a period of time inside of a thread. This thread performs incoming requests to either get, set, or delete certain fields of data. I use WriteLn and ReadLn in a loop, evaluating EOF. Below is an example of how I read the data:

FFile = TextFile;

...

function TFileWrapper.ReadData(const Name: String): String;
var
  S: String; //Temporary line to be parsed
  N: String; //Temporary name of field
begin
  Result:= '';
  Reset(FFile);
  while not EOF(FFile) do begin
    ReadLn(FFile, S);
    N:= UpperCase(Copy(S, 1, Pos('=', S)-1));
    if N = UpperCase(Name) then begin
      Delete(S, 1, Pos('=', S));
      Result:= S;
      Break;
    end;
  end;
end;

...and then I trigger an event which informs sender of result. The requests are inside of a queue, which is sort of a message pump for these requests. The thread simply processes the next request in the queue repeatedly, similar to how typical applications work.

I have procedures ready to be able to write and delete these fields, but I don't know what I have to do to actually perform the action on the file.

procedure TFileWrapper.WriteData(const Name, Value: String);
var
  S: String; //Temporary line to be parsed
  N: String; //Temporary name of field
begin
  Result:= '';
  Reset(FFile);
  while not EOF(FFile) do begin
    ReadLn(FFile, S);
    N:= UpperCase(Copy(S, 1, Pos('=', S)-1));
    if N = UpperCase(Name) then begin
      //How to re-write this line?
      Break;
    end;
  end;
end;

procedure TFileWrapper.DeleteData(const Name: String);
var
  S: String; //Temporary line to be parsed
  N: String; //Temporary name of field
begin
  Result:= '';
  Reset(FFile);
  while not EOF(FFile) do begin
    ReadLn(FFile, S);
    N:= UpperCase(Copy(S, 1, Pos('=', S)-1));
    if N = UpperCase(Name) then begin
      //How to delete this line?
      Break;
    end;
  end;
end;

In the end, I need to avoid loading the entire file into the memory to be able to accomplish this.

Answer 1

I find this an interesting question, so I made a small console app.

I used 3 methods:

• TStringList
• Streamreader/StreamWriter
• Text file

All methods are timed and repeated 100 times with a text file of 10kb in size and a text file 1Mb in size. Here is the program:

program Project16;

{$APPTYPE CONSOLE}

uses
  SysUtils, Classes, StrUtils, Diagnostics, IOUtils;

procedure DeleteLine(StrList: TStringList; SearchPattern: String);

var
  Index : Integer;

begin
 for Index := 0 to StrList.Count-1 do
  begin
   if ContainsText(StrList[Index], SearchPattern) then
    begin
     StrList.Delete(Index);
     Break;
    end;
  end;
end;

procedure DeleteLineWithStringList(Filename : string; SearchPattern : String);

var StrList : TStringList;

begin
 StrList := TStringList.Create;
 try
  StrList.LoadFromFile(Filename);
  DeleteLine(StrList, SearchPattern);
  // don't overwrite our input file so we can test
  StrList.SaveToFile(TPath.ChangeExtension(Filename, '.new'));
 finally
  StrList.Free;
 end;
end;

procedure DeleteLineWithStreamReaderAndWriter(Filename : string; SearchPattern : String);

var
  Reader    : TStreamReader;
  Writer    : TStreamWriter;
  Line      : String;
  DoSearch  : Boolean;
  DoWrite   : Boolean;

begin
 Reader := TStreamReader.Create(Filename);
 Writer := TStreamWriter.Create(TPath.ChangeExtension(Filename, '.new'));
 try
  DoSearch := True;
  DoWrite := True;
  while Reader.Peek >= 0 do
   begin
    Line := Reader.ReadLine;
    if DoSearch then
     begin
      DoSearch := not ContainsText(Line, SearchPattern);
      DoWrite := DoSearch;
     end;
    if DoWrite then
     Writer.WriteLine(Line)
    else
     DoWrite := True;
   end;
 finally
  Reader.Free;
  Writer.Free;
 end;
end;

procedure DeleteLineWithTextFile(Filename : string; SearchPattern : String);

var
 InFile    : TextFile;
 OutFile   : TextFile;
 Line      : String;
 DoSearch  : Boolean;
 DoWrite   : Boolean;


begin
 AssignFile(InFile, Filename);
 AssignFile(OutFile, TPath.ChangeExtension(Filename, '.new'));
 Reset(InFile);
 Rewrite(OutFile);
 try
  DoSearch := True;
  DoWrite := True;
  while not EOF(InFile) do
   begin
    Readln(InFile, Line);
    if DoSearch then
     begin
      DoSearch := not ContainsText(Line, SearchPattern);
      DoWrite := DoSearch;
     end;
    if DoWrite then
     Writeln(OutFile, Line)
    else
     DoWrite := True;
   end;
 finally
  CloseFile(InFile);
  CloseFile(OutFile);
 end;
end;

procedure TimeDeleteLineWithStreamReaderAndWriter(Iterations : Integer);

var
  Count : Integer;
  Sw    : TStopWatch;

begin
 Writeln(Format('Delete line with stream reader/writer - file 10kb, %d iterations', [Iterations]));
 Sw := TStopwatch.StartNew;
 for Count := 1 to Iterations do
  DeleteLineWithStreamReaderAndWriter('c:empext10kb.txt', 'thislinewillbedeleted=');
 Sw.Stop;
 Writeln(Format('Elapsed time : %d milliseconds', [Sw.ElapsedMilliseconds]));
 Writeln(Format('Delete line with stream reader/writer - file 1Mb, %d iterations', [Iterations]));
 Sw := TStopwatch.StartNew;
 for Count := 1 to Iterations do
  DeleteLineWithStreamReaderAndWriter('c:empext1Mb.txt', 'thislinewillbedeleted=');
 Sw.Stop;
 Writeln(Format('Elapsed time : %d milliseconds', [Sw.ElapsedMilliseconds]));
end;

procedure TimeDeleteLineWithStringList(Iterations : Integer);

var
  Count : Integer;
  Sw    : TStopWatch;

begin
 Writeln(Format('Delete line with TStringlist - file 10kb, %d iterations', [Iterations]));
 Sw := TStopwatch.StartNew;
 for Count := 1 to Iterations do
  DeleteLineWithStringList('c:empext10kb.txt', 'thislinewillbedeleted=');
 Sw.Stop;
 Writeln(Format('Elapsed time : %d milliseconds', [Sw.ElapsedMilliseconds]));
 Writeln(Format('Delete line with TStringlist - file 1Mb, %d iterations', [Iterations]));
 Sw := TStopwatch.StartNew;
 for Count := 1 to Iterations do
  DeleteLineWithStringList('c:empext1Mb.txt', 'thislinewillbedeleted=');
 Sw.Stop;
 Writeln(Format('Elapsed time : %d milliseconds', [Sw.ElapsedMilliseconds]));
end;

procedure TimeDeleteLineWithTextFile(Iterations : Integer);

var
  Count : Integer;
  Sw    : TStopWatch;

begin
 Writeln(Format('Delete line with text file - file 10kb, %d iterations', [Iterations]));
 Sw := TStopwatch.StartNew;
 for Count := 1 to Iterations do
  DeleteLineWithTextFile('c:empext10kb.txt', 'thislinewillbedeleted=');
 Sw.Stop;
 Writeln(Format('Elapsed time : %d milliseconds', [Sw.ElapsedMilliseconds]));
 Writeln(Format('Delete line with text file - file 1Mb, %d iterations', [Iterations]));
 Sw := TStopwatch.StartNew;
 for Count := 1 to Iterations do
  DeleteLineWithTextFile('c:empext1Mb.txt', 'thislinewillbedeleted=');
 Sw.Stop;
 Writeln(Format('Elapsed time : %d milliseconds', [Sw.ElapsedMilliseconds]));
end;

begin
  try
    TimeDeleteLineWithStringList(100);
    TimeDeleteLineWithStreamReaderAndWriter(100);
    TimeDeleteLineWithTextFile(100);
    Writeln('Press ENTER to quit');
    Readln;
  except
    on E: Exception do
      Writeln(E.ClassName, ': ', E.Message);
  end;
end.

Output:

Delete line with TStringlist - file 10kb, 100 iterations
Elapsed time : 188 milliseconds
Delete line with TStringlist - file 1Mb, 100 iterations
Elapsed time : 5137 milliseconds
Delete line with stream reader/writer - file 10kb, 100 iterations
Elapsed time : 456 milliseconds
Delete line with stream reader/writer - file 1Mb, 100 iterations
Elapsed time : 22382 milliseconds
Delete line with text file - file 10kb, 100 iterations
Elapsed time : 250 milliseconds
Delete line with text file - file 1Mb, 100 iterations
Elapsed time : 9656 milliseconds
Press ENTER to quit

As you can see is TStringList the winner here. Since you are not able to use TStringList, TextFile is not a bad choice after all...

P.S. : this code omits the part where you have to delete the inputfile and rename the outputfile to the original filename