SCUA can now read a file of any size

SCUA reads a file of any size in a fixed amount of memory. Counting the lines in a 124 MB log takes about 11 MB, and that figure stays flat as the file grows: a gigabyte costs the same as a megabyte. fs.open hands you a live file handle you pull the file through a piece at a time, so what you hold at any moment is one line or one block.

File size and memory are now separate questions. A single SCUA value is capped at 256 MiB, which is a deliberate bound on one object rather than on your data, and streaming means the size of a file no longer runs into it. A log-processing tool can work on whatever it is pointed at.

#What it looks like

import fs
import str

match fs.open("access.log")
  Ok(f) -> do
    let errors = 0
    let more = true
    while more do
      match fs.read_line(f)
        Ok(line) ->
          if line == nil then
            more = false
          else
            if str.contains(line, "ERROR") then errors = errors + 1 end
          end
        Error(why) -> do
          print(`read failed: {why}`)
          more = false
        end
      end
    end
    print(`{errors} errors`)
    fs.close(f)
  end
  Error(why) -> print(`could not open: {why}`)
end

fs.read_chunk(f, max) is the same shape for raw bytes, which is what you want for binary files or when a line is not the unit you care about.

#The two readers end differently, on purpose

fs.read_chunk is finished when it gives you an empty block, matching how reading from a network connection, an HTTP response and a child process already work in SCUA.

fs.read_line is finished when it gives you nil, and that difference is deliberate. An empty line is a real line. A blank row in a CSV, a paragraph break, the record separator in half the world's log formats. If an empty string meant "finished", a reader would stop at the first blank line in your file and never say so, and you would get a wrong answer that looked like a right one. So the line reader gets its own signal.

One consequence worth knowing before you reach for it: fs.read_line folds Windows line endings to \n, the same as fs.read_text does. That is convenient when you are reading text and wrong when you are rewriting a file, because a Windows-formatted file read line by line and written back loses every carriage return. When you need the bytes exactly as stored, use fs.read_chunk, which changes nothing. The rule across the module is that text verbs give you \n and byte verbs give you what is on disk.

#Writing works the same way

match fs.open("report.csv", { mode = "write" })
  Ok(f) -> do
    fs.write_chunk(f, "id,name,total\n")
    fs.close(f)
  end
  Error(why) -> print(`could not open: {why}`)
end

"write" starts a new file or empties an existing one. "append" continues an existing one, or starts it if it is not there yet, so a log's first run needs no special case. fs.write is still there for content you already hold in one piece; fs.write_chunk is for output you produce as you go, which is what a transform or a report generator does.

Writes are not buffered. Each one goes to disk, so a failure is reported by the call that caused it rather than surfacing later when you close the file. When one fails partway, the error tells you how many bytes got through, and the file is exactly its previous length plus that many. So you know where the file got to, rather than having to discard it and start again. The trade is that a chunk is a trip to the operating system, so build up a few kilobytes and write that rather than one short line at a time.

#Closing files, and snapshots

Close what you open. Nothing enforces it while a program runs. Files are closed when the program ends, but a loop that opens files without closing them will exhaust the operating system's file descriptors, and the failure then lands somewhere with no obvious connection to the file you forgot: a network connection that will not open, a log that will not write. If a loop opens a file, put the close in the same loop.

A program holding an open file cannot be snapshotted. A file descriptor does not survive being saved and reloaded, so SCUA refuses rather than writing a snapshot that comes back pointing at nothing. If you are writing a handler that needs to survive a snapshot, finish with the file and close it before the handler ends. The refusal now names the handle that caused it.

#Under async I/O

With --io=async, a read hands the worker back to other tasks rather than holding it, so one handler reading a large file does not stall the others. Line reading yields when it needs more data from disk rather than on every line, which keeps the cost of yielding close to the cost of not yielding: a handler streaming a file runs at about the speed it would if it never gave the worker up.

#Where to start

The how-to page reading a file too big to load opens with a table of which reader to use for what, which is the first thing most people need. examples/streaming_files.scua reads a file line by line, reads it again as bytes, copies it through a second handle, and shows what each refusal looks like.