Data tables you can do real work with

SCUA has had a frame since 0.12.0: named columns, exact decimal money, and a query pipeline you write as method chains. What it could do was total a column, average it, and sum one column grouped by another. Enough for a report you already knew the shape of, and not enough for the part before that, where you are still working out what the data says.

0.27.0 fills that in. The calls you would otherwise open a Python notebook for are in the language, with nothing to install first.

#Improvements to working with frames

fn main()
  let orders = csv(fs.read_text("orders.csv")?)
  print(orders.describe())
end

column  count  mean               std                 min   p25     median   p75     max
amount  124    146.465            90.34930824306294   6.70  52.765  151.355  217.91  307.88
units   124    4.975806451612903  2.5164942373840855  1     3.0     4.5      7.0     9

One row per numeric column: how many values are there, the mean, the spread, the minimum, the quartiles and the maximum. Those are the figures as computed, not rounded for the page, so a mean of a hundred and twenty-four prices reads as long as it is. Text columns are left out rather than padded with blanks, because a column of region names has no mean. count counts the cells that hold a value, so comparing it against orders.rows() tells you how much of that column is missing.

The question a text column does answer is which values it holds:

print(orders.value_counts("region"))

region  count
north   61
south   38
east    25

Most frequent first. Between those two calls you know the shape of a file you have never opened.

#Grouping frame operations

Grouping used to mean one aggregate per call, so "revenue and order count and the typical order, by region" meant three passes and two joins. Now you name them together:

let by_region = orders.group_by("region", {
  revenue = "sum:amount",
  orders  = "count",
  typical = "median:amount",
  biggest = "max:amount",
  reps    = "count_distinct:rep",
})

region  revenue  orders  typical  biggest  reps
north   8633.75  61      149.94   307.88   4
south   6140.13  38      168.315  305.87   3
east    3387.78  25      127.63   286.29   2

Each entry names a reducer and the column it reads, or a bare count for the rows in the group. Thirteen reducers are available: sum, mean, median, min, max, std, variance, count, count_distinct, first, last, product and mode. Every one also works on a whole column, as orders.median("amount") or orders.count_distinct("rep"), and they are the same functions in both places, so a grouped median and a plain one cannot disagree.

Four reshaping verbs arrived with them. distinct() drops duplicate rows and keeps the first of each where it was, drop(names) is the complement of pick, rename leaves column order alone, and tail(n) is the far end of a sort from head. sort_by takes a list of columns and a direction for each, so "by region, then by amount descending" is one call.

#Decimal columns stay exact

A decimal column is exact by default, and 0.27.0 carries that exactness through the statistics. A total of prices is a price. So is a maximum, a minimum, a mode and a median. When the row count is even the median is the true midpoint of the two middle values, at one more decimal place only if it needs one, so the midpoint of 20.00 and 35.00 is 27.50.

Python can do exact decimal work as well, once you assemble it: import the decimal module, keep the values in a column of Python objects rather than a numeric dtype, and accept that the fast paths you picked the dataframe library for do not apply to that column. In SCUA the exact column is in the language and there is nothing to assemble.

The reducers that hand back a cell keep that cell's type, which is why the mode of a text column is text and the maximum of a money column is money. Standard deviation and variance are floats, because a deviation of money is not a money amount. A column that mixes currencies is refused rather than summed, and the error names the row.

#Pain free data processing

In Python the same job usually starts with a virtual environment and a package install, and whoever you send the script to repeats both before it runs. Here there is no install step, no virtual environment, no import and no capability grant. frame, csv, describe and the rest are in the language, and a script that reads a CSV and answers a question about it is one file you can send someone.

The data tables guide has the full surface, and examples/frames.scua in the distribution runs every verb in this post.