---
title: "Getting started with grumpy"
vignette: >
  %\VignetteIndexEntry{Getting started with grumpy}
  %\VignetteEngine{quarto::html}
  %\VignetteEncoding{UTF-8}
---

```{r setup}
library(grumpy)
```


## Motivation

This package allows users to read a wide variety of `.npy` and `.npz` files in R. These file formats are commonly used in Python for storing NumPy arrays and compressed archives of arrays, respectively. By providing a convenient interface for reading these files, `grumpy` enables R users to easily access and work with data that has been saved in these formats, facilitating interoperability between Python and R.

We envision users may want to perform some steps of their data analysis in Python and others in R.

It is thus important to be able to read and write files in both languages. Note however that `grumpy` does not support writing as we want to explicitly encourage users to use dedicated formats designed for interoperability.

In particular, the following formats are designed for large datasets, high-performance, and partial or lazy reading, including on cloud storage:

- if you are working with array-like data (the most likely use case for `.npy` files), we recommend using [Zarr](https://zarr-specs.readthedocs.io/en/latest/v3/core/) instead. Zarr datasets are supported by the `{Rarr}` Bioconductor package.
- if you are working with tabular data, we recommend using [Apache Arrow](https://arrow.apache.org/) instead. Arrow datasets are supported by the `{arrow}` CRAN package.

## Using grumpy

Use `read_npy()` and `read_npz()` to read `.npy` and `.npz` files, respectively. These functions will return R objects that are equivalent to the original NumPy arrays, allowing users to easily manipulate and analyze the data in R.

```{r}
read_npy(system.file("extdata", "test_2d.npy", package = "grumpy"))
```


If supported on your platform (POSIX-compatible platforms; i.e., Linux and macOS), `grumpy` can also memory-map the data payload of `.npy` files, resulting in reduced memory usage for large arrays. This is done by setting the `lazy` argument to `TRUE` when calling `read_npy()`.

`read_npz()` uses a similar interface to read `.npz` files, which are compressed archives of multiple `.npy` files. By default, `read_npz()` will read all arrays in the archive, but users can also specify a subset of arrays to read by providing their names or indices.

```{r}
read_npz(system.file("extdata", "test.npz", package = "grumpy"))

# we can read only some of the arrays in the archive
read_npz(
  system.file("extdata", "test.npz", package = "grumpy"),
  arrays = "x"
)
# or equivalently
read_npz(
  system.file("extdata", "test.npz", package = "grumpy"),
  arrays = 1L
)
```

### Structured datatypes

A more complex data structure is provided by structured datatypes, where each element of the array is a record with named fields.

To keep the output consistent and as conceptually close as possible to the original NumPy array, `grumpy` returns a list of list, with a `dim()` attribute to preserve the original shape of the array.

It behaves like a standard R array, but each element is a list of the fields of the original structured datatype.

```{r}
struct <- read_npy(
  system.file("extdata", "test_structured.npy", package = "grumpy")
)
struct
dim(struct)
struct[[1L]]
```

Note that in many cases, this is not efficient for any downstream analysis, and users may want to convert the output to a more standard R data structure such as a `data.frame` or `data.table` for easier manipulation.

```{r}
unlist(struct) |>
  matrix(ncol = length(struct[[1L]]), byrow = TRUE) |>
  as.data.frame()
```

Doing so loses the original shape of the array, but it is unclear if this is a problem in practice, as structured datatypes are often used to store what should be tabular data.