Newshot

datasette-arrows 0.1a0

Summary

Release: datasette-arrows 0.1a0 Datasette 1.0a43 added the ability for plugins to implement streaming export formats. Datasette core supports streaming CSV and TSV - you can export millions of lines of data from a table without needing to load it all into memory, because those two file formats support streaming data in chunks. The register_output_renderer() plugin hook can now be used to implement the same pattern for other formats. I've learned never to release a new plugin feature without accompanying it with a plugin that uses it. Claude Fable 5.2 suggested the Apache Arrow streaming format as a good option, and built this plugin in almost a single shot. Add the plugin to Datasette ( datasette install datasette-arrows ) and the export dialog gains this new Arrow option: This screenshot is from Datasette's main branch - the most recent release doesn't yet group streaming formats such as Arrow with the TSV and CSV in the "All rows in this table" section, but that's coming in the next…

Original text(via Simon Willison)

Datasette 1.0a43 added the ability for plugins to implement streaming export formats. Datasette core supports streaming CSV and TSV - you can export millions of lines of data from a table without needing to load it all into memory, because those two file formats support streaming data in chunks. The register_output_renderer() plugin hook can now be used to implement the same pattern for other formats.

I've learned never to release a new plugin feature without accompanying it with a plugin that uses it.

Claude Fable 5.2 suggested the Apache Arrow streaming format as a good option, and built this plugin in almost a single shot.

Add the plugin to Datasette (datasette install datasette-arrows) and the export dialog gains this new Arrow option:

Export data modal for the threads table. The TSV and CSV exports are now joined by Arrow, showing "All 927 rows" and the text Apache Arrow IPC stream, for DuckDB, pandas, Polars and other tools. There is a Download button and a Copy URL button.
Export data modal for the threads table. The TSV and CSV exports are now joined by Arrow, showing "All 927 rows" and the text Apache Arrow IPC stream, for DuckDB, pandas, Polars and other tools. There is a Download button and a Copy URL button.

This screenshot is from Datasette's main branch - the most recent release doesn't yet group streaming formats such as Arrow with the TSV and CSV in the "All rows in this table" section, but that's coming in the next alpha release.

The text above was extracted automatically from the source page and may miss images, tables or code blocks. The original is authoritative.

Published by Simon Willison
This site only summarises and aggregates; copyright stays with the original author. Headlines and summaries follow the source — where they differ, the original wins.
Read the original ↗