Alex Wein
Follow

A question for the D3 pros: what tools do you use for data munging? Javascript, Spreadsheets, SQL (or an ORM), R or Python, something else? If you use JS for most of it, I'd love to know more specifics regarding libraries (or maybe you're just aces with Array.prototype methods)

@alexwein I like to work with ndjson (new line-delimited json). Bostock has some great command-line utils for transforming data in that format github.com/mbostock/ndjson-cli. For anything that’s more involved, I fall back to Python because the pydata ecosystem is so solid. 🐍

@alexwein python w/ pandas is really great for tabular data, networkx for graphs.

@alexwein I use python generally (pandas) but I've also used those ndjson tools by Mike bostock, which are JavaScript and can even be used in the terminal, but they are just a bit too cryptic for me. Pandas is quite easy to get stuff done in. I've been looking for a JavaScript data library as well though.

@alexwein I do a lot of cleaning work in Google Spreadsheets, because my datasets aren't usually enormous. When I do have something larger, I usually work through Javascript's .filter() .reduce() and .map() methods. This helped me a lot: learnjsdata.com/index.html

@alexwein almost all JS/D3 methods. slowly learning python which is great for larger sets and file export. also just came across npmjs.com/package/datalib which underpins Vega, haven't used it yet

@alexwein D3 array manipulation, Excel, OpenRefine, R, Mr Dataconverter, Mapshaper, ShapeEscape and some manual manipulation in Excel and text editor.

Sign in to participate in the conversation
vis.social

A social space for anyone in data, visualization, creative coding, and related arts and research. Share your work, discuss, critique, and ask for help! Come one, come all:

→ creative coders
→ data scientists and visualizers
→ generative artists
→ visual researchers, curators, and critics
→ anyone else data- or visualization-adjacent

If you are curious about the world and creativity, you are welcome here. Learn more.