What Category Theory Teaches Us About DataFrames

4 comments

• getnormality 3 minutes ago
  Hmm. Folks trying to discover the elegant core of data frame manipulation by studying... pandas usage patterns. When R's dplyr solved this over a decade ago.
• rich_sasha 1 hour ago
  The article starts well, on trying to condense pandas' gaziliion of inconsistent and continuously-deprecated functions with tens of keyword arguments into a small, condensed set of composable operations - but it lost me then.

  The more interesting nugget for me is about this project they mention: https://modin.readthedocs.io/en/latest/index.html called Modin, which apparently went to the effort of analysing common pandas uses and compressed the API into a mere handful of operations. Which sounds great!

  Sadly for me the purpose seems to have been rather to then recreate the full pandas API, only running much faster, backed by things like Ray and Dask. So it's the same API, just much faster.

  To me it's a shame. Pandas is clearly quite ergonomic for various exploratory interactive analyses, but the API is, imo, awful. The speed is usually not a concern for me - slow operations often seem to be avoidable, and my data tends to fit in (a lot of) RAM.

  I can't see that their more condensed API is public facing and usable.
• few 1 hour ago
  I felt like one or two decades ago, all the rage was about rewriting programs into just two primitives: map and reduce.

  For example filter can be expressed as:

    is_even = lambda x: x % 2 == 0
    mapped = map(lambda x: [x] if is_even(x) else [], data)
    filtered = reduce(lambda x, y: x + y, mapped, [])
  

  But then the world moved on from it because it was too rigid
  [-]
  • mememememememo 35 minutes ago
    Performance aside it seems you could do most maybe a the ops with those three. I say three because your sneaky plus is a union operation. So map, reduce and union.

    But you are also allowing arbitrary code expressions. So it is less lego-like.
• jiehong 1 hour ago
  Dups of a few days ago:

  - https://news.ycombinator.com/item?id=47567087