TinyLoRA – Learning to Reason in 13 Parameters

(arxiv.org)

64 points | by sorenjan 4 days ago

3 comments

  • a-t-c-g 15 minutes ago
    The quality of custom models trained with proper reasoning datasets[0] even with small parameters (3-7B is sweet spot) is incredible now

    [0]: cartesien.io or Salesforce's WebscaleRL

  • measurablefunc 1 hour ago
    With four parameters I can fit an elephant, and with five I can make him wiggle his trunk so there is still room for improvement.
    • esafak 55 minutes ago
      Except learning to reason is a far cry from curve fitting. Our brains have more than five parameters.
      • voxelghost 4 minutes ago
        After a quick content browse, my understanding is this is more like with a very compressed diff vector, applied to a multi billion parameter model, the models could be 'retrained' to reason (score) better on a specific topic , e.g. math was used in the paper
      • ekuck 6 minutes ago
        speak for yourself!
      • est 21 minutes ago
        reasoning capability might just be some specific combinations of mirror neurons.

        even some advanced math usually evolves applying patterns found elsewhere into new topics

  • ValveFan6969 19 minutes ago
    [dead]