Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
kipukun
10 months ago
|
parent
|
context
|
favorite
| on:
The best way to use text embeddings portably is wi...
To the second footnote: you could utilize Polar's lazyframe API to do that cosine similarity in a streaming fashion for large files.
minimaxir
10 months ago
[–]
That would get around memory limitations but I still think that would be slow.
kipukun
10 months ago
|
parent
[–]
You'd be surprised. As long as your query is using Polars natives and not a UDF (which drops it down to Python), you may get good results.
jononor
10 months ago
|
root
|
parent
[–]
A (simple) benchmark would be great to figure out where the practical limits of such an approach are. Runtime is expected to grow with O(n*2) which will get painful at some point.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
