- Java 91.9%
- C++ 7.3%
- CMake 0.5%
- C 0.2%
7de66791d2getMergeThreadCountAndExecutorService() created a new FixedThreadPool on every invocation but never shut it down. Since this is called per field per segment during flush/merge, thread pools accumulated unboundedly (77k+ leaked threads observed, causing OOM kills). Replace Executors.newFixedThreadPool() with a pool that has allowCoreThreadTimeOut(true) and a 1ms keepAlive. Lucene's HnswConcurrentMergeBuilder uses invokeAll which blocks until all tasks complete, so threads are guaranteed busy for the entire merge. The 1ms keepAlive reclaims them almost immediately after each merge finishes. Includes unit tests verifying thread culling, survival during active work, GC eligibility of executor objects, and an integration test exercising real Lucene HNSW merges with a short timeout. Signed-off-by: Ian Dees <iand@ziprecruiter.com> |
||
|---|---|---|
| .github | ||
| .idea | ||
| benchmarks | ||
| build-tools | ||
| buildSrc | ||
| gradle | ||
| jni | ||
| qa | ||
| release-notes | ||
| remote-index-build-client | ||
| sandbox | ||
| scripts | ||
| src | ||
| .codecov.yml | ||
| .gitignore | ||
| .gitmodules | ||
| .licenserc.json | ||
| .whitesource | ||
| ADMINS.md | ||
| build.gradle | ||
| CHANGELOG.md | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| DEVELOPER_GUIDE.md | ||
| gradle.properties | ||
| GRADLE_GRAPH.md | ||
| gradlew | ||
| gradlew.bat | ||
| LICENSE.txt | ||
| lombok.config | ||
| MAINTAINERS.md | ||
| NOTICE.txt | ||
| README.md | ||
| RELEASING.md | ||
| RFC.md | ||
| SECURITY.md | ||
| settings.gradle | ||
| TRIAGING.md | ||
OpenSearch k-NN
Welcome!
OpenSearch k-NN enables you to run the nearest neighbor search on billions of documents across thousands of dimensions with the same ease as running any regular OpenSearch query. You can use aggregations and filter clauses to further refine your similarity search operations. k-NN similarity search powers use cases such as product recommendations, fraud detection, image and video search, related document search, and more.
Project Resources
- Project Website
- Downloads
- Documentation
- Need help? Try the Forum
- Project Principles
- Contributing to OpenSearch k-NN
- Maintainer Responsibilities
- Release Management
- Admin Responsibilities
- Security
Credits and Acknowledgments
This project uses two similarity search libraries to perform Approximate Nearest Neighbor Search: the Apache 2.0-licensed Non-Metric Space Library and the MIT licensed Faiss library. Thank you to all who have contributed to those projects including Bilegsaikhan Naidan, Leonid Boytsov, Yury Malkov and David Novak for nmslib and Hervé Jégou, Matthijs Douze, Jeff Johnson and Lucas Hosseini for Faiss.
Code of Conduct
This project has adopted the Amazon Open Source Code of Conduct. For more information see the Code of Conduct FAQ, or contact opensource-codeofconduct@amazon.com with any additional questions or comments.
License
This project is licensed under the Apache v2.0 License.
Copyright
Copyright OpenSearch Contributors. See NOTICE for details.