8eaa27667c* feat: Implement PPL convert command with 5 conversion functions - Added convert command syntax and AST nodes (Convert, ConvertFunction) - Implemented 5 conversion functions: auto, num, rmcomma, rmunit, none - Full Calcite pushdown support via CalciteRelNodeVisitor - Logical and physical operators (LogicalConvert, ConvertOperator) - Comprehensive test coverage (26 tests total): * 13 unit tests (CalcitePPLConvertTest) * 8 integration tests with pushdown (ConvertCommandIT) * 8 non-pushdown tests (CalciteConvertCommandIT) * 3 explain tests (ExplainIT) * 2 cross-cluster tests (CrossClusterSearchIT) * Anonymizer and v2 unsupported tests - User documentation (docs/user/ppl/cmd/convert.md) - Code cleanup: removed dead code, simplified javadocs - Version: 3.5 (experimental) All tests compile and pass successfully. Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing integration tests Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing ymal files for IT explain tests Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fix cross-cluster IT failure Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Making code more readable and removing unnecessary logic Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Refactor: Extract BaseConversionUDF to eliminate duplication Centralized common conversion function logic into BaseConversionUDF base class, reducing code from 219 to 162 lines (26% reduction) while maintaining all functionality. Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing CI failure and refactoring Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * trigger CI Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Addressing CodeRabbit comments Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding support for memk function Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing formatting Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Updating documentation Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Refactoring code to avoid regestering none() as a convert function and separating concerns into each respective function class Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * refactor: Simplify Convert command using Let expressions Replace ConvertFunction class with standard Let expressions to handle none() at AST level and reduce code complexity. Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Trigger CI Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Refactoring code to use Template Method Design Pattern Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Updating documentation Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Trigger CI Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Updating convert example with stats Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Renaming unit test class, as ConversionUtil class was removed Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Fixing IT test case Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Resolve merge conflict: Add testTransposeCommand alongside testConvertCommand Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Add null and empty string tests for NumConvertFunction Add testNumConvertNullAndEmpty() to ensure NumConvertFunction.convert() properly returns null for null, empty string, and whitespace-only inputs, matching the coverage pattern of other conversion functions. Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Add null and empty string tests for RmunitConvertFunction Add testRmunitConvertNullAndEmpty() to ensure RmunitConvertFunction.convert() properly returns null for null, empty string, and whitespace-only inputs, completing the test coverage pattern across all conversion functions. Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Verify convert AS clause preserves original field Update testConvertWithAlias to verify that using AS clause preserves the original field alongside the converted field. Test now checks both balance (bigint) and balance_num (double) are present in results. Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Adding edge test cases, as recommended by Coderabbit Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Trigger CI Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Removing timeformat parameter for now, will add later Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Trigger CI Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * Re-trigger CI Signed-off-by: Aaron Alvarez <aaarone@amazon.com> * fix merge conflicts Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix cross cluster tests Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix merge conflict Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * fix merge conflicts Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> * use singleton for udf instance Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> --------- Signed-off-by: Aaron Alvarez <aaarone@amazon.com> Signed-off-by: Ritvi Bhatt <ribhatt@amazon.com> Signed-off-by: ritvibhatt <53196324+ritvibhatt@users.noreply.github.com> Co-authored-by: Aaron Alvarez <aaarone@amazon.com> |
||
|---|---|---|
| .github | ||
| api | ||
| async-query | ||
| async-query-core | ||
| benchmarks | ||
| build-tools | ||
| buildSrc | ||
| common | ||
| core | ||
| datasources | ||
| direct-query | ||
| direct-query-core | ||
| docs | ||
| doctest | ||
| gradle/wrapper | ||
| integ-test | ||
| language-grammar | ||
| legacy | ||
| opensearch | ||
| plugin | ||
| ppl | ||
| prometheus | ||
| protocol | ||
| release-notes | ||
| scripts | ||
| spark | ||
| sql | ||
| .codecov.yml | ||
| .gitignore | ||
| .pre-commit-config.yaml | ||
| .whitesource | ||
| ADMINS.md | ||
| build.gradle | ||
| CODE_OF_CONDUCT.md | ||
| CONTRIBUTING.md | ||
| CONTRIBUTORS.md | ||
| DEVELOPER_GUIDE.rst | ||
| gradle.properties | ||
| gradlew | ||
| gradlew.bat | ||
| LICENSE.txt | ||
| lombok.config | ||
| MAINTAINERS.md | ||
| NOTICE | ||
| README.md | ||
| RELEASING.md | ||
| SECURITY.md | ||
| settings.gradle | ||
| THIRD-PARTY | ||
OpenSearch SQL
OpenSearch enables you to extract insights out of OpenSearch using the familiar SQL or Piped Processing Language (PPL) query syntax. Use aggregations, group by, and where clauses to investigate your data. Read your data as JSON documents or CSV tables so you have the flexibility to use the format that works best for you.
The following projects are related to the SQL plugin, but stored in the different repos. Please refer to links below for details. This document will focus on the SQL plugin for OpenSearch.
Code Summary
SQL Engine
| Test and build | [![SQL CI][sql-ci-badge]][sql-ci-link] |
| Code coverage |  |
| Distribution build tests |   |
| Backward compatibility tests |  |
Repository Checks
| DCO Checker |  |
| Link Checker |  |
Issues
 |
 |
 |
 |
 |
 |
Highlights
Besides basic filtering and aggregation, OpenSearch SQL also supports complex queries, such as querying semi-structured data, JOINs, set operations, sub-queries etc. Beyond the standard functions, OpenSearch functions are provided for better analytics and visualization. Please check our documentation for more details.
Recently we have been actively improving our query engine primarily for better correctness and extensibility. Behind the scene, the new enhanced engine has already supported both SQL and Piped Processing Language. Please find more details in SQL Engine V2 - Release Notes.
Documentation
Please refer to the SQL Language Reference Manual, Piped Processing Language (PPL) Reference Manual, OpenSearch SQL/PPL Engine Development Manual and Technical Documentation for detailed information on installing and configuring plugin.
Forum
For additional help with the plugin, including questions about opening an issue, visit the OpenSearch Forum.
Contributing
See developer guide and how to contribute to this project.
Attribution
This project is based on the Apache 2.0-licensed elasticsearch-sql project. Thank you eliranmoyal, shi-yuan, ansjsun and everyone else who contributed great code to that project. Read this for more details Attributions.
Code of Conduct
This project has adopted an Open Source Code of Conduct.
Security
If you discover a potential security issue in this project we ask that you notify OpenSearch Security directly via email to security@opensearch.org. Please do not create a public GitHub issue.
License
See the LICENSE file for our project's licensing. We will ask you to confirm the licensing of your contribution.
Copyright
Copyright OpenSearch Contributors. See NOTICE for details.