Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[feature-wip](multicatalog)support apche druid catalog #27270

Closed
wants to merge 27 commits into from

Conversation

vinlee19
Copy link
Contributor

Proposed changes

we can use Apache Druid catalog as follows:

    "type"="jdbc",
    "jdbc_url" = "jdbc:avatica:remote:url=http://ip:coordinator_port/druid/v2/sql/avatica/",
    "user" = "admin",
    "password" = "password1",
    "driver_url" = "avatica-1.23.0.jar",
    "driver_class" = "org.apache.calcite.avatica.remote.Driver"
);
    "type"="jdbc",
    "jdbc_url" = "jdbc:avatica:remote:url=http://ip:broker_port/druid/v2/sql/avatica/",
    "user" = "admin",
    "password" = "password1",
    "driver_url" = "avatica-1.23.0.jar",
    "driver_class" = "org.apache.calcite.avatica.remote.Driver"
);
APACHE DRUID TYPE Doris Type  
BIGINT BIGINT  
FLOAT DOUBLE  
DOUBLE DOUBLE 扩大一个数
TIMESTAMP DATETIME  
VARCHAR STRING  
COMPLEX JSON  

catalog:

CREATE CATALOG `jdbc_druid_auth` PROPERTIES (
"type" = "jdbc",
"jdbc_url" = "jdbc:avatica:remote:url=http://localhost:8888/druid/v2/sql/avatica/",
"driver_url" = "avatica-1.23.0.jar",
"user"="admin",
"password" = "password1",
"driver_class" = "org.apache.calcite.avatica.remote.Driver"
);

doris table:

CREATE TABLE `kttm1_doris` (
  `__time` datetimev2(6) NOT NULL,
  `agent_category` varchar(*) NULL,
  `agent_type` varchar(*) NULL,
  `browser` varchar(*) NULL,
  `browser_version` varchar(*) NULL,
  `city` varchar(*) NULL,
  `continent` varchar(*) NULL,
  `country` varchar(*) NULL,
  `version` varchar(*) NULL,
  `event_type` varchar(*) NULL,
  `event_subtype` varchar(*) NULL,
  `loaded_image` varchar(*) NULL,
  `adblock_list` varchar(*) NULL,
  `forwarded_for` varchar(*) NULL,
  `language` varchar(*) NULL,
  `number` varchar(*) NULL,
  `os` varchar(*) NULL,
  `path` varchar(*) NULL,
  `platform` varchar(*) NULL,
  `referrer` varchar(*) NULL,
  `referrer_host` varchar(*) NULL,
  `region` varchar(*) NULL,
  `remote_address` varchar(*) NULL,
  `screen` varchar(*) NULL,
  `session` varchar(*) NULL,
  `session_length` bigint(20) NOT NULL,
  `timezone` varchar(*) NULL,
  `timezone_offset` varchar(*) NULL,
  `window` varchar(*) NULL
)DUPLICATE KEY(`__time`)
DISTRIBUTED BY HASH(`agent_category`) BUCKETS 10
PROPERTIES (
"replication_allocation" = "tag.location.default: 1"
);

case 1:

mysql> select count(*) from kttm1_doris;
+----------+
| count(*) |
+----------+
|   465346 |
+----------+
1 row in set (0.12 sec)

mysql> select count(*) from jdbc_druid_auth.druid.kttm1;
+----------+
| count(*) |
+----------+
|   465346 |
+----------+
1 row in set (0.72 sec)

case 2:

mysql> CREATE TABLE `wikipedia` (
    ->   `__time` datetimev2(6) NOT NULL,
    ->   `isRobot` varchar(*) NULL,
    ->   `channel` varchar(*) NULL,
    ->   `flags` varchar(*) NULL,
    ->   `isUnpatrolled` varchar(*) NULL,
    ->   `page` varchar(*) NULL,
    ->   `diffUrl` varchar(*) NULL,
    ->   `added` bigint(20) NOT NULL,
    ->   `comment` varchar(*) NULL,
    ->   `commentLength` bigint(20) NOT NULL,
    ->   `isNew` varchar(*) NULL,
    ->   `isMinor` varchar(*) NULL,
    ->   `delta` bigint(20) NOT NULL,
    ->   `isAnonymous` varchar(*) NULL,
    ->   `user` varchar(*) NULL,
    ->   `deltaBucket` bigint(20) NOT NULL,
    ->   `deleted` bigint(20) NOT NULL,
    ->   `namespace` varchar(*) NULL,
    ->   `cityName` varchar(*) NULL,
    ->   `countryName` varchar(*) NULL,
    ->   `regionIsoCode` varchar(*) NULL,
    ->   `metroCode` bigint(20) NOT NULL,
    ->   `countryIsoCode` varchar(*) NULL,
    ->   `regionName` varchar(*) NULL
    -> )DUPLICATE KEY(`__time`)
    -> DISTRIBUTED BY HASH(`__time`) BUCKETS 10
    -> PROPERTIES (
    -> "replication_allocation" = "tag.location.default: 1"
    -> );
Query OK, 0 rows affected (0.04 sec)

mysql> insert into wikipedia select * from jdbc_druid_auth.druid.wikipedia; 
Query OK, 24433 rows affected (0.95 sec)
{'label':'label_35fd9215e010495d_8a9292f5b3251936', 'status':'VISIBLE', 'txnId':'2004'}

mysql> select count(*) from wikipedia;
+----------+
| count(*) |
+----------+
|    24433 |
+----------+
1 row in set (0.05 sec)

mysql> select count(*)  from jdbc_druid_auth.druid.wikipedia; 
+----------+
| count(*) |
+----------+
|    24433 |
+----------+
1 row in set (0.18 sec)

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@vinlee19 vinlee19 mentioned this pull request Jan 31, 2024
Copy link
Contributor

We're closing this PR because it hasn't been updated in a while.
This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and feel free a maintainer to remove the Stale tag!

@github-actions github-actions bot added the Stale label May 19, 2024
@github-actions github-actions bot closed this May 20, 2024
@alphaadidas
Copy link

+1

@alphaadidas
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2 participants