VOOZH about

URL: https://dzone.com/articles/introducing-cloudera-sql-stream-builder-ssb

โ‡ฑ Introducing Cloudera SQL Stream Builder (SSB)


Related

  1. DZone
  2. Data Engineering
  3. Databases
  4. Introducing Cloudera SQL Stream Builder (SSB)

Introducing Cloudera SQL Stream Builder (SSB)

SSB is an improved release of Eventador's SQL Stream Builder with integration into Cloudera Manager, Cloudera Flink, and other streaming tools.

Likes
Comment
Save
14.8K Views

Join the DZone community and get the full member experience.

Join For Free

Cloudera SQL Stream Builder (SSB)

The initial release of Cloudera SQL Stream Builder as part of the CSA 1.3.0 release of Apache Flink and friends from Cloudera shows an integrated environment well integrated into Cloudera's Data Platform. SSB is an improved release of Eventador's SQL Stream Builder with integration into Cloudera Manager, Cloudera Flink, and other streaming tools.

In this initial release, the user is given a complete web UI to build, develop, test, and deploy enterprise Continuous SQL queries into YARN Apache Flink clusters. The initial source is any number of Kafka clusters and the first set of outputs are Kafka clusters and webhooks. You are also given the ability to build Materialized Views that act as constantly updated, but fast sources of data for REST clients to consume. This makes for a great interface to Kafka data from non-Kafka consumers.

All you need to do is type SQL against tables and the results can be in Materialized Views, Kafka topics, or webhooks. You can do rich SQL thanks to Apache Calcite including Joins, Order By, Aggregates, and more.

CSA 1.3.0 is now available with Apache Flink 1.12 and SQL Stream Builder! Check out this white paper for some details. You can get full details on the Stream Processing and Analytics available from Cloudera here.

This is an awesome way to query Kafka topics with continuous SQL that is deployed to scalable Flink nodes in YARN or K8. We can also easily define functions in JavaScript to enhance, enrich and augment our data streams. No Java to write; no heavy deploys or build scripts; we can build, test and deploy these advanced streaming applications all from your secure browser interface.

Example Queries:


SQL




xxxxxxxxxx
1


1
SELECT location, max(temp_f) as max_temp_f, avg(temp_f) as avg_temp_f,
2
                 min(temp_f) as min_temp_f
3
FROM weather2 
4
GROUP BY location



SQL




xxxxxxxxxx
1
13
9


1
SELECT HOP_END(eventTimestamp, INTERVAL '1' SECOND, INTERVAL '30' SECOND) as windowEnd,
2
       count(`close`) as closeCount,
3
       sum(cast(`close` as float)) as closeSum, avg(cast(`close` as float)) as closeAverage,
4
       min(`close`) as closeMin,
5
       max(`close`) as closeMax,
6
       sum(case when `close` > 14 then 1 else 0 end) as stockGreaterThan14 
7
FROM stocksraw
8
WHERE symbol = 'CLDR'
9
GROUP BY HOP(eventTimestamp, INTERVAL '1' SECOND, INTERVAL '30' SECOND)


SQL




x


1
SELECT scada2.uuid, scada2.systemtime, scada2.temperaturef, scada2.pressure,     
2
       scada2.humidity, scada2.lux, scada2.proximity
3
       scada2.oxidising,scada2.reducing , scada2.nh3,    
4
       scada2.gasko,energy2.`current`,    
5
       energy2.voltage,energy2.`power`,energy2.`total`,energy2.fanstatus
6
FROM energy2 JOIN scada2 ON energy2.systemtime = scada2.systemtime


SQL




xxxxxxxxxx
1
10


1
SELECT symbol, uuid, ts, dt, `open`, `close`, high, volume, `low`, `datetime`,          'new-high' message, 'nh' alertcode, 
2
       CAST(CURRENT_TIMESTAMP AS BIGINT) alerttime 
3
FROM stocksraw st 
4
WHERE symbol is not null 
5
AND symbol <> 'null' 
6
AND trim(symbol) <> '' 
7
AND CAST(close as DOUBLE) > 
8
(SELECT MAX(CAST(`close` as DOUBLE))
9
FROM stocksraw s 
10
WHERE s.symbol = st.symbol)



SQL




xxxxxxxxxx
1


1
SELECT  * 
2
FROM statusevents
3
WHERE lower(description) like '%fail%'



SQL




xxxxxxxxxx
1
26


1
SELECT  sensor_id as device_id,
2
  HOP_END(sensor_ts, INTERVAL '1' SECOND, INTERVAL '30' SECOND) as windowEnd,
3
  count(*) as sensorCount,
4
  sum(sensor_6) as sensorSum,
5
  avg(cast(sensor_6 as float)) as sensorAverage,
6
  min(sensor_6) as sensorMin,
7
  max(sensor_6) as sensorMax,
8
  sum(case when sensor_6 > 70 then 1 else 0 end) as sensorGreaterThan60
9
FROM iot_enriched_source
10
GROUP BY
11
  sensor_id,
12
  HOP(sensor_ts, INTERVAL '1' SECOND, INTERVAL '30' SECOND)



SQL




x


1
SELECT title, description, pubDate, `point`, `uuid`, `ts`, eventTimestamp
2
FROM transcomevents



Source Code:

Example SQL Stream Builder Run

We log in, then build our Kafka data source(s), unless they were predefined.

Next, we build a few virtual table sources for Kafka topics we are going to read from. If they are JSON we can let SSB determine the schema for us. Or we can connect to the Cloudera Schema Registry for it to determine the schema for AVRO data.

We can then define virtual table syncs to Kafka or webhooks.

We then run a SQL query with some easy to determine parameters and if we like the results we can create a materialized view.

References:

sql Stream processing Database kafka Apache Flink

Published at DZone with permission of Tim Spann. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Next-Gen Data Pipes With Spark, Kafka and k8s
  • Harnessing Real-Time Insights With Streaming SQL on Kafka
  • Real-Time Streaming Architectures: A Technical Deep Dive Into Kafka, Flink, and Pinot
  • Kafka JDBC Source Connector for Large Data

Partner Resources

ร—

Comments

The likes didn't load as expected. Please refresh the page and try again.

Let's be friends: