Skip to main content
Version: 5.3

hdhsource

Description​

Allows retrieving data from Hadoop Hive.

Note: To use the hdhsource source, an SM Hadoop Proxy connector must be configured.

Syntax​

hdhsource <db_name.table_name> [qsize=<int>]

Required Arguments​

  • db_name.table_name

Optional Arguments​

  • qsize — maximum number of records to retrieve

Source Configuration​

All source configurations are stored in settings.yml.

sme:
  dbs:
    hadoop_hive:
      connector_url: 'http://localhost'
      connector_port: 18081
      hadoop_hive_connection_string: 'jdbc:hive2://127.0.0.1:10000'
      hadoop_hive_user: ''
      hadoop_hive_password: ''

Description of Configuration Parameters​

  • connector_url — URL of the SM Hadoop Proxy connector
  • connector_port — port for the SM Hadoop Proxy connector
  • hadoop_hive_connection_string — connection string for Hadoop Hive
  • hadoop_hive_user — Hadoop Hive user
  • hadoop_hive_password — Hadoop Hive user password

Examples​

Example 1​

In this example, hdhsource reads the hr.employee_list table, filters records with the status Уволен (Dismissed) and keeps only the user_name field.

hdhsource 'hr.employee_list'
| search status="Уволен"
| fields user_name

Example 2​

In this example, a subquery to Hadoop Hive forms a list of values for format, which is then used in the search on the main source accessLogs.

hdhsource accessLogs
| search [ hdhsource 'hr.employee_list' | search status="Уволен" | fields user_name | format ]